[c4lliope]

NixOS is used by deployment engineers for reproducible builds that can be pushed to any edge. Omnigraph belongs there, and the simplest approach is to build for the Nix package manager with a flake.nix at the root of this codebase.

Rather than open an issue, I'd like to share the code I have already, so that I can ask for help with a compilation error that I'm already experiencing.

This PR only adds, no removals - and the additions are all in .nix.\

I began with a build process using Crane,
because I was inspired by @mulatta's agent-devtools.nix.
This quickly became clunky, and depended on a Lance patch
that is probably unnecessary. I searched around more, and found some guides that recommended Naersk, a more modern, community-recommended build manager. Either way, I end up seeing the same error while compiling the lance-linalg dependency:

 > error: could not compile `lance-linalg` (lib) due to 1 previous error
 > warning: build failed, waiting for other jobs to finish...
 > error: intrinsic signature mismatch for `llvm.x86.avx512.vpdpbusd.512`: expected signature `<16 x i32> (<16 x i32>, <16 x i32>, <16 x i32>)`, found `<16 x i32> (<16 x i32>, <64 x i8>, <64 x i8>)`
 >  --> /rustc/b940084d7eb6a299eb4bfeb8e34901bc051e7ac4/library/core/src/../../stdarch/crates/core_arch/src/x86/avx512vnni.rs:892:4
 >
 >
 > [naersk] cargo returned with exit code 101, exiting

I'm new to Lance and to SIMD - seems like there's some processor magic at play here. I tried to add RUST_FLAGS with target features, as you can see - no luck on anything there.

Are there other build-time options that you use to conform the lance dependency to your needs? Naersk seems to be picking up the other Rust configs easily, so maybe there's an environmental consideration I'm missing.

[mulatta]

That's the nixpkgs LLVM issue. LLVM < 22 cannot process intrincics LTO in build time.
And current nixpkgs LLVM is under 22, so you should override LLVM version if you want to build without patch (may cause rebuild whole LLVM dependent package).

I think you can try as follows if you gonna use crane maybe?

toolchain = pkgs.rust-bin.stable."1.97.1".default;
craneLib = (crane.mkLib pkgs).overrideToolchain toolchain;

maybe this will avoid rebuild whole nixpkgs LLVM transitive dependecy while overriding LLVM dependent rust-toolchain packages

[c4lliope]

Superb! Naersk can load rust-toolchain.toml, I think this is all that's needed.
It goes directly to rust compilation, so it's using a precompiled LLVM from cache.

[c4lliope]

➜ result/bin/omnigraph version
omnigraph 0.14.0
internal-schema 16 (serves v14 to v16)
http-contract 0.14

[mulatta]

By the way, do you have any idea with CI/CD actions?
Otherwise, nix packaging should be maintained by manually every new commits. (Every hash will be changed for each new commits)

[ragnorc]

Thanks for this, @c4lliope, and @mulatta for the LLVM diagnosis. I pushed eace543 on top of a merge of main to make the flake build the way our release archives do:

• tokio_unstable was leaking into the binaries. naersk puts its path remapping in CARGO_BUILD_RUSTFLAGS, and Cargo merges that with the workspace .cargo/config.toml [build] rustflags, which hold the test-only --cfg tokio_unstable. Every release build sets RUSTFLAGS to replace that list (docs/dev/ci.md), so the flake now does too: the remapping plus, on arm64 macOS, the archives' -C code-model=large. Checked on the built dependency layer: tokio's fingerprint shows only the two --remap-path-prefix flags, and the runtime closure is glibc and gcc-lib alone.
• Same packages and features as the archives: --locked -p omnigraph-cli -p omnigraph-server -p omnigraph-azure-admission --features omnigraph-server/aws.
• One toolchain source: fenix's fromToolchainFile reads rust-toolchain.toml (naersk already uses fenix, so it now follows it). nixpkgs-mozilla and flake-utils are gone, and nixpkgs' own rustc (the LLVM < 22 one) is no longer on the build PATH.
• Version is read from crates/omnigraph-cli/Cargo.toml (it was hard-coded to 0.14.0). A clean tree stamps OMNIGRAPH_SOURCE_VERSION, so omnigraph version reports the commit. That stamp goes on the top-level build only, so a new commit doesn't rebuild the dependencies.
• Systems: x86_64-linux, aarch64-linux and aarch64-darwin, the archive targets other than Windows.
• CI: nix.yml builds all three on flake/toolchain changes and weekly, and checks that the binaries run and report their commit. It isn't a required check.
• Install docs (docs/user/install.md#nix) and a release note.

On @mulatta's question about maintenance: naersk vendors from Cargo.lock, so there's no per-commit hash. The only hash is the toolchain manifest's, and it changes when rust-toolchain.toml is bumped.

Verified locally with a full nix build on aarch64-linux; CI covers the other two systems.

[mulatta]

Since omnigraph build is heavy, i recommend you hestia for build caching. You don't need to host cache server. Just works with github actions and its cache maybe