master CI broken: crates.io 403s curl UA on /api/v1 download; importCargoLock needs static.crates.io

[igor-sirotin]

Summary

master CI is broken. The build-and-test job fails at the Build module step (nix build -L) on both ubuntu-latest and macos-latest. It is not a code regression — re-running master's own last-green commit (9b22b52, originally green 2026-05-26) now fails reproducibly.

Symptom

The Nix build fails fetching a handful of Rust crates from crates.io:

trying https://crates.io/api/v1/crates/alloy-rlp/0.3.13/download
curl: (22) SSL certificate ... unable to get local issuer certificate (20)
curl: (22) The requested URL returned error: 403
error: cannot download crate-alloy-rlp-0.3.13.tar.gz from any mirror

Only 4 crates fail (all RLN/zerokit deps pulled in via the logos-chat rust bundle):
alloy-rlp, ark-bn254, ark-circom, ark-crypto-primitives.

Failing runs (all PRs against master, plus master re-run):

• master re-run of 9b22b52: run 26462630550 → failure
• experiment/try-remove-deferred-emit: run 26589693972 → failure
• expand_e2e_coverage: runs 26575763494, 26478213518, 26463159157 → failure

Root cause

crates.io now enforces its data-access policy by User-Agent. The /api/v1/crates/{name}/{version}/download endpoint returns 403 for a bare curl/* UA:

$ curl -s https://crates.io/api/v1/crates/serde/1.0.0/download
{"errors":[{"detail":"We are unable to process your request at this time. This usually means that you are in violation of our API data access policy ... "}]}

$ curl -A "cargo 1.93.0" -o /dev/null -w "%{http_code}" https://crates.io/api/v1/crates/serde/1.0.0/download
302   # works with a cargo UA

Nixpkgs' importCargoLock (used by buildRustPackage { cargoLock.lockFile = ...; } in logos-chat/nix/rust_bundle.nix and nix/libchat.nix) fetches each crate tarball via fetchurl from exactly that endpoint, with a curl/* UA → 403.

Why only 4 crates / why now: most crate tarballs are served from the logos-co Cachix cache, so they never re-fetch. These 4 are a cache miss (verified: their FOD narinfo returns 404 from logos-co.cachix.org), so they fall back to crates.io and fail. Master was green on 2026-05-26 off a warm cache / before crates.io began enforcing this.

The pinned nixpkgs (e9f00bd893…) hardcodes:

"https://github.com/rust-lang/crates.io-index" = "https://crates.io/api/v1/crates";

nixpkgs fixed this in commit f830e6112b4d (2026-05-27) — "rustPlatform.importCargoLock: download crates from static.crates.io" — switching to:

"https://github.com/rust-lang/crates.io-index" = "https://static.crates.io/crates";

static.crates.io serves the same path format and is not UA-gated (verified 200).

Fix options

1. Durable (recommended): in logos-messaging/logos-chat, bump the nixpkgs input used by the rust bundle to a rev that includes nixpkgs f830e6112b4d (2026-05-27) or later, so importCargoLock fetches from static.crates.io. Then bump the logos-chat input rev in this repo's flake.nix / flake.lock. (Same cargoLock change applies to nix/libchat.nix.)
2. Immediate unblock (no code change): re-warm the logos-co Cachix cache with the 4 missing crate FODs (e.g. build with a nixpkgs that uses static.crates.io and cachix push). Restores the green state without touching the pins.

Notes

• Also surfaced (non-blocking): actions/checkout@v4 and cachix/cachix-action@v15 run on the soon-deprecated Node.js 20.

[igor-sirotin]

Same issue already solved in status-im/status-go (prior art)

status-im hit the identical crates.io 403 (same crate family: alloy-rlp, ark-*, ethers-core, scale-info; same cargo-vendor-dir cascade). Their fix chain:

• fix: bring in zerokit and circom-compat-ffi with fixes to fetching from crates.io status-im/status-go#7485 — bumps the zerokit + circom-compat-ffi input revs to pull in the fixes (analogous to bumping our logos-chat input).
• nix: fix fetching crates via user agent vacp2p/zerokit#415 — "nix: fix fetching crates via user agent": adds a nixpkgs overlay that forces a User-Agent on the crate fetcher (their nixpkgs uses the newer fetchCargoVendor, a Python fetch-cargo-vendor-util whose default python-requests/* UA gets WAF-403'd). Also bumps cargoHash.
• nix: fix fetching of crates which now require agent overlay logos-storage/circom-compat-ffi#11 — "nix: fix fetching of crates which now require agent overlay".

Note: our failing build uses the older importCargoLock/curl path (curl/* UA → crates.io/api/v1/.../download), not the python-requests path, so the patch site differs slightly. Two proven routes:

1. Bump nixpkgs in logos-chat's rust bundle to ≥ f830e6112b4d (2026-05-27) so importCargoLock uses static.crates.io (not UA-gated).
2. UA-overlay approach as in zerokit#415, adapted to the curl fetcher.

Then bump the logos-chat input rev here.