fix: repin nixpkgs with fetchCargoVendor User-Agent hotfix (crates.io 403 breaking all module releases) - #173
Closed
danisharora099 wants to merge 1 commit into
Closed
Conversation
… 403) crates.io now rejects python-requests' default User-Agent with HTTP 403. The nixpkgs rev this flake follows (e9f00bd8, via logos-nix) predates the upstream fix (NixOS/nixpkgs@8209ba2b), so every fetchCargoVendor vendor-staging derivation — i.e. cargo vendoring in every module release that follows this flake's nixpkgs — fails ecosystem-wide. Fix: pin nixpkgs to e9f00bd8 + exactly one commit that applies the upstream User-Agent fix ONLY to the fixed-output vendorStaging derivation (the only network-facing step). Because fixed-output derivations are absorbed by hashDerivationModulo, no input-addressed derivation changes: the package set is byte-identical, every store path (Qt, toolchains, even the -vendor outputs) is unchanged, binary-cache reuse is unaffected and all existing cargoHash values stay valid. Only the staging .drv changes — its fetch now sends a proper User-Agent. Alternatives rejected: advancing to current nixos-unstable (~10 months of toolchain churn — the eventual path via logos-nix, not a hotfix) and the naive whole-script cherry-pick (cascades through the input-addressed -vendor step into python cryptography and Qt rebuilds). Fixes logos-co#159 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
danisharora099
added a commit
to logos-co/lez-faucet
that referenced
this pull request
Aug 3, 2026
No Nix build of this module has ever got past vendoring. Every leg of
every run, on every platform, dies on its first crate:
Failed to fetch file from
https://crates.io/api/v1/crates/adler2/2.0.1/download. Status code: 403
darwin-arm64 fails the same way, so this is not something the Linux port
introduced -- it is what blocks today's macOS releases too, and it is the
only thing between PR #6 and six green legs.
crates.io answers 403 to the User-Agent nixpkgs' cargo vendor fetcher
sends. The fetcher sets none, so requests sends `python-requests/*`;
measured against the failing URL, that is a 403, `curl/*` is a 403, and a
descriptive one is a 200. Upstream fixed it in NixOS/nixpkgs#512735
(2026-04-26) and later moved the download off the API endpoint entirely.
The nixpkgs this flake follows through logos-module-builder is e9f00bd8
(2025-09), which predates both, and repinning it is not available: it is
this module's whole Qt/C++ toolchain.
So the fix is local. fetch-cargo-vendor.nix is re-expressed in the flake,
its two helper scripts rebuilt from the pinned tree with `readFile`, and
three substitutions applied to the fetching one.
The third is the one that actually resolves it: fetch tarballs from
`https://static.crates.io/crates/<name>/<name>-<version>.crate` instead
of `https://crates.io/api/v1/crates/<name>/<version>/download`. That is
the CDN cargo itself downloads from -- the registry's `dl` key -- and
where upstream moved after #512735. It applies neither the User-Agent
policy that produces the 403 nor the per-IP rate limit that produces the
429 six concurrent legs would hit next; measured, it returns 200 for
every user-agent tried, `python-requests/2.32.3` and `curl/*` included,
on the exact URLs that 403.
The other two are belt. A descriptive User-Agent, so the request is not
one crates.io has a policy against wherever it lands. And 429 added to
the urllib3 `Retry` upstream already mounts but scopes to 5xx only -- so
the one status that means "back off and come back" becomes the one it
retries -- with jitter so parallel legs do not retry in lockstep,
`respect_retry_after_header` pinned so a `Retry-After` beats our own
curve, and a bound of 12 attempts / 60s so a genuinely down registry
fails in minutes rather than riding a 90-minute timeout. 403 is
deliberately not retried: that is a policy answer, and retrying it would
only reach the same error slower. Retry alone was measured insufficient
in the sibling swap module -- legs rode the full budget and still came
out with `RetryError: ... too many 429 error responses` -- which is why
the CDN switch is the fix and this is only the belt.
Scoping matters as much as the substitutions. This patches a private
copy via `replaceStrings`, not the shared `fetch-cargo-vendor-util` via
`applyPatches`. Patching the real one moves the `outPath` of every Rust
package in nixpkgs -- `qt6.qtdeclarative`,
`python3Packages.cryptography` and `cargo-auditable` all shift -- which
would cost this build its binary-cache hits on exactly the cold CI
runners it is meant to help, and could mean hours of Qt rebuilds. The
scoped form leaves all three at their stock paths and moves nothing but
lez-faucet-ffi; nixpkgs' own `rustPlatform` is untouched. The scripts are
re-expressed rather than generated with `toFile` because a generated file
embeds `${nixpkgs}` store paths as text, and registering those as
references makes evaluation die with `path '...-source' is not valid` on
CI's Determinate Nix while evaluating clean on stock Nix.
The substitutions are asserted, so a nixpkgs bump that reshapes the
fetcher fails evaluation loudly with the string it could not find,
instead of silently reverting to a 403. All of it deletes once
logos-module-builder's nixpkgs carries the upstream fix
(logos-co/logos-module-builder#173) -- but nothing here waits on that.
cargoHash is unchanged, and now proven rather than asserted. It becomes
`cargoDeps.hash` so vendoring goes through the patched fetcher, but it
covers only the fixed-output vendor-staging tree -- checksum-verified
crate tarballs, git checkouts and Cargo.lock -- so it tracks the
lockfile, not the nixpkgs building it or the host serving the bytes. The
script checksums every tarball against Cargo.lock, so a changed download
host cannot quietly vendor something else; it would fail loudly.
Verified on an aarch64 macOS host:
- `lez-faucet-ffi-0.2.0-vendor-staging` builds cold and lands on the
committed sha256-bynsEIeov27VyVr8NdHVcpPwEQr8Kbmd93o+B61XWEM= -- a
real download of the full crate set through static.crates.io, matching
a hash measured against the API endpoint. Same bytes, different host.
- `nix build .#packages.aarch64-darwin.lez-faucet-ffi` succeeds, the
first time any Nix build of this module has. 9.3M dylib, install name
`@rpath/liblez_faucet_ffi.dylib`, exporting exactly the ten locked ABI
entry points and the header beside it.
- `nix eval .#packages.<sys>.lgx-portable.drvPath` instantiates the full
derivation graph for aarch64-darwin, x86_64-linux and aarch64-linux,
forcing every input including the vendor FOD.
- qt6.qtdeclarative, python3Packages.cryptography and cargo-auditable
evaluate to their stock outPaths, confirming the scoping.
Co-Authored-By: Claude <noreply@anthropic.com>
danisharora099
added a commit
to logos-co/lez-faucet
that referenced
this pull request
Aug 3, 2026
No Nix build of this module has ever got past vendoring. Every leg of
every run, on every platform, dies on its first crate:
Failed to fetch file from
https://crates.io/api/v1/crates/adler2/2.0.1/download. Status code: 403
darwin-arm64 fails the same way, so this is not something the Linux port
introduced -- it is what blocks today's macOS releases too, and it is the
only thing between PR #6 and six green legs.
crates.io answers 403 to the User-Agent nixpkgs' cargo vendor fetcher
sends. The fetcher sets none, so requests sends `python-requests/*`;
measured against the failing URL, that is a 403, `curl/*` is a 403, and a
descriptive one is a 200. Upstream fixed it in NixOS/nixpkgs#512735
(2026-04-26) and later moved the download off the API endpoint entirely.
The nixpkgs this flake follows through logos-module-builder is e9f00bd8
(2025-09), which predates both, and repinning it is not available: it is
this module's whole Qt/C++ toolchain.
So the fix is local. fetch-cargo-vendor.nix is re-expressed in the flake,
its two helper scripts rebuilt from the pinned tree with `readFile`, and
three substitutions applied to the fetching one.
The third is the one that actually resolves it: fetch tarballs from
`https://static.crates.io/crates/<name>/<name>-<version>.crate` instead
of `https://crates.io/api/v1/crates/<name>/<version>/download`. That is
the CDN cargo itself downloads from -- the registry's `dl` key -- and
where upstream moved after #512735. It applies neither the User-Agent
policy that produces the 403 nor the per-IP rate limit that produces the
429 six concurrent legs would hit next; measured, it returns 200 for
every user-agent tried, `python-requests/2.32.3` and `curl/*` included,
on the exact URLs that 403.
The other two are belt. A descriptive User-Agent, so the request is not
one crates.io has a policy against wherever it lands. And 429 added to
the urllib3 `Retry` upstream already mounts but scopes to 5xx only -- so
the one status that means "back off and come back" becomes the one it
retries -- with jitter so parallel legs do not retry in lockstep,
`respect_retry_after_header` pinned so a `Retry-After` beats our own
curve, and a bound of 12 attempts / 60s so a genuinely down registry
fails in minutes rather than riding a 90-minute timeout. 403 is
deliberately not retried: that is a policy answer, and retrying it would
only reach the same error slower. Retry alone was measured insufficient
in the sibling swap module -- legs rode the full budget and still came
out with `RetryError: ... too many 429 error responses` -- which is why
the CDN switch is the fix and this is only the belt.
Scoping matters as much as the substitutions. This patches a private
copy via `replaceStrings`, not the shared `fetch-cargo-vendor-util` via
`applyPatches`. Patching the real one moves the `outPath` of every Rust
package in nixpkgs -- `qt6.qtdeclarative`,
`python3Packages.cryptography` and `cargo-auditable` all shift -- which
would cost this build its binary-cache hits on exactly the cold CI
runners it is meant to help, and could mean hours of Qt rebuilds. The
scoped form leaves all three at their stock paths and moves nothing but
lez-faucet-ffi; nixpkgs' own `rustPlatform` is untouched. The scripts are
re-expressed rather than generated with `toFile` because a generated file
embeds `${nixpkgs}` store paths as text, and registering those as
references makes evaluation die with `path '...-source' is not valid` on
CI's Determinate Nix while evaluating clean on stock Nix.
The substitutions are asserted, so a nixpkgs bump that reshapes the
fetcher fails evaluation loudly with the string it could not find,
instead of silently reverting to a 403. All of it deletes once
logos-module-builder's nixpkgs carries the upstream fix
(logos-co/logos-module-builder#173) -- but nothing here waits on that.
cargoHash is unchanged, and now proven rather than asserted. It becomes
`cargoDeps.hash` so vendoring goes through the patched fetcher, but it
covers only the fixed-output vendor-staging tree -- checksum-verified
crate tarballs, git checkouts and Cargo.lock -- so it tracks the
lockfile, not the nixpkgs building it or the host serving the bytes. The
script checksums every tarball against Cargo.lock, so a changed download
host cannot quietly vendor something else; it would fail loudly.
Verified on an aarch64 macOS host:
- `lez-faucet-ffi-0.2.0-vendor-staging` builds cold and lands on the
committed sha256-bynsEIeov27VyVr8NdHVcpPwEQr8Kbmd93o+B61XWEM= -- a
real download of the full crate set through static.crates.io, matching
a hash measured against the API endpoint. Same bytes, different host.
- `nix build .#packages.aarch64-darwin.lez-faucet-ffi` succeeds, the
first time any Nix build of this module has. 9.3M dylib, install name
`@rpath/liblez_faucet_ffi.dylib`, exporting exactly the ten locked ABI
entry points and the header beside it.
- `nix eval .#packages.<sys>.lgx-portable.drvPath` instantiates the full
derivation graph for aarch64-darwin, x86_64-linux and aarch64-linux,
forcing every input including the vendor FOD.
- qt6.qtdeclarative, python3Packages.cryptography and cargo-auditable
evaluate to their stock outPaths, confirming the scoping.
Co-Authored-By: Claude <noreply@anthropic.com>
Author
|
@Khushboo-dev-cpp @dlipicar unable to assign you as reviewers |
| # so the package set is byte-identical and all fixed-output hashes stay | ||
| # valid. Revert to `nixpkgs.follows = "logos-nix/nixpkgs"` once logos-nix | ||
| # advances past 2026-04-26 (first nixos-unstable containing the fix). | ||
| nixpkgs.url = "github:danisharora099/nixpkgs/eaec81d3b8a8d2339e25c718a1650b2c45adf726"; |
Contributor
There was a problem hiding this comment.
seems like the infra team needs to look at this ? and the fix should land in logos-nix repo ? @dlipicar , who do we ping ?
Contributor
There was a problem hiding this comment.
yeah I'm guessing the fix should go in logos-nix.
@danisharora099 could you raise the PR with whatever you changed in your fork there? then we re-pin this one
Contributor
There was a problem hiding this comment.
8 lines of useless LLM slop comment, that must be a new record. You do know it does this so you pay more for input tokens when re-parsing files?
As for nixpkgs commit picked here, why do you need to use your fork of nixpkgs from @danisharora099 c?
Second of all, the fetchCargoVendor fix was applied in NixOS/nixpkgs#512735 so why can't you just update nixpkgs in logos-nix to one that includes that? Windows nixpkgs uses b5aa0fbd538984f6e3d201be0005b4463d8b09f8 commit which already includes the rustPlatform.fetchCargoVendor: set custom User-Agent in v2 fix, so why not just unify it?
Contributor
There was a problem hiding this comment.
Also, what in the hell is this flake.lock...
~/work/logos-module-builder master
> wc -l flake.lock
76166 flake.lock
~/work/logos-module-builder master
> jq '.nodes | keys' flake.lock | grep nixpkgs | wc -l
1016
Author
There was a problem hiding this comment.
8 lines of useless LLM slop comment, that must be a new record. You do know it does this so you pay more for input tokens when re-parsing files?
XD flip side is it saves working out what that line is doing from scratch every time - cheaper to read it than to rediscover it. Happy to trim it though.
As for nixpkgs commit picked here, why do you need to use your fork of nixpkgs from @danisharora099 c?
Second of all, the fetchCargoVendor fix was applied in NixOS/nixpkgs#512735 so why can't you just update nixpkgs in logos-nix to one that includes that? Windows nixpkgs uses b5aa0fbd538984f6e3d201be0005b4463d8b09f8 commit which already includes the rustPlatform.fetchCargoVendor: set custom User-Agent in v2 fix, so why not just unify it?
Fair points. WIP
Contributor
There was a problem hiding this comment.
it saves working out what that line is doing from scratch every time
A comment should be one line, two lines if there's a link to a related issue or PR. Anything beyond that is noise.
And besides, actual reason for a change is in Git history. Just use git blame.
Author
|
Closing in favour of the logos-nix route the reviewers asked for, now concrete: logos-co/logos-nix#6 carries the fetchCargoVendor User-Agent fix as an overlay on the current pin (no Qt bump), and #220 is the one-line hookup that applies it where this repo imports nixpkgs. Once both land plus a lock bump, #159 can close. The Qt bump itself proceeds separately on logos-co/logos-nix#5 per its 2026-08-27 plan comment. Nothing from this branch is needed after that. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #159 — every module release's cargo vendoring currently dies with
crates.io … Status code: 403.Problem
crates.io now rejects requests carrying python-requests' default
User-Agent. The nixpkgs rev this flake follows (e9f00bd8, vialogos-nix/nixpkgs, 2025-09-28) predates the upstream fix, so itsfetchCargoVendorhelper (pkgs/build-support/rust/fetch-cargo-vendor-util.py) creates arequests.Session()with noUser-Agent— every*-vendor-stagingfixed-output derivation gets a 403. Because module flakes donixpkgs.follows = "logos-module-builder/nixpkgs", this breaks the whole ecosystem's release CI (evidence in #159).Upstream fixed it in NixOS/nixpkgs@
8209ba2b(merged 2026-04-26 in NixOS/nixpkgs#512735). It was never backported to release-25.05 or release-25.11 — only nixos-unstable ≥ 2026-04-26 has it.Fix: patched pin, scoped to the fixed-output fetch (zero rebuild fallout)
Options assessed:
e9f00bd8— tried first, rejected: the patched util script is also referenced by the input-addressed*-vendorstep, so the change cascades throughpython3Packages.cryptographyinto Qt rebuilds (empirically observed: qtdeclarative 6.9.2 compiling from source). On ephemeral CI runners that means hours per release.e9f00bd8+ one commit that applies the UA only to the fixed-outputvendorStagingderivation (the only network-facing step). Fixed-output derivations are absorbed byhashDerivationModulo, so no input-addressed derivation changes at all.Store-path invariance, verified by evaluation at both refs (unpatched
e9f00bd8vs patchedeaec81d3):cargo-auditable68xislcb…68xislcb…(identical)python3Packages.cryptographyqlpkk5k0…qlpkk5k0…(identical)qt6.qtdeclarativefn7iqpps…fn7iqpps…(identical)fetchCargoVendor-vendoroutputh41y5c2x…h41y5c2x…(identical)-vendor-staging.drvsbp0k9gd…hknibz45…(only this changes)So: binary-cache substitution from cache.nixos.org is completely unaffected, existing
cargoHashvalues stay valid, and the only thing that changes is that the staging fetch now sendsUser-Agent: nixpkgs-fetchCargoVendor/1 (https://github.com/NixOS/nixpkgs).A plain overlay was ruled out:
fetchCargoVendorUtilis alet-binding inside nixpkgs'fetch-cargo-vendor.nix,makeRustPlatformwires the fetcher via a nixpkgs-relativecallPackagepath, and module flakes import the followed nixpkgs themselves — nothing short of changing the nixpkgs source reaches theirrustPlatform.buildRustPackage { cargoHash = …; }callsites transparently.The patched branch is
danisharora099/nixpkgs#patch/fetch-cargo-vendor-ua-e9f00bd8(single commiteaec81d3, +20/-1 on one file). Recommended follow-up for maintainers: push that same commit tologos-co/nixpkgs(which already hosts patch branches, e.g.patch/mingw-qt6) and flip the owner inflake.nix— rev and narHash stay identical, so only the lock'sownerfield changes.Validation
Before (pin
e9f00bd8) — minimalrustPlatform.fetchCargoVendorof a lockfile containingbitflags 2.9.4(the exact crate from the CI evidence), fresh store path:After (patched
eaec81d3) — identical derivation, staging output force-deleted first:CI-parity acid test — scratch clone of
logos-co/eth-lez-atomic-swaps(the repo from the #159 evidence),nix build .#lgx-portableinswap-module/, with the pre-seededswap-ffi/zerokitvendor-staging, vendor, and compiled outputsnix store deleted so the crates.io fetches actually run cold: the full ~380-crate staging fetch completes with zero 403s, swap-ffi compiles, and the lgx bundle assembles end-to-end (EXIT=0). Full details + logs in the #159 comment.Cross-validation: current nixos-unstable's official fetcher produces a byte-identical vendor staging (same FOD hash) as this patch on the same input — the patch is behaviorally equivalent to upstream.
Two adjacent findings (out of scope here, but release CI needs them)
logos-deliverypins its own nixpkgs (23d72dab, 2026-02-07 — verified also UA-less), which is what vendors zerokit. That pin is outside this flake's follows chain, so zerokit vendoring stays broken until logos-delivery gets the same fix. A ready-made branch exists:patch/fetch-cargo-vendor-ua-23d72dab(used successfully in the acid test via--override-input delivery_module/logos-delivery/nixpkgs). Worth a sibling issue on logos-delivery.cargoHashis stale (sha256-Xu7WL+…): a genuine cold fetch yieldssha256-lADceazcOPwMkeBwUElibZ5oMPsLD1gIZdVGkGQfX0E=— confirmed independently by current nixos-unstable's official fetcher. The pre-seeded-store release workaround has been masking this; that repo needs the hash bump regardless of this PR.Consumers pick this fix up by bumping their
logos-module-builderinput. Revert path is documented in the flake comment: return tofollows = "logos-nix/nixpkgs"once logos-nix advances past 2026-04-26.🤖 Generated with Claude Code