fix(add): stop the similar-name gate refusing popular packages - #658
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Pull request overview
Fixes the add supply-chain similar-name (typosquat) gate so it no longer blocks popular packages, and makes the gate depend on a genuinely download-ranked popularity corpus that is correctly generated and shipped in release builds.
Changes:
- Stop the similar-name scan once it reaches the requested package’s own popularity entry, so only more popular lookalikes can be suggested.
- Add a build-time “ranked corpus” signal (
popular_package_names_are_ranked) and skip the similar-name gate when the build only has the unranked primer-name fallback. - Update release/mac build workflows to generate and ship
popular-top100000-v1.jsonsoAUBE_REQUIRE_PRIMER=1builds no longer panic.
Reviewed changes
Copilot reviewed 2 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| vendor/aube/crates/aube/src/commands/add_supply_chain.rs | Fixes similar-name candidate selection (stop at self) and skips the gate on unranked corpora; adds unit tests. |
| vendor/aube/crates/aube-resolver/src/primer.rs | Introduces popular_package_names_are_ranked() and documents ranked vs fallback corpora; adds invariant test. |
| vendor/aube/crates/aube-resolver/src/lib.rs | Re-exports popular_package_names_are_ranked() for consumers. |
| vendor/aube/crates/aube-resolver/build.rs | Emits a “ranked corpus” signal during build when embedding the popularity corpus. |
| .github/workflows/release.yml | Generates + uploads the popularity corpus alongside the primer for release builds. |
| .github/workflows/mac-build.yml | Generates the popularity corpus so mac-build doesn’t fail under AUBE_REQUIRE_PRIMER=1. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The typosquat gate compared the requested name against every entry in the popularity corpus except an exact self-match, so being IN the corpus was no defence. Measured against the real 100,000-name corpus, that rejects almost every package worth installing — and the name it points at is frequently the squatter: react -> "did you mean preact?" lodash -> "did you mean loadash?" express -> "did you mean expres?" debug -> "did you mean dbug?" axios -> "did you mean gaxios?" ms -> "did you mean ws?" A package among the 100,000 most-downloaded on npm is not a typosquat, so a request the corpus lists at all is now cleared outright. A name the corpus does not list is still scanned against all of it, so lodahs -> lodash is untouched. The blunt form of that rule is what the precision requires. Restricting candidates to names that OUTRANK the request looks sufficient — it fixes every line above — but still refuses 847 of the top 10,000, measured by running the gate over the corpus itself: ws for ms, qs for ms, micromatch for picomatch, safer-buffer for safe-buffer. All legitimate, and this gate hard-fails a non-interactive install. Clearing corpus members outright takes that to 0 of the top 10,000. What it gives up is a squat popular enough to rank (expres, #58196), which has real download volume and is the OSV check's and the downloads floor's territory rather than a spelling heuristic's. The gate reads absence from the corpus as "nobody installs this", which is only true of the real list. When popular-top100000-v1.json is absent, build.rs falls back to the metadata-primer names, which write_package_blob derives from a BTreeMap — ALPHABETICALLY — and which number 100 in dev and 2000 in release. There, absence means nothing and the reported "top-100,000 popularity rank #N" is an alphabetical position: that is how the gate came to refuse `nub add ms` in the nightly lockfile-mutation differential, where qs sits at alphabetical position 70. build.rs now reports whether the corpus is the real download-ranked one, and the gate declines to run when it is not. Release builds set AUBE_REQUIRE_PRIMER=1, under which build.rs treats the corpus as mandatory — but nothing produced it, so every Build leg of the release died in write_popular_names_blob (run 30727744051; the same panic remote-build.ts already records hitting). release.yml generates and ships it in the primer artifact now, and mac-build.yml generates its own: having node is not enough there, because build.rs regenerates the primer by shelling out to generate-primer.mjs WITHOUT --popular-names-out. This is also what makes the gate live for released binaries for the first time, so it lands together with the two fixes above rather than after them.
b624a48 to
919041b
Compare
![pullfrog[bot]](https://avatars.githubusercontent.com/in/1768019?s=60&v=4){kind=small}
There was a problem hiding this comment.
Important
The break fix is correct and a large improvement, but the rank comparison is necessary rather than sufficient. Measured against the real 100,000-name corpus, the post-fix algorithm still refuses ws, qs, micromatch, safer-buffer and http-proxy-agent from the top 300 — and this PR is what makes the gate live on release binaries. Worth resolving the residue, or the sequencing, before merge.
Reviewed changes — full read of the 6-file diff, plus the surrounding gate, build.rs, generate-primer.mjs, and both workflows.
- Similar-name scan stops at the request's own rank —
find_similar_package_namenowbreaks instead ofcontinueing on an exact corpus hit, so only names that outrank the request are candidates. - New ranked-corpus flag —
build.rsemitscargo:rustc-env=AUBE_POPULAR_NAMES_RANKED=1whenpopular-top100000-v1.jsonis present, surfaced asaube_resolver::popular_package_names_are_ranked(). - The gate declines to run on an unranked build —
similar_name_gatereturns early, so the alphabetical metadata-primer fallback can no longer be reported as a popularity rank. This is the direct fix for the nightlynub add ms@2.1.3failure. - The corpus is generated and shipped in CI — new steps in
release.yml(with 3× retry, uploaded on themetadata-primerartifact) andmac-build.yml, unblocking theBuildlegs that were panicking underAUBE_REQUIRE_PRIMER=1. - Six new unit tests covering the rank comparison both ways, the unlisted-request scan, and the ranked-flag invariant.
⚠️ The corpus goes live in the same change that fixes the algorithm
Shipping popular-top100000-v1.json is what turns this gate on for released binaries for the first time, and the PR body says so explicitly. That makes the residual false-positive rate a release-blocking property rather than a follow-up: on a binary built from this branch, nub add ws in CI exits non-zero. Whether to land all three parts together, or to land the two fixes now and the corpus once the residue is closed, is a call for the maintainer.
Technical details
# Sequencing: algorithm fix vs. corpus enablement
## Context
- `.github/workflows/release.yml:363-381` and `.github/workflows/mac-build.yml:97-104` generate the ranked corpus.
- `.github/workflows/release.yml:393-395` adds it to the `metadata-primer` artifact, which the `build` job downloads into `vendor/aube/crates/aube-resolver/data/`.
- Consequence: `popular_package_names_are_ranked()` is true for every released binary, so `similar_name_gate` runs for real for the first time.
## Required outcome
- A decision, recorded in the PR, on one of:
1. Close the residual false positives (see the inline comment on `add_supply_chain.rs:873`) in this PR, then ship the corpus.
2. Ship the two fixes now and gate the corpus behind a follow-up.
3. Ship as-is, accepting that ~6% of the top 2000 npm packages hard-fail `nub add` in non-interactive contexts.
## Open questions for the human
- Is a hard refusal the right default for this gate at all, given the OSV advisory check and the weekly-downloads floor already run? A warning plus a prompt-only-when-interactive posture would make a false positive recoverable rather than blocking.ℹ️ Nitpicks
ranked_flag_matches_the_embedded_corpus(vendor/aube/crates/aube-resolver/src/primer.rs:438) never executes its assertion in CI. No job both embeds the corpus and runscargo test—release.yml'stestandconformancejobs and all ofci.ymlbuild without it. The invariant is worth stating, but it currently only fires on a developer's release-style build.mac-build.yml's corpus step has no retry, whilerelease.yml's wraps the same command in a 3× loop with 30s backoff. A transientregistry.npmjs.orgblip fails the whole mac build. Low stakes given it is manually dispatched, but the asymmetry is unintentional-looking.
Claude Opus ｜ 𝕏
| for (index, candidate) in corpus.lines().enumerate() { | ||
| if name == candidate { | ||
| continue; | ||
| break; |
There was a problem hiding this comment.
break is the right change, but "the lookalike outranks the request" is necessary rather than sufficient — a legitimate package that happens to be outranked by a similar name is still refused. Measured over the real 100,000-name corpus, this goes from 154 of the top 300 refused down to 5, but the survivors are ws→ms, qs→ms, micromatch→picomatch, safer-buffer→safe-buffer, http-proxy-agent→https-proxy-agent, and 118 of the top 2000 including globby→glob, vite→vitest, chai→chalk and expect→execa. Since confirm_similar_package hard-refuses without a TTY, nub add ws fails outright in CI.
Technical details
# Residual false positives after the rank comparison
## Affected sites
- `vendor/aube/crates/aube/src/commands/add_supply_chain.rs:873` — the scan stops at the request's own rank, which removes candidates *less* popular than the request but leaves every more-popular lookalike eligible regardless of how popular the request itself is.
- `vendor/aube/crates/aube/src/commands/add_supply_chain.rs:1459-1468` — `similar_name_still_flags_a_request_outranked_by_its_lookalike` asserts `qs` → `ms`, which is one of the false positives above. Any fix has to change this test.
- `vendor/aube/crates/aube/src/commands/add_supply_chain.rs:977-983` — `LowDownloadPrompt::Terminal` without a TTY on both stdin and stderr is an unconditional `Err(ERR_AUBE_SIMILAR_PACKAGE_NAME)`, so this is a hard failure in CI, not a prompt.
## How this was measured
Fetched the live corpus (`https://raw.githubusercontent.com/jdx/aube-primer-packages/main/data/popular.json`, exactly 100,000 download-ranked names — the same source `generate-primer.mjs --popular-names-out` writes) and reimplemented `comparable_name_parts` + the bounded OSA distance with the same thresholds (2 when `max(len) >= 5`, else 1), running each of the top N corpus names back through the scan as the requested name.
| top N | refused with `continue` | refused with `break` |
| --- | --- | --- |
| 300 | 154 | 5 |
| 2000 | — | 118 |
Ranks in that corpus: `ms` #9, `chalk` #13, `ws` #56, `qs` #119, `vite` #788, `vitest` #768.
## Required outcome
- A request that is itself unambiguously popular must not be refused as a typosquat of another popular package. `ws`, `qs`, `micromatch`, `safer-buffer`, `http-proxy-agent`, `globby` and `vite` should all install without a prompt on a ranked build.
- `lodahs`, `raect`, `axois`, `expres` must stay refused.
## Suggested approach (optional)
Add a second condition alongside the rank stop: skip the gate when the requested name is itself inside some high-confidence prefix of the corpus. A package in the top few thousand by weekly downloads is not plausibly a squat, whatever it resembles. The threshold is a judgement call, but the measurement above gives a way to calibrate it — pick the smallest cutoff that empties the top-N residue while leaving the known squats flagged.
A relative-distance rule (require `distance` to be small *relative to name length*) would separately kill the 2-character cases like `ws`/`ms` and `qs`/`ms`, where a single-character edit is half the name.| /// gates itself on this flag instead. | ||
| fn write_popular_names_blob(out_dir: &Path, source: &Path, fallback_names: &[String]) { | ||
| if source.is_file() { | ||
| println!("cargo:rustc-env=AUBE_POPULAR_NAMES_RANKED=1"); |
There was a problem hiding this comment.
The flag is emitted on file existence alone, before the JSON is parsed and before the names.len() != POPULAR_NAMES_TOP check below — and that length check only runs under primer_required(). So a short, truncated or hand-made popular-top100000-v1.json in a non-release build claims a ranking it cannot back, which is exactly what the doc comment above says the flag must never do. Emitting it after validation, gated on the length matching POPULAR_NAMES_TOP, would make the flag mean what it says in every build configuration.
| #[test] | ||
| fn similar_name_still_flags_a_request_outranked_by_its_lookalike() { | ||
| assert_eq!( | ||
| find_similar_package_name("qs", "ms\nws\nqs\n"), |
There was a problem hiding this comment.
There was a problem hiding this comment.
Important
The new exemption rule is correct and the false-positive problem is genuinely solved. One premise it rests on does not survive measurement: the weekly-downloads floor cannot be the backstop for an in-corpus squat, because top-100,000 membership implies download volume far above the default floor. Measured live: dbug 107,160/wk, loadash 38,910/wk, expres 6,194/wk, crossenv 2,219/wk, against a lowDownloadThreshold default of 1000.
Reviewed changes since b624a484 — the rank comparison was replaced with an outright exemption, and the tests and prose were rewritten around it.
break→return Noneon an exact corpus hit, so any request the corpus lists is cleared regardless of what surrounds it. Returning immediately (rather than breaking) correctly discards a lookalike already accumulated inbest, which is what makes the exemption depend on the whole corpus rather than on scan order.- Tests rewritten —
similar_name_clears_any_request_the_corpus_lists,similar_name_clears_a_listed_request_found_after_its_lookalike(thepicomatch/micromatchordering case), andsimilar_name_flags_a_request_the_corpus_does_not_list. All three fail against the previous commit's code, so they pin the new behavior rather than restate it. - Prose updated in the module header,
similar_name_gate, and thefind_similar_package_namedoc comment, including the measured 847 → 0 comparison.
The build.rs comment from the previous review (AUBE_POPULAR_NAMES_RANKED emitted on file existence, before the length check) is unaddressed and still stands.
ℹ️ For the record: a middle ground exists, at a cost you may well reject
Since the previous rule's 847-in-10,000 number is what motivated the switch, it is worth having the third data point on file. Exempting a corpus member unless some candidate outranks it by a large factor — 50× on rank — refuses 1,151 of the 100,000 corpus members (1.15%) while catching all four in-corpus squats I found. That is two orders of magnitude better than the outrank rule and still 1,151 hard non-interactive failures, so choosing 0 over 1,151 is defensible. No change requested; recording it so the decision does not have to be re-derived.
Technical details
# Measured comparison of the three candidate rules
Corpus: https://raw.githubusercontent.com/jdx/aube-primer-packages/main/data/popular.json
(100,000 download-ranked names, the same source `generate-primer.mjs --popular-names-out` writes).
Method: reimplement `comparable_name_parts` + the bounded OSA distance with the same thresholds
(2 when `max(len) >= 5`, else 1) and run every corpus name back through the scan as the request.
| rule | false positives | in-corpus squats caught |
| --- | --- | --- |
| scan whole corpus (pre-PR) | 154 of top 300 | — |
| candidates must outrank the request (`b624a484`) | 5 of top 300, 118 of top 2000, ~9% of ranks 10k–11k | `expres`, `loadash`, `dbug`, `crossenv` |
| clear any corpus member (current) | 0 by construction | none |
| clear unless a candidate outranks by ≥50× | 1,151 of 100,000 (1.15%) | `expres`, `loadash`, `dbug`, `crossenv` |
At 100× the count drops to 742 but `crossenv` is missed. At 20× it is 2,051.
## Open questions for the human
- Is 0 false positives worth exempting every squat that has already accrued traffic, given the
downloads floor demonstrably does not cover them?
- If yes, no code change is needed — only the justification in the doc comment.Claude Opus ｜ 𝕏
| /// | ||
| /// What it gives up is a squat popular enough to be in the corpus itself | ||
| /// (`expres`, #58196). That one has real download volume, which is the OSV | ||
| /// advisory check's and the downloads floor's territory rather than a |
There was a problem hiding this comment.
This is the one claim I could falsify. The default lowDownloadThreshold is 1000 (vendor/aube/crates/aube-settings/settings.toml:583), and every in-corpus squat I checked sits well above it — live last-week counts: expres 6,194, loadash 38,910, dbug 107,160, crossenv 2,219. Top-100,000 membership is a download ranking, so a squat that qualifies for the exemption has by construction cleared the floor; only the OSV MAL-* check is left, and that covers confirmed advisories only.
Technical details
# The downloads floor is not a backstop for an in-corpus squat
## Affected sites
- `vendor/aube/crates/aube/src/commands/add_supply_chain.rs:872-875` — attributes the exempted case to "the OSV advisory check's and the downloads floor's territory". The downloads-floor half is false for every case the exemption creates.
- The PR body carries the same sentence.
- `vendor/aube/crates/aube/src/commands/add_supply_chain.rs:14-20` — the module header lists the three gates in signal order; a reader following it will infer the same false coverage.
## Evidence
`lowDownloadThreshold` default is 1000 (`vendor/aube/crates/aube-settings/settings.toml:583`).
Weekly downloads via `api.npmjs.org/downloads/point/last-week/<pkg>`, the same endpoint the floor
queries: `dbug` 107,160 (rank #25236) · `loadash` 38,910 (#30758) · `expres` 6,194 (#58196) ·
`crossenv` 2,219 (#96565). `crossenv` and `loadash` are the 2017 npm credential-stealing squats.
## Required outcome
- The comment states accurately what covers the exempted case. If OSV is the only remaining
layer, say that — a future maintainer weighing this tradeoff should not be told a gate covers
it when it cannot.
- `expres` is a poorly chosen illustration: at 6,194 weekly downloads it is the weakest of the
four, and it reads as if the exempted set were marginal. `dbug` at 107,160/wk is the honest
example.
|
Shipped in v0.7.0: https://github.com/nubjs/nub/releases/tag/v0.7.0 |
Three failures, one root cause chain. The nightly lockfile-mutation differential has been red since 2026-07-30, and every
Buildleg of the release now dies inbuild.rs. Both trace to the popular-name similarity gate that arrived with the aube v1.35.0 sync (#621).The gate refuses popular packages
find_similar_package_nameskipped only an exact self-match and kept scanning, so being in the popularity corpus was no defence. Measured against the real 100,000-name corpus, that rejects almost everything worth installing — and the name it points at is frequently the squatter:nub add …reactlodashexpressdebugaxiosmslodahsraectaxoisA package among the 100,000 most-downloaded on npm is not a typosquat, so a request the corpus lists at all is now cleared outright instead of being compared against the rest of it. A name the corpus does not list is still scanned against all of it.
The blunt form of that rule is what the precision requires, and the intermediate version is worth recording because it looks sufficient and is not. Restricting candidates to names that outrank the request fixes every row above — but still refuses 847 of the top 10,000, measured by running the gate over the corpus itself:
ws→ms,qs→ms,micromatch→picomatch,safer-buffer→safe-bufferThose are all legitimate packages, and this gate hard-fails a non-interactive install. What the strict rule gives up is a squat popular enough to be in the corpus itself (
expres, #58196) — one with real download volume, which is the OSV advisory check's and the downloads floor's territory rather than a spelling heuristic's.The corpus was not a ranking
The gate reads absence from the corpus as "nobody installs this", which is only true of the real 100,000-name list. When
popular-top100000-v1.jsonis absent,build.rsfalls back to the metadata-primer names — whichwrite_package_blobderives from aBTreeMap, alphabetically, and which number 100 in dev and 2000 in release. There, absence means nothing, and the reported "top-100,000 popularity rank #N" is an alphabetical position.That is how it came to refuse
nub add ms@2.1.3in the nightly mutation harness: in the primer's top 100 sorted alphabetically,qssits at position 70, which is verbatim the "popularity rank #70" in the failing run. Nowbuild.rsreports whether the corpus is genuinely download-ranked, and the gate declines to run when it is not.The corpus was never shipped
Release builds set
AUBE_REQUIRE_PRIMER=1, under whichbuild.rstreats the corpus as mandatory — but nothing produced it. Upstream aube's release workflow passes--popular-names-out; nub's copy was not updated in the sync, so fourBuildlegs of run 30727744051 died with:This was latent until the version-check fix in fdd18fe unblocked the
verifygate that had been failing ahead of it.scripts/remote-build.tsalready records hitting the same panic.mac-build.ymlsets the same flag and is broken identically. Having Node on the runner is not enough:build.rsregenerates the primer by shelling out togenerate-primer.mjswithout--popular-names-out, so nothing there ever writes the corpus. Both workflows now generate it explicitly.Shipping the corpus is what makes the gate live for released binaries for the first time — which is why it lands together with the two fixes above rather than after them.
Verification
Against a binary built from this branch with the unranked corpus, which is what
cargo build -p nub-cliproduces in CI:ms react lodash express debug axios chalkall install, and the exact failing fixturenub add ms@2.1.3exits 0.lodahs reactt expresssare still refused — by the weekly-downloads floor, not the similarity gate. The gates are defence in depth, so declining to run this one on an unranked build does not leave squatters unprotected.Against a binary built with the ranked corpus embedded, so the gate is live: the same popular packages install, and
lodahs reactt expresssare refused by the similarity gate itself with the correct suggestion. Both false-positive counts in the table above are measured by running each rule over the corpus.Also verified: the new corpus step run against the live source writes exactly 100,000 rank-ordered names;
option_env!picks up a conditionalcargo:rustc-envand leaves no stale value when the flag is removed; both workflow files parse.Unit tests cover the exemption, a lookalike appearing earlier in the corpus than the request it must not refuse, the unlisted-request scan, and the ranked-flag invariant.