Skip to content

Commit 35de754

Browse files
Fix npm crawler missing Bun, Deno and Yarn 4 stores (#366, #373, #405, #495) (#496)
* Start fix for #366, #373, #405, #495 Assisted-by: Claude Code:claude-opus-5-5 * Find packages in Bun, Deno and Yarn 4 stores With Bun's isolated linker, Deno's isolated nodeModulesDir or Yarn 4's pnpm linker, a transitive dependency lives only in the package manager's store, which the crawler never looked in. Agent-mode apply and scan reported those packages as not installed and left them unpatched with exit 0, and VEX treated a hosted or vendored Bun pin as having no installed copy, so it attested without checking the bytes. The crawler now walks node_modules/.bun and node_modules/.deno like pnpm's store, in scan, apply's resolver and the peer-copy fan-out. In Yarn 4's node_modules/.store it finds each package at the entry's package/ dir, which the entry's own node_modules link points to. Assisted-by: Claude Code:claude-opus-5-5 * Test agent apply on a Yarn 4 pnpm-linker store Installs is-odd with the real yarn 4 pnpm linker, so is-number lives only in node_modules/.store, then checks agent-mode apply patches it, is-odd loads the patched copy, and rollback restores it. Assisted-by: Claude Code:claude-opus-5-5 * Test VEX judges a hosted Bun pin by its store copy With bun.lock pinning the hosted tarball and the package installed only in node_modules/.bun, a pristine store copy must be reported not_applied, and a patched one attested. Before the crawler fix VEX saw no installed copy and attested the pristine one. Assisted-by: Claude Code:claude-opus-5-5 * Test agent apply in Bun and Deno isolated stores A real bun install with linker = "isolated" puts is-number only in node_modules/.bun; apply must patch it and is-odd must load the patched copy. A Deno-shaped node_modules/.deno tree gets the same check without needing deno installed. Assisted-by: Claude Code:claude-opus-5-5 * Document the Bun, Deno and Yarn 4 store walks Assisted-by: Claude Code:claude-opus-5-5 * Drop unrelated rustfmt churn from this PR b5985fc ran cargo fmt over the whole workspace, so 124 files outside the crawler fix changed formatting only. That buried the real change for reviewers and invites conflicts with every other open PR. Restore those files to their main versions; npm_crawler.rs keeps its fix. Co-Authored-By: Claude <noreply@anthropic.com> * Resolve bundled copies inside a Yarn 4 store package A Yarn 4 pnpm-linker store entry keeps its package at <entry>/package and links <entry>/node_modules/<name> to it. The scan descends into that package's own node_modules through the link, but the resolver never follows links, so it missed bundled dependencies there. Apply then patched the regular store copy and reported success while Node kept loading the unpatched bundled copy. When a store entry links to its own package dir, the resolver now also enqueues that dir's node_modules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2DjA5mBwq29D5rrFa5Fvd * Compare the loaded bundled copy by canonical path The bundled-copy e2e test matched Node's resolved path against a forward-slash substring, which fails on Windows where Node prints backslashes. Compare the canonical path of what `require('parent')` resolves to with the bundled copy's instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2DjA5mBwq29D5rrFa5Fvd --------- Signed-off-by: Mikola Lysenko <mikolalysenko@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
1 parent a79de97 commit 35de754

5 files changed

Lines changed: 1053 additions & 52 deletions

File tree

‎crates/socket-patch-cli/tests/e2e_vex_redirect.rs‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2162,6 +2162,69 @@ fn vlt_redirect_ledger_is_judged_by_the_vlt_store_copy_while_the_lock_pins_it()
21622162
}
21632163
}
21642164

2165+
/// #405: Bun's isolated linker installs a transitive package only in its
2166+
/// store (`node_modules/.bun/<entry>/node_modules/<name>`; the entry is
2167+
/// `<name>@<version>` for a stale pre-reinstall tree, a mangled URL for a
2168+
/// fresh install of the hosted tarball). While `bun.lock` pins the hosted
2169+
/// tarball, that store copy is the evidence: a pristine copy is unpatched
2170+
/// code (`not_applied`), never a "nothing installed" lockfile attestation.
2171+
#[test]
2172+
fn bun_hosted_ref_is_judged_by_the_bun_store_copy() {
2173+
let (pristine, patched) = (
2174+
&b"module.exports = 'pristine'\n"[..],
2175+
&b"module.exports = 'patched'\n"[..],
2176+
);
2177+
let purl = "pkg:npm/left-pad@1.3.0";
2178+
let url = hosted_npm_url("left-pad", "1.3.0", UUID);
2179+
let entries = [
2180+
"left-pad@1.3.0".to_string(),
2181+
format!("left-pad@{}", url.replace([':', '/'], "+")),
2182+
];
2183+
for entry in entries {
2184+
let tmp = tempfile::tempdir().unwrap();
2185+
let cwd = tmp.path();
2186+
put(
2187+
cwd,
2188+
"package.json",
2189+
br#"{ "name": "app", "version": "1.0.0", "dependencies": { "dep": "1.0.0" } }"#,
2190+
);
2191+
put(
2192+
cwd,
2193+
"bun.lock",
2194+
format!(
2195+
"{{\n \"lockfileVersion\": 1,\n \"workspaces\": {{\n \"\": {{\n \
2196+
\"name\": \"app\",\n \"dependencies\": {{\n \"dep\": \"1.0.0\",\n \
2197+
}},\n }},\n }},\n \"packages\": {{\n \
2198+
\"dep\": [\"dep@1.0.0\", \"\", {{ \"dependencies\": {{ \"left-pad\": \"1.3.0\" }} }}, \
2199+
\"sha512-{dep}==\"],\n\n \
2200+
\"left-pad\": [\"left-pad@{url}\", {{}}, \"{SRI}\"],\n }}\n}}\n",
2201+
dep = "D".repeat(86),
2202+
)
2203+
.as_bytes(),
2204+
);
2205+
let store = format!("node_modules/.bun/{entry}/node_modules/left-pad");
2206+
put(
2207+
cwd,
2208+
&format!("{store}/package.json"),
2209+
br#"{ "name": "left-pad", "version": "1.3.0" }"#,
2210+
);
2211+
put(cwd, &format!("{store}/index.js"), pristine);
2212+
let (_rt, server) = serve_patch_views(vec![(
2213+
UUID.to_string(),
2214+
one_file_view(UUID, purl, "package/index.js", pristine, patched),
2215+
)]);
2216+
let args = ["--proxy-url", &server.uri()];
2217+
2218+
let (code, env) = vex_json(cwd, &args);
2219+
assert_eq!(code, Some(1), "{entry}: a pristine store copy: {env}");
2220+
assert_eq!(skipped_reason(&env, purl), "not_applied", "{entry}: {env}");
2221+
2222+
put(cwd, &format!("{store}/index.js"), patched);
2223+
let (code, env) = vex_json(cwd, &args);
2224+
assert_attested(cwd, code, &env, UUID, "the store copy verifies");
2225+
}
2226+
}
2227+
21652228
/// The patch view for `name@version` (the [`left_pad_view`] shape).
21662229
fn npm_view(name: &str, version: &str, after_hash: &str) -> Value {
21672230
let mut view = left_pad_view(after_hash);

‎crates/socket-patch-cli/tests/e2e_yarn4_pnpm_linker_build.rs‎

Lines changed: 323 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -803,3 +803,326 @@ fn yarn4_pnpm_linker_vendor_fresh_checkout_installs_patched_bytes_and_reverts()
803803
);
804804
eprintln!("REVERT OK");
805805
}
806+
807+
// ── agent-mode transitive capstone (#495) ─────────────────────────────
808+
809+
/// #495: under the pnpm linker a TRANSITIVE dependency exists only at
810+
/// `node_modules/.store/<slug>-npm-<v>-<hash>/package`, reachable through
811+
/// the entry's own `node_modules/<name> -> ../package` link (yarn 4; yarn 3
812+
/// wrote a real dir there). Agent-mode `apply` must find and patch that
813+
/// copy, the code yarn's runtime loads must carry the patch, and
814+
/// `rollback` must restore it.
815+
#[test]
816+
fn yarn4_pnpm_linker_agent_apply_patches_transitive_store_copy() {
817+
if !has_corepack_pm(yarn_berry()) {
818+
skip!(
819+
"SKIP e2e_yarn4_pnpm_linker_build (agent transitive): `corepack {}` unavailable",
820+
yarn_berry()
821+
);
822+
return;
823+
}
824+
825+
let tmp = tempfile::tempdir().unwrap();
826+
let proj = tmp.path().join("proj");
827+
std::fs::create_dir_all(&proj).unwrap();
828+
std::fs::write(
829+
proj.join("package.json"),
830+
r#"{"name":"yarn4-pnpm-linker-transitive","version":"0.0.0","private":true,"dependencies":{"is-odd":"3.0.1"}}"#,
831+
)
832+
.unwrap();
833+
std::fs::write(proj.join(".yarnrc.yml"), YARNRC_PNPM).unwrap();
834+
let global = tmp.path().join("yarn-global");
835+
let install = corepack(
836+
&proj,
837+
yarn_berry(),
838+
&["install"],
839+
&[("YARN_GLOBAL_FOLDER", global.to_str().unwrap())],
840+
);
841+
if !install.status.success() {
842+
skip!(
843+
"SKIP e2e_yarn4_pnpm_linker_build (agent transitive): fixture `yarn install` \
844+
failed (registry unreachable?):\n{}",
845+
yarn_berry_common::yarn_output(&install)
846+
);
847+
return;
848+
}
849+
850+
// The layout under test: no importer-level `is-number`, one store
851+
// entry whose package dir is the only physical copy.
852+
let nm = proj.join("node_modules");
853+
assert!(
854+
std::fs::symlink_metadata(nm.join("is-number")).is_err(),
855+
"is-number must be transitive-only (not linked at the importer root)"
856+
);
857+
let entry = std::fs::read_dir(nm.join(".store"))
858+
.unwrap()
859+
.map(|e| e.unwrap().path())
860+
.find(|p| {
861+
p.file_name()
862+
.and_then(|n| n.to_str())
863+
.is_some_and(|n| n.starts_with("is-number-npm-6.0.0-"))
864+
})
865+
.expect(".store must hold an is-number-npm-6.0.0 entry");
866+
let index = entry.join("package").join("index.js");
867+
let orig = std::fs::read(&index).expect("store copy of is-number/index.js");
868+
assert!(!orig.starts_with(MARKER.as_bytes()));
869+
let patched: Vec<u8> = [MARKER.as_bytes(), orig.as_slice()].concat();
870+
stage_patch(&proj, "pkg:npm/is-number@6.0.0", &orig, &patched);
871+
// The before blob too, so the offline rollback can restore.
872+
std::fs::write(proj.join(".socket/blobs").join(git_sha256(&orig)), &orig).unwrap();
873+
874+
let (code, stdout, stderr) = run_socket(
875+
&proj,
876+
&[
877+
"apply",
878+
"--json",
879+
"--offline",
880+
"--cwd",
881+
proj.to_str().unwrap(),
882+
],
883+
);
884+
assert_eq!(
885+
code, 0,
886+
"apply failed.\nstdout:\n{stdout}\nstderr:\n{stderr}"
887+
);
888+
let env: serde_json::Value = serde_json::from_str(&stdout)
889+
.unwrap_or_else(|e| panic!("apply --json output is not JSON: {e}\nstdout:\n{stdout}"));
890+
assert_eq!(env["status"], "success", "envelope: {env}");
891+
assert_eq!(
892+
env["summary"]["applied"], 1,
893+
"transitive copy applied: {env}"
894+
);
895+
assert_eq!(
896+
std::fs::read(&index).unwrap(),
897+
patched,
898+
"store copy patched"
899+
);
900+
901+
// RUNTIME PROOF: is-odd's own require of is-number loads the patch.
902+
let resolve = "process.stdout.write(require('fs').readFileSync(require.resolve('is-number', \
903+
{paths: [require('path').dirname(require.resolve('is-odd'))]})))";
904+
let out = corepack(&proj, yarn_berry(), &["node", "-e", resolve], &[]);
905+
assert!(
906+
out.status.success(),
907+
"`yarn node` failed:\n{}",
908+
yarn_berry_common::yarn_output(&out)
909+
);
910+
assert_eq!(
911+
out.stdout, patched,
912+
"is-odd must load the patched is-number"
913+
);
914+
915+
let (code, stdout, stderr) = run_socket(
916+
&proj,
917+
&[
918+
"rollback",
919+
"--json",
920+
"--offline",
921+
"--cwd",
922+
proj.to_str().unwrap(),
923+
],
924+
);
925+
assert_eq!(
926+
code, 0,
927+
"rollback failed.\nstdout:\n{stdout}\nstderr:\n{stderr}"
928+
);
929+
assert_eq!(
930+
std::fs::read(&index).unwrap(),
931+
orig,
932+
"rollback restores the store copy"
933+
);
934+
}
935+
936+
/// #496 review: a package bundling is-number@7.0.0 lands in the pnpm
937+
/// linker's store as `.store/parent-…/package/node_modules/is-number`,
938+
/// beside the regular `.store/is-number-npm-7.0.0-…/package`, and Node
939+
/// loads the BUNDLED copy for `parent`. That copy is reachable only
940+
/// through the entry's `node_modules/parent -> ../package` link, so apply
941+
/// must still find it: both copies get patched, `parent` loads the patch,
942+
/// and rollback restores both.
943+
#[test]
944+
fn yarn4_pnpm_linker_agent_apply_patches_bundled_copy_inside_store_package() {
945+
if !has_corepack_pm(yarn_berry()) || !has_command("tar") {
946+
skip!(
947+
"SKIP e2e_yarn4_pnpm_linker_build (agent bundled): `corepack {}` or `tar` unavailable",
948+
yarn_berry()
949+
);
950+
return;
951+
}
952+
953+
let tmp = tempfile::tempdir().unwrap();
954+
let proj = tmp.path().join("proj");
955+
std::fs::create_dir_all(&proj).unwrap();
956+
std::fs::write(proj.join(".yarnrc.yml"), YARNRC_PNPM).unwrap();
957+
let global = tmp.path().join("yarn-global");
958+
let install = |manifest: &str| {
959+
std::fs::write(proj.join("package.json"), manifest).unwrap();
960+
corepack(
961+
&proj,
962+
yarn_berry(),
963+
&["install"],
964+
&[("YARN_GLOBAL_FOLDER", global.to_str().unwrap())],
965+
)
966+
};
967+
968+
// The registry copy first, so the bundled one can be byte-identical:
969+
// one patch's before-hashes then fit both.
970+
let out = install(
971+
r#"{"name":"yarn4-pnpm-linker-bundled","version":"0.0.0","private":true,"dependencies":{"is-number":"7.0.0"}}"#,
972+
);
973+
if !out.status.success() {
974+
skip!(
975+
"SKIP e2e_yarn4_pnpm_linker_build (agent bundled): fixture `yarn install` \
976+
failed (registry unreachable?):\n{}",
977+
yarn_berry_common::yarn_output(&out)
978+
);
979+
return;
980+
}
981+
let nm = proj.join("node_modules");
982+
let registry_copy = nm.join("is-number");
983+
984+
let stage = tmp.path().join("parent-stage").join("package");
985+
std::fs::create_dir_all(&stage).unwrap();
986+
std::fs::write(
987+
stage.join("package.json"),
988+
r#"{"name":"parent","version":"1.0.0","main":"index.js","bundleDependencies":["is-number"],"dependencies":{"is-number":"7.0.0"}}"#,
989+
)
990+
.unwrap();
991+
std::fs::write(
992+
stage.join("index.js"),
993+
"module.exports = require.resolve('is-number');\n",
994+
)
995+
.unwrap();
996+
copy_dir_recursive(
997+
&registry_copy,
998+
&stage.join("node_modules").join("is-number"),
999+
);
1000+
let tgz = proj.join("parent-1.0.0.tgz");
1001+
let tar = Command::new("tar")
1002+
.args(["-czf", tgz.to_str().unwrap(), "package"])
1003+
.current_dir(stage.parent().unwrap())
1004+
.output()
1005+
.expect("failed to run tar");
1006+
assert!(
1007+
tar.status.success(),
1008+
"tar: {}",
1009+
String::from_utf8_lossy(&tar.stderr)
1010+
);
1011+
1012+
let out = install(
1013+
r#"{"name":"yarn4-pnpm-linker-bundled","version":"0.0.0","private":true,"dependencies":{"is-number":"7.0.0","parent":"file:./parent-1.0.0.tgz"}}"#,
1014+
);
1015+
assert!(
1016+
out.status.success(),
1017+
"`yarn install` with the bundling tarball failed:\n{}",
1018+
yarn_berry_common::yarn_output(&out)
1019+
);
1020+
1021+
// The layout under test: two physical is-number@7.0.0 copies in the
1022+
// store, the bundled one inside parent's `package` dir.
1023+
let store_entry = |prefix: &str| {
1024+
std::fs::read_dir(nm.join(".store"))
1025+
.unwrap()
1026+
.map(|e| e.unwrap().path())
1027+
.find(|p| {
1028+
p.file_name()
1029+
.and_then(|n| n.to_str())
1030+
.is_some_and(|n| n.starts_with(prefix))
1031+
})
1032+
.unwrap_or_else(|| panic!(".store must hold a {prefix}* entry"))
1033+
};
1034+
let regular_index = store_entry("is-number-npm-7.0.0-").join("package/index.js");
1035+
let bundled_index = store_entry("parent-").join("package/node_modules/is-number/index.js");
1036+
let orig = std::fs::read(&regular_index).expect("regular store copy of is-number/index.js");
1037+
assert_eq!(
1038+
std::fs::read(&bundled_index).expect("bundled copy inside parent's package dir"),
1039+
orig,
1040+
"the bundled copy is byte-identical to the registry copy"
1041+
);
1042+
1043+
// Which copy Node loads for `parent`: the bundled one.
1044+
let load = "process.stdout.write(require('fs').readFileSync(require('parent')))";
1045+
let loaded_path = corepack(
1046+
&proj,
1047+
yarn_berry(),
1048+
&["node", "-p", "require('parent')"],
1049+
&[],
1050+
);
1051+
// Compared as canonical paths: Node prints the platform's separators.
1052+
let loaded = PathBuf::from(String::from_utf8_lossy(&loaded_path.stdout).trim());
1053+
assert_eq!(
1054+
std::fs::canonicalize(&loaded).ok(),
1055+
std::fs::canonicalize(&bundled_index).ok(),
1056+
"parent must load its bundled is-number:\n{}",
1057+
yarn_berry_common::yarn_output(&loaded_path)
1058+
);
1059+
1060+
let patched: Vec<u8> = [MARKER.as_bytes(), orig.as_slice()].concat();
1061+
stage_patch(&proj, "pkg:npm/is-number@7.0.0", &orig, &patched);
1062+
std::fs::write(proj.join(".socket/blobs").join(git_sha256(&orig)), &orig).unwrap();
1063+
1064+
let (code, stdout, stderr) = run_socket(
1065+
&proj,
1066+
&[
1067+
"apply",
1068+
"--json",
1069+
"--offline",
1070+
"--cwd",
1071+
proj.to_str().unwrap(),
1072+
],
1073+
);
1074+
assert_eq!(
1075+
code, 0,
1076+
"apply failed.\nstdout:\n{stdout}\nstderr:\n{stderr}"
1077+
);
1078+
let env: serde_json::Value = serde_json::from_str(&stdout)
1079+
.unwrap_or_else(|e| panic!("apply --json output is not JSON: {e}\nstdout:\n{stdout}"));
1080+
assert_eq!(env["status"], "success", "envelope: {env}");
1081+
assert_eq!(
1082+
std::fs::read(&regular_index).unwrap(),
1083+
patched,
1084+
"regular copy patched"
1085+
);
1086+
assert_eq!(
1087+
std::fs::read(&bundled_index).unwrap(),
1088+
patched,
1089+
"bundled copy patched"
1090+
);
1091+
1092+
// RUNTIME PROOF: the copy `parent` actually loads carries the patch.
1093+
let out = corepack(&proj, yarn_berry(), &["node", "-e", load], &[]);
1094+
assert!(
1095+
out.status.success(),
1096+
"`yarn node` failed:\n{}",
1097+
yarn_berry_common::yarn_output(&out)
1098+
);
1099+
assert_eq!(
1100+
out.stdout, patched,
1101+
"parent must load the patched is-number"
1102+
);
1103+
1104+
let (code, stdout, stderr) = run_socket(
1105+
&proj,
1106+
&[
1107+
"rollback",
1108+
"--json",
1109+
"--offline",
1110+
"--cwd",
1111+
proj.to_str().unwrap(),
1112+
],
1113+
);
1114+
assert_eq!(
1115+
code, 0,
1116+
"rollback failed.\nstdout:\n{stdout}\nstderr:\n{stderr}"
1117+
);
1118+
assert_eq!(
1119+
std::fs::read(&regular_index).unwrap(),
1120+
orig,
1121+
"rollback restores the regular copy"
1122+
);
1123+
assert_eq!(
1124+
std::fs::read(&bundled_index).unwrap(),
1125+
orig,
1126+
"rollback restores the bundled copy"
1127+
);
1128+
}

0 commit comments

Comments
 (0)