Repository navigation
R2 v8.85.0: indexing and defaults the host can read (#1038) - #1039
Merged
Merged
Conversation
…y in explain (closes #876) A project that ships a bundle as its bin has that file pulled into the index as a declared entrypoint, although srcDirs never reached it. On this repository that was gen-context.js, a 42,000-line bundle: every src/ function was listed twice, the budget paid for both, and the changes and todos sections reported the bundle's copies. generatedReason(path, head, tail) in src/util/file-class.js reads four independent signals, none of them a size guess: a build-output directory or a .generated. / .pb. name; a comment in the first lines that OPENS with Generated / @generated / auto-generated, says DO NOT EDIT, or names a generated bundle; a bundler's or compiler's own runtime helpers at the top (webpack, esbuild, parcel, __esModule); and a trailing sourceMappingURL. declaredEntrypoints judges each declared path before the already-walked test, so a bundle is also left out when srcDirs is the whole repository, and it now honours .contextignore. Run over the 8,596 JS and TS files in the 54 benchmark clones (outside build directories) the signals flag 34 files, and the first version of the header rule flagged one that is hand-written: a wrapped JSDoc line that began with a lowercase "generated JS client)". A bare "generated" must now open a sentence. The remaining flagged files are generated: fastify's validators, svelte's message tables, excalidraw's wasm glue and fonts, source-map fixtures, esbuild and parcel bundles. The MCP explain_file tool says "looks generated" for such a file where it said "Run: node gen-context.js", advice that cannot help. sigmap explain gains the verdict: "EXCLUDED: generated entrypoint declared in package.json (<reason>)", and a hand-written declared entrypoint outside srcDirs is no longer called "not under any srcDir" while being indexed. Measured against untouched develop in a clean worktree, same layout, run-to-run noise 0: hard 62/90, mined 39/60, easy 18/20, jvm 41.0% and xrepo 115/208 are unchanged, no hit status flips, nine tasks change rank (all deep misses). The one task that leaves the index (easy t006, which expects gen-context.js) was a miss at rank 175. This repository's own run reads 213 files instead of 214 and 432,315 raw tokens instead of 969,825, so its token reduction reads 96%, not 98%. The hand-written CLI core inside gen-context.js leaves the index with the bundle. refs #1038
…ads, and say what mcp install advertises (refs #995) Codex loads at most project_doc_max_bytes of an AGENTS.md (32 KiB by default) and cuts the rest without a word; Claude Code asks for a short CLAUDE.md. The codex and claude outputs wrote their whole map: 43 KB on gin, 887 lines of CLAUDE.md. src/util/host-limits.js names each limit and where it comes from: Codex's own config.toml (CODEX_HOME honoured) else 32 KiB; 200 lines for Claude Code; the new codexMaxBytes / claudeMaxLines config keys override either, and 0 turns a cap off. The core measures the file as it will be WRITTEN, the lines a person wrote above SigMap's block included, and re-sizes the same entries with the existing token budget until it fits, so the lowest-priority files go first (tests, config, generated, least recent, fewest signatures) and the artifact says how many it left out. The optional sections (deps, versions, todos, changes) are dropped for that one file only when the limit is tighter than they are, and a limit the human lines already exceed is said and the block shrinks to its fixed header. A first version of the search overshot below the fit (60 lines kept 0 files while using 31); it now reaches up after every fit and the same limits keep 2, 7 and 15 files. Every other output, and every strategy but the single budgeted file, is unchanged; the retrieval index still holds every file. packages/adapters/codex.js gains a pure place() so the finished file can be measured before it is written. sigmap doctor gains an "Instruction files" row: each file's size against its limit, a warning when one is over and what the host does with the rest. AGENTS.md leaves package.json "files": no host reads a package's own AGENTS.md inside node_modules (225 files, 1425 KB packed). CHANGELOG.md stays for the release-hygiene issue (#999). The MCP default is documented, not changed: a plain `sigmap mcp install claude` registers the server and writes nothing into the repository, so it advertises all 22 tools; `sigmap --init` is what writes the 7-tool core profile. One test installs into a fresh repository, starts the server it registered and pins the count against the docs. Measured against untouched develop in a clean worktree, same layout, run-to-run noise 0: xrepo is unchanged at 115/208. The self-scored splits moved with the index, not with any ranking code (this commit adds one indexed source file and edits two): hard 62 -> 63 of 90 (h062 7 -> 5), mined 39 -> 40 of 60 (m003 6 -> 3 and m053 6 -> 5 gain, m006 5 -> 7 loses), easy and jvm unchanged. That is drift, to be read as such. The new tests fail on develop's code (8 of 13, with only the new module supplied); the two limit-resolution tests and three guards pass there by construction. Item 4 (the default content waits for the agent-level A/B, #974) is not done. refs #1038
…defaults and CommonJS methods, and say what an unread file held (closes #996) The TypeScript and JavaScript extractors dropped four common shapes without a word, and `sigmap explain` then gave a remedy that could not apply. In zod 33% of the top-level exports carry a `$`. Read now: a `$` in a name (const, function, interface, type, class); generic arrows and arrows behind a type annotation that itself contains `=>`; `export const x = make(...)` as a const entry, name and anchor, no arguments; `export default function () {}` and `export default (a) =>`; and CommonJS assignment methods (`app.use = function use(fn)`, `exports.x`, `module.exports.x`) on the module's exported object only. A statement that follows `;` or `}` on the same line is read too, as the issue's own one-line fixture needs. The class-body scan took the `=` in a generic default (`class ZodError<T = any>`) for an assignment and gave up; found by the new recall row, fixed here. The shapes read since this change are listed AFTER the ones always read (late-last.js). A first version interleaved them in file order and `function` recall fell from 87.0% to 81.5%: zod's api.ts went from 116 entries to a full 200 and lost 43 functions to the cap. Now nothing previously listed can be pushed past the cap and a file with none of the new shapes is byte-identical. Checked over the 8,610 JavaScript and TypeScript files of the 54 pinned clones and this repository's sources, old extractors against new: 335 files gain 1,377 signatures, and no signature that was listed before is missing (the only differences are the count in a "+N more" notice). Time +9%. A run now prints how many JavaScript and TypeScript files declare exports and list none, and the first few paths (barrels, declaration files, tests and generated files are not counted). `explain` shows the export lines it saw and says the gap is the extractor's, not the configuration's, instead of "check that the file contains function/class definitions". The extraction-recall harness gains TypeScript, measured against the compiler's own parse (typescript is not a dependency: --typescript, SIGMAP_TYPESCRIPT or a package that resolves; otherwise unmeasured, and no floors, as for C and C++). 326 files of zod, vue-core, excalidraw and vitepress, extractor population, before -> after: function 87.0 -> 87.2%, class 40.8 -> 49.3%, interface 48.4 -> 75.0%, type 61.4 -> 76.7%, arrow 74.3 -> 89.4%, factory 0 -> 38.6%, anonymous default 0 -> 100%, `$` names 0 -> 57.8%. The shipped index holds 284 of the 326 files where it held 227. What is still missing is mostly the 200-signature cap: 216 of 223 factory misses and 211 of 218 `$` misses are in four zod files that hold 283 to 768 entries each. zod's core/checks.ts goes 0 -> 200 signatures and core/parse.ts 0 -> 37. The saved extraction-recall report reproduces Python, Ruby and C++ leaf for leaf and adds TypeScript. Measured against the previous commit in a clean worktree, same layout: no hit flips in hard, mined, easy or jvm (16 small rank shifts); xrepo 115 -> 116 of 208, zod x202 from a miss to rank 4, x021 3 -> 4, x037 1 -> 2, x039 4 -> 2, x044 2 -> 1 and x070 3 -> 4 move within the top 5, nothing lost. The xrepo baseline is re-recorded once, after the last commit of this change. refs #1038
… roots until the code is covered, and say how much of the repository they cover (closes #997) Detection counted a candidate's code two levels deep and kept six roots. A layout that keeps its code deeper (Rails: <gem>/lib/<gem>/..., three levels down) scored 0 for most directories, and a first run indexed 42% of the code (659 of 1,578 files in the pinned clone) while the summary read "Indexed 100%". - scorer/resolver: count a candidate's code to eight levels; order roots by the code they hold (then by name); past six roots, add the next-ranked candidates until the roots hold 95% of the candidates' code, at most 40. A name-penalised directory (docs, examples, tests, documentation-*), a dot directory and a candidate that holds other candidates (packages/ over packages/a, src/ over src/test/java) keep the old two-level count: the deeper count lifted each of them above zero and into the roots (flask took examples, libuv docs, exposed documentation-website, rust-analyzer .github, a JS monorepo collapsed to packages/, and a lone src/test/java to src). - coverage-score: repositoryCoverage / formatRepositoryCoverage, the one sentence "N% of the repository's code (X of Y files) is under the source roots", printed identically by generate (with the roots, and a warning naming the largest directories left out under 80%), validate (and its --json), doctor (config row turns to a warning) and roots (with each root's file count and how many scored candidates the cap left out). Measured against a control worktree at the previous commit (reach = files under the roots / files present, source-root gate): rails 659 -> 1,509 of 1,578 (41.8 -> 95.6%), kotlinx-coroutines 42 -> 383 of 401, akka 947 -> 1,170 of 1,240, upickle 31 -> 91, shiny 180 -> 293, dio 74 -> 100, excalidraw 501 -> 518, sidekiq 105 -> 113, rust-analyzer 1,343 -> 1,387, okhttp 290 -> 296, clap 120 -> 124; tokio 481 -> 478, zod 218 -> 217 (the sixth root is a different directory). xrepo 116 -> 118 of 208: Rails x084 and x087 become hits (its four unreachable tasks, none now), no hit is lost; x089 4 -> 2, x093 1 -> 2, x161 4 -> 5 within the top 5. Baselines moved, through the gates' own writers or entry by entry: xrepo (--save: 115 -> 118, which also records zod x202 from the previous commit), source-root (the 11 entries this moved), signal-arms (--only xrepo), then latest.json / README / version.json and the guide tables a test pins to them. Two existing fixtures changed because they encoded the old behaviour: the MAX_ROOTS fixture (eight one-file directories are now eight roots; the cap is tested where six roots hold the code) and the gate's collapse fixture (code buried five levels deep is now reached; it is buried ten deep).
…ile stalled CI (refs #996, #1038) The #996 commit asked "does `export` begin a statement" with `(?<=[;}][ \t]*)` in ten regexes. V8 up to Node 22 tries a variable-length lookbehind at every position and walks back over the whole run of blanks behind it, and a string literal is masked to blanks before the scan: excalidraw's 790 KB base64 wasm module took minutes, the xrepo job's warm-cache step timed out after 45 minutes, and the gate step before it took 22. Node 25 optimises the lookbehind away, which is why a laptop never showed it. `atStatementStart(text, index)` looks back from a match instead: one short walk per `export`. On Node 22: 200 000 blanks 171 s -> 35 ms, 1 000 000 blanks 148 ms. The same ten shapes are read; the JavaScript copies also stop accepting a literal `t` (their class was written `[ \\t]`) and accept a tab. Tests: no lookbehind in either extractor (the mechanism, so a Node 25 clock cannot hide it), a 1 MB literal finishes and what follows it is read, atStatementStart's cases, and same-line exports after `;` or `}` are still read.
This was referenced Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking: #1038 (R2, v8.85.0 — "Indexing and defaults the host can read"). One commit per sub-issue. No version bump and no CHANGELOG entry: those belong to
/update-docs.What changed
d8e4839bin/mainthat is generated (bundle header,@generated, build-output directory, source-map trailer) is skipped by the index, andexplainsays whyf06e2c1codexAGENTS.md is sized toproject_doc_max_bytes(32 KiB, or the host's ownconfig.tomlvalue,0= off);claudeCLAUDE.md to about 200 lines, with the lines a person wrote above SigMap's block intact and a disclosure when those alone exceed the limit;doctorreports each instruction file against its limit;mcp installbehaviour is unchanged and the docs now state the default (all 22 tools,coreis something--initwrites)8edba8f$names, generic and wrapped consts, anonymous default exports and CommonJS methods; a file whose exports it could not read is said in the summary and inexplain; a TypeScript adapter joins the extraction-recall harness (zod, three vue-core packages, excalidraw, vitepress)eb0ec9806ee6daexportbegin a statement" with a variable-length lookbehind, which Node 22 runs quadratically over a run of blanks, and excalidraw's 790 KB base64 wasm file is one. Replaced by a look-back from each match (src/util/statement-start.js)Measurements (each against a control worktree at the previous commit, noise floor 0)
#997 — source-root gate (files under the roots / files present): rails 659 → 1,509 of 1,578 (41.8% → 95.6%), kotlinx-coroutines 42 → 383 of 401, akka 947 → 1,170, upickle 31 → 91, shiny 180 → 293, dio 74 → 100, excalidraw 501 → 518, sidekiq 105 → 113, rust-analyzer 1,343 → 1,387, okhttp 290 → 296, clap 120 → 124. Lower: tokio 481 → 478 and zod 218 → 217 (the sixth root is a different directory). Everything else in the 41-repo gate is unchanged.
xrepo (zero-config, 208 tasks): 116 → 118. Rails x084 and x087 become hits (its four unreachable tasks: none now; x085 and x088 are reachable and miss). No hit is lost. Rank moves inside the top 5: x089 4 → 2, x093 1 → 2, x161 4 → 5. Unreachable tasks 13 → 8.
Retrieval gate on the clean committed tree, against a clean control: hard 63/90 (70.0%), mined 40/60, easy 18/20, jvm 25/61 — identical hit/miss on every task, only the rank of some misses moved (h036 3 → 4 stays a hit). The hard floor is 0.68; it reads 70.0%.
Earlier commits: #876 no flips (hard 62, mined 39, xrepo 115 on the tree then); #995 xrepo 115; #996 xrepo 116 (+zod x202).
Baselines moved (said out loud)
benchmarks/xrepo-baseline.json— through the gate's own--save: 115 → 118. The save also records zod x202 from the typescript:$names, generic arrows, anonymous default exports andobj.x = functionmethods are not indexed, andexplaingives the wrong fix #996 commit, which was never saved.benchmarks/source-root-baseline.json— edited entry by entry: the 11 entries this change moved (akka, clap, dio, kotlinx-coroutines, okhttp, rails, rust-analyzer, shiny, sidekiq, upickle, zod). Entries the change did not move are left as they were, including some that already differ from develop.benchmarks/reports/signal-arms.json—run-signal-verdicts.mjs --only xrepo --save; the other five corpora stay as recorded.benchmarks/latest.json,README.md,version.jsonthroughgen-benchmark-latest.mjsandsync-metrics.mjs; the third-party headline in the README and on the comparison page moves from 55.3% (115) to 56.7% (118), and the guide tables a test pins to the baseline moved with it.Decisions and deviations
examplesand lost x032, libuv tookdocsand lost x094, exposed tookdocumentation-websiteand lost x120), made a candidate that holds other candidates beat its members in the parent-wins dedupe (a JS monorepo resolved topackages/, a lonesrc/test/javatosrc), and made rust-analyzer's.githuba root. Name-penalised, docs-like, container and dot directories now keep the two-level count they were scored on before. Pinned by new tests (some of them are guards that also pass on the old code — they exist for this change's side effects).v650-source-root-resolvercapped eight one-file directories at six (they are now eight roots; the cap is tested where six roots hold the code, and the extension past six has its own test), andsource-root-gate's collapse fixture buried code five levels deep (it is reached now; it is buried ten deep).mcp installwrites no config key and advertises all 22 tools; a test holdsdocs-vp/guide/mcp.mdand the README to the count.sigmap roots --explainshows the number of candidates still left out (omitted).Not in this PR, and open calls
AGENTS.mdin the npm tarball overlaps with release: 9 of the last 59 npm releases changed no runtime file, CHANGELOG.md is 16% of every tarball, and CONTRIBUTING's test counts are stale #999./update-docs, say so and I will split it out.gen-context.jsis a generated bundle, so it is skipped by the index and its hand-written CLI core left the index with it, and the repository's own token reduction reads 96% because the denominator shrank; tokio and zod lose a file or three of reach.06ee6dafixes it. The xrepo job's warm-cache step timed out after 45 minutes on excalidraw'sharfbuzz-wasm.ts(a 787,586-character line, masked to blanks before the scan), and the gate step before it took 22 minutes. Reproduced innode:22-bookworm-slim: the old extractor took 171 s on 200,000 blanks (quadratic; 1,000,000 would take hours), the fixed one 35 ms (148 ms at 1,000,000). Node 25, which runs here, optimises the lookbehind away, so the new tests read the mechanism (no lookbehind in either extractor) and the clock (a 1 MB literal finishes) rather than the clock alone. Rest of the change on Node 22 in Docker:validate:warm-cacheover all 54 clones 54/54 equivalent, and no TypeScript/JavaScript file of the 8,798 in the clones takes over 500 ms. Measured after the fix on the clean tree: xrepo 118/208 and--no-regresspasses; retrieval gate status identical on all 170 tasks (hard 70.0%, mined 66.7%, easy 90.0%, jvm 41.0%; onlyh089's rank moved, 5 → 4, from the corpus gaining a file). Not run: the other Linux-only code paths (the earlier CI failure on R1 (v8.84.0): verify says what it checked; the map no longer depends on the checkout path #1035 was a clock, this one a regex engine).Ruby's
(?<!\bdef\s+)insrc/extractors/ruby.js:241is also a variable-length lookbehind (older than this PR, not touched); I did not test it on Node 22.Verification
node test/integration/all.js: 272 passed, 0 failed, before and after06ee6da(new:host-readable-defaults,ts-declaration-shapes,generated-entrypoint,source-root-coverage; extended:extraction-recall).execSync/shell spawns in the published surface, no install scripts,mainexportsextract, rank, buildSigIndex, scan, score, adapt, no host fingerprinting, tarball 226 files.build-bundle --check,check-version-meta,validate:source-roots,validate:xrepo --no-regressall pass.