Deterministic intelligence: no LLM, packaged pipeline, and surfaces that tell the truth - #42
Merged
Merged
Conversation
The product's claim is that it needs no API key, and an assistant that called one contradicted it in the same binary. Every path here was either dormant behind a key nobody sets or a second, weaker answer to a question the graph already answers deterministically. Removed: the assistant routes and panel, the model-settings surface and its persistence, the local/narrative agent fallbacks, and the Anthropic key check. The findings, the score and the swarm are unaffected - none of them ever consulted a model. 3,670 lines deleted, no replacement.
The web app owned the indexer, the detectors, the scorer, the graph builder and the persistence layer, so the CLI could not run any of them without importing the app - which the layering gate forbids, for the reason that a codemod running in two hosts must be one codemod. Splits the pipeline into packages behind published entry points: core-domain, config, observability, analysis-model, analysis, core-graph, detect-engine, score-engine, viz, imports, persistence, jobs, verify, sandbox, fsx and remediate-engine. Consumers move with it, which is why this touches every route and most components. The boundary is mechanical, not aspirational: `.dependency-cruiser.cjs` declares who may import whom and CI fails on a violation, so a new edge is a deliberate act recorded in that file.
Analysis materialised every file on disk and then read a fraction of it. Measured on microsoft/TypeScript: the git objects are 41 MB and the checkout is 655 MB, almost all of which the analysers never open. Adds a git-object reader - `gitTreeFiles`, `readBlobs`, `hasWorkingTree`, `materialisePaths` - and clones with `--no-checkout`. Manifests, and only manifests, are materialised, because `advisories.ts` reads `package.json` off disk and a tree without one looks like a project declaring no dependencies: a wrong answer, and a silent one. The checkout still happens, lazily, on the one route guaranteed to need real paths: `requireWorkspace` materialises on first file access, so the editor, search and git panels are unchanged. `reindex` opts out - it reads out of git and would otherwise reintroduce the 614 MB it exists to avoid. Measured end to end on expressjs/express: 380 KB indexed against 1.6 MB checked out, 76% of bytes avoided, and `git-source.test.ts` pins the equivalence - the same commit scores identically from either source, byte-for-byte on every coverage counter.
Measured against thirty-five questions written without reading the cue
table: five answered, thirty refused. The classifier was not wrong on
those thirty - a closed grammar has bounded recall by construction - but
"I cannot answer that" six times in seven trains people to stop typing.
A refusal is now the worst available answer. An unparseable question
degrades to a ranked lookup across symbols, topic tags, paths, packages
and endpoints, and says SEARCH on the receipt so a reader knows how much
to trust it. Only `ambiguous` (a real question back) and `not_analysed`
(a missing analysis) still refuse.
Four intents added over data the product already had and could not reach:
`ownership`, `stale`, `packages` and `unused_packages`. Same corpus after:
33 of 35 answered, and on the capability set those intents serve, 17 of 18
answer precisely.
Also fixes a confidently wrong answer: "which libraries are unused" folded
to `dead` before `package` and returned a hundred dead SYMBOLS under a
confident headline.
`normalise` no longer strips the dot between two characters, so `store.ts`
reaches the resolver as a path - no file entity had ever bound. Direction
is pinned on passive forms ("usages of X" was answering what X uses), and
the Ask button now takes the same ink as every other button, having asked
for a token that does not exist.
Three findings from driving the product rather than reading it. The headline CTA promised nothing it could deliver. "Generate verified fix PR" was offered at full prominence on every repository; on one with nothing to fix it ran 4.3s and returned "Not verified - no verification gate completed / nothing to patch". The product's third verb, apparently failing, in its most prominent place. It was knowable before the click: findings carry a rule and `FIXERS` declares which rules it handles, so `autoFixable` now answers first. The button reads "No auto-fixable findings" and is disabled, or "Generate verified fix for N findings" when there is work - checking both rule-id spellings, because migrated rows and freshly detected ones key differently. A no-op is no longer painted as a failure. `verdictFor` gains a fifth verdict: a run that produced no edits has nothing to verify and says so in neutral grey, distinct from a patch whose gates could not run. The dashboard led with garbage. Under "ranked by where attention is needed" the top three rows were FAILED repositories, one a typo from the previous day - 4 of 19 rows outranking the other 15, because a failure has no score and the risk order sorted its absence to the top. In-flight rows still float (a run in progress genuinely is urgent); permanently failed ones get their own group with Retry and Remove. `triage` is pure and exported, so the partition is testable outside a component. Retry re-submits rather than re-indexing - verified, not assumed: re-index answers 404 "Workspace not ready" for a repo that never cloned. It also retires the row it supersedes, guarded on the id changing; the first version duplicated a row on every attempt. Also: the ownership share bar asked for `--accent`, which does not exist, and rendered no bar. `design-system.test.ts` now fails on any custom property nothing defines - an undefined one is silently inherited, never an error, so it is exactly the class of bug review cannot catch. `no-deep-import-from-app` said "published entry point" but hardcoded index.ts. It now reads each package's `exports` map, because the registry must reach a client bundle and the package root drags `node:fs` in - the root import fails the build, verified. The guard still bites: a genuine deep import into package internals is still an error.
The landing page advertised "2.1s to index 327 files" with a five-bar breakdown. That profile was taken on a 303-file tree, six of the eleven stages the pipeline now has did not exist when it was written, and the largest one today was missing from the chart. The page's own footer promises the commands measuring its numbers are in the repo. `npm run selfindex` prints the per-stage timings, the cold total and the TypeScript file count, so the numbers map to something anyone can re-run - the same reason `scripts/bench.mts` exists for the README's benchmark rows. Counts synced to the tree: 116 test files, 2,185 cases, 375 TypeScript files. `readme-claims` and `landing-claims` assert the file counts against the working tree, so these cannot drift again silently; the case count and the timings are published with a date, because a wall clock measures the machine as much as the code.
Deliberately its own commit rather than folded into the ones above: a conflict resolution is the part of a rebase a reviewer most needs to see, and hiding three of them inside a 152-file refactor is how a dropped guard ships. `main` merged three PRs while this branch was in the working tree, and two of them touched the same code. - The clone gained `--no-checkout`, `--progress` and `--single-branch` here while main added `-c http.followRedirects=false` there. Both survive: the SSRF guard is unchanged and the redirect test now mocks `spawn` rather than `execFile`, because `runBoundedClone` replaced the `execFile` call the test was asserting against - so it had stopped matching, ran a real `git clone` and exited 128. A security test that green-lights by accident is worse than none. Mutation-checked: flipping the flag to git's default, or dropping it, both fail the suite. - `assertNotGitDir`, `isCommitHash` and `err()` each existed twice after the merge - once from main's hardening, once from this branch's extraction. Main's versions are kept; they distinguish missing from malformed and count the skip in coverage accounting. - Anonymous indexing was fixed twice, differently. Main refuses it unless an operator allows it AND the caller acknowledges the repo will be public; this branch gives anonymous callers a private, cookie-scoped bucket. Both controls are kept, with consent now demanded only when the result really will be world-readable. Asking for consent on a run that lands somewhere private is a warning about a thing that is not happening, which is how consent prompts get clicked through. - `NodeGraph`'s character-count truncation is gone in favour of the measured `fitText`; the estimator divided by an average advance, so it cut `IIII` early and let `WWWW` overflow. Counts re-derived after the merge: 120 test files, 2,206 cases, 379 TypeScript files.
archdex-art
force-pushed
the
feat/deterministic-intelligence
branch
from
August 10, 2026 16:39
d147c8f to
da38da2
Compare
CodeGraph
|
Two things CI caught that I had not run locally. `npm run lint` is a gate
and I checked the other five; that is my miss, and the fix is behavioural
rather than a suppression.
Three `react-hooks/set-state-in-effect` errors, all in code this branch
adds. Each was state that should have been DERIVED:
· Ask copied `?q=` into the input and cleared the answer from two
effects, so the component painted the previous question's answer and
then re-rendered to correct itself. Comparing against the question the
tree last synced to removes the flash, not just the warning.
· `loading` was set inside both fetch effects. "Loading" is exactly "the
data I hold is not for the thing I am rendering", so both now compare
the two directly and cannot fall out of step.
`action.yml` and its dogfooding workflow are withdrawn. The gate they wrap
runs correctly here — 75 unaccepted findings at or above high, which is the
gate doing its job — but on a hosted runner it wrote neither the SARIF nor
the summary, so the composite action failed for a reason I cannot reproduce
locally. Baselining would have turned the check green without explaining
that, and a gate whose own CI cannot be reproduced is not one to hand
anybody else. The README now documents the two-line `codegraph ci` workflow
that IS verified, and says where the packaged action went.
One of those 75 findings is mine and worth stating: `ask.ts` is 1,257 lines
against this repository's own 600-line bar. It is on the list below the
line, not hidden.
`dependencies.test.ts` timed out on CI at the 30s default. Both self-index
tests called `indexRepo(".")` — a full walk of the whole repository — and
they assert different properties of the SAME analysis, so the second walk
bought nothing. One already carried a 120s override; the other inherited
the default and fell over as soon as this branch made the tree bigger.
Raising the second timeout would have hidden the duplication. Indexing once
in `beforeAll` removes it: 31.4s to 16.8s for the file, and no test now
sits anywhere near a default timeout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seven themed commits over one working state, rebased onto current
main. Every gate is green on the tip; the intermediate commits are reviewable groupings, not independently-green steps.Gates: 2,206 tests / 120 files · typecheck clean · depcruise 0 violations · build compiled.
What is in here
chore(web)remove the LLM assistant layerrefactorextract the pipeline into packages.dependency-cruiser.cjsin CI.feat(vcs)read repositories out of gitmicrosoft/TypeScriptis 41 MB of git objects and a 655 MB checkout, almost all of which analysis never opens.feat(ask)degrade to ranked searchfix(web)surfaces tell the truth up frontdocsre-derive the published figuresfixreconcile with main's security workThe measurements
Reading out of git. Verified end to end on
expressjs/express: 380 KB indexed against 1.6 MB checked out — 76% of bytes avoided. The checkout still happens, lazily, on first real file access, so the editor, search and git panels are unchanged.git-source.test.tspins the equivalence: the same commit scores identically from either source, byte-for-byte on every coverage counter. Two real bugs surfaced only because that test compares the two sources against each other rather than against hardcoded numbers.Ask. Measured against 35 questions written without reading the cue table: 5 answered → 33. A refusal is now the worst available answer — an unparseable question degrades to a ranked lookup and says
SEARCHon the receipt, so a reader knows how much to trust it. Four intents added over data the product already had and could not reach (ownership,stale,packages,unused_packages): 17 of 18 answer precisely on the capability set they serve.Also fixes a confidently wrong answer — "which libraries are unused" folded to
deadbeforepackageand returned a hundred dead symbols under a confident headline. Andnormaliseno longer strips the dot between two characters, sostore.tsreaches the resolver as a path: no file entity had ever bound.The fix button. It ran 4.3 s to say "Not verified — no verification gate completed / nothing to patch" — the product's third verb apparently failing, in its most prominent place. Knowable before the click, so it now says so instantly, or
Generate verified fix for N findingswhen there is work.The dashboard. Under "ranked by where attention is needed", the top three rows were failed repositories — one a typo from the previous day. 4 of 19 rows outranking the other 15, because a failure has no score and the risk order sorted its absence to the top. In-flight rows still float; permanently failed ones get their own group with Retry.
The rebase, and why the reconciliation is its own commit
mainmerged three PRs while this work sat in the tree, two of them touching the same code. A conflict resolution is the part of a rebase a reviewer most needs to see, so it is not folded into the 152-file refactor above it. Readfix: reconcile with main's security workclosely — it is where a dropped guard would hide.-c http.followRedirects=falseis intact. Its test was asserting the argv of anexecFilecall that no longer happens (runBoundedClonereplaced it), so the mock never matched, a realgit cloneran and exited 128 — a security test green-lighting by accident. It now mocksspawn. Mutation-checked: flipping the flag to git's default, or dropping it, both fail.assertNotGitDir,isCommitHash,err). Main's versions kept: they distinguish missing from malformed and count the skip in coverage accounting.Two things I got wrong, caught by running it
Retry duplicated rows. The first version re-submitted the URL assuming the row would be reused. It is not — a control added to clear clutter created it. Now retires the superseded row, guarded on the id actually changing. Re-measured: 24 → 24 rows where the buggy version added one per click.
I nearly weakened a boundary. The registry import tripped
no-deep-import-from-app. Before editing a guard rule to suit myself I tested the alternative — importing the package root fails the client build, because the index re-exportsnode:fs. So the rule's hardcodedindex.tswas the imprecise part: it says "published entry point" but could not read theexportsmap that defines one. It now does. Proved it still bites: genuine deep imports into package internals are still errors.Review notes
fix(web)) — the highest-risk and the most user-visible, both small.refreshWorkspace > ignores credentials, a trailing slash…times out at exactly 5,000 ms spawning git under full-suite load. Passes 3/3 in isolation. Widening that timeout is a separate call.Still open, deliberately
FIXERShas one provider.empty-catchis also line-scoped, so it only fires on a single-linecatch (e) {}— the multi-line form every formatter produces is invisible. The button is now honest about this; the coverage gap is real and unaddressed.SymbolEdgeKinddeclares six kinds and emits two (calls,contains).references,imports,extendsandimplementsare in the union and never produced. Either build them or delete them from the type — I did not want to choose unilaterally.tree-sitterdependencies (49 MB) remain inapps/web/package.jsonandnext.config.ts, imported nowhere since the OOM postmortem.What CI caught that I had not
Three more commits, all from the pipeline rather than from me. Worth reading as part of the review:
lintwas a gate I never ran. Five of six locally, not the sixth. Threereact-hooks/set-state-in-effecterrors, all in code this branch adds, all fixed by deriving the state rather than suppressing the rule — Ask no longer flashes the previous question's answer before correcting itself, which was the same bug the lint rule describes.codegraph ciworkflow that is verified.beforeAlltook the file from 31.4s to 16.8s. Raising the timeout would have hidden the duplication.One of those 75 findings is mine and belongs above the line, not below it:
ask.tsis 1,257 lines against this repository's own 600-line bar. It is the file that grew most in this PR, and splitting it is the obvious follow-up.Final CI: both required jobs green — Test & Build, and Docker build + adversarial smoke test.