Repository navigation
Conversation
…s budget on them The product already detected that half of a run's insights restate another insight from the same run. It detected it last — at 97% progress, after the validation agent had spent its fixed budget checking those restatements, and after the recommender had made its selection on the result. The decision now happens at the end of each analysis area, before that area is validated. A marked restatement is skipped with no charge against the run cap, and excluded from Phase 5 on the mark rather than on its verdict: a nil verdict fails open, so filtering on the verdict alone would make consolidating a finding promote it. Two proposal tests run and are unioned, because measured on real run output they find largely different duplicates — 18 similarity pairs, 16 containment pairs, 3 in common. Containment is the one similarity cannot see: a finding whose headline is one row of another's table scores 0.71-0.87, and four of seven hand-applied folds on one run sat below the 0.88 floor. Which member survives goes through a provider seam. Scored against a reviewer over three runs, document order picks the wrong survivor most of the time: applying one run's agreed folds required unmarking four of five chosen survivors. With no provider registered the community rule is the one the end-of-run check already applies, so a community deployment gets today's marks, decided earlier. Three guards that are load-bearing rather than defensive. Clusters are built from direct pairwise relations: transitive closure over containment proposed suppressing 23 of 55 insights where the direct relations yield 9. Every fold is re-verified against the canonical the mark will actually point at. And the write guard is one transitive rule, not a set of per-case checks — three hand-run passes produced three variants of the same bug, each guarded in turn and each followed by the next, the worst a mutual cycle in which neither finding had a visible canonical. A run still embeds each insight exactly once: the pass fills a per-run vector cache keyed on the embedding text and the end-of-run index reads it. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…ads one struct deeper Four from a self-review before the review loop. A merged description was gated on the RUN's fold counter, so once any cluster had folded something, every later cluster's rewrite was applied to a canonical nothing had been folded into. Gated on its own cluster's folds now. Which test proposed a cluster was read from every relation touching any member, not from the relations inside it. A member of a containment cluster usually also resembles documents in a neighbouring star, so a containment-only cluster was reported as "both" — and that field is the one thing that says whether the similarity threshold could have seen the duplicate at all. Two overlapping maximal stars is the case that tells the two readings apart, and it is the test. A restore that clears an unresolvable mark now clears the verdict that existed only because of it. skipped_duplicate is stamped BY the mark, so a finding visible again with that stamp still on it rendered as a restatement of nothing. A verdict the verifier actually produced is untouched. And the API's AnalysisStep mirror declares the consolidation summary. Its class guard reads top-level int counters, so a whole nested block of them could be persisted by the agent and dropped by the mirror with every test green — the same failure the guard exists for, one struct deeper. The guard now also requires every pointer-to-struct field, verified by removing the field and watching it fail. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… and marks settle once Codex round 1, four findings, all valid. The end-of-run check could fold a document another finding already points at: a mid-run mark B -> A is written and persisted, so folding A into an earlier run's X left B pointing at a hidden row — and B cannot be re-pointed there, because X belongs to another run and its vector is not in hand to re-verify against. A document with dependants is now left alone and its cross-run fold deferred. A member the PROVIDER omitted from Duplicates is an explicit judgement that it states a different claim, and nothing recorded it — so the end-of-run check folded the pair anyway on document order, overriding the one decision this feature exists to take. Those pairs are recorded now, and only for clusters the provider decided: the community fallback's non-folds ARE the end-of-run rule, and freezing those would change the community outcome rather than preserve it. Clusters are overlapping stars, so two verdicts can fold different members into the same survivor — and each merged description replaces its text, so the second erased what the first preserved while both folds stood. First rewrite wins. And a verdict that asks to rewrite an EARLIER area's survivor now gives its folds back rather than applying them without the rewrite: the rewrite was asked for because the survivor is missing something the folded members carry, so folding anyway is exactly how a fold loses a caveat. Marks now settle once, after the loop. The run's analysis areas come out of a map, so their order is not stable between attempts and an area that folded into another can restore BEFORE it — clearing on the spot threw away a good mark whose target had not registered its ids yet, and nothing revisited it. settleRunMarks re-points those and clears only what the run genuinely does not hold, on both the accumulator and each area's step. Also adds DISCOVERY_CONSOLIDATION_ENABLED (default on), forwarded to the agent. The similarity floor is a threshold and cannot express "do not consolidate", and the containment test does not read it — so an operator who wanted the previous behaviour had nothing to set. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found re-reading the round-1 fix. Reverting a cluster's folds cleared the mark off the insight but left it in the chain map, so a later cluster resolving its canonical through that map would resolve through a mark that no longer exists and point a real fold at the wrong document — and keepStanding read the pair as folded and recorded no keep for it, so the end-of-run check would fold it the way the revert had just refused. Also records the embedding request-count caveat and the new configuration in the changelog: the token count is unchanged, the number of batches is not. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… a merged survivor Two more from re-reading. A cluster given back decremented Folded but not the per-test bucket it had been counted in, so the log line could read folded=0 beside by_containment=1 — and the fold rate per test is the one number that line exists to make legible. Both directions go through one function now. And a merged survivor is re-projected, so a later cluster folding into it runs its containment re-check against what the survivor now says rather than against the text it said before the rewrite. The vector is necessarily stale, which is the conservative direction: a cosine re-check can refuse a fold that would now hold, never accept one that would not. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e either Codex round 2, four findings, all valid. A mark into a restored area is now STORED by the id the run's checkpoints hold rather than the one this attempt minted. A third attempt restores that area from its own checkpoint, so it can only translate the ORIGINAL id — a mark naming the second attempt's id resolved against nothing, and settleRunMarks correctly cleared it, costing the fold and returning an unvalidated finding to the recommender. Same discipline as Phase 5's stored citations, and only the stored copy is translated: the live mark must keep naming the live id or the dashboard renders a suppression pointing at an id no row carries. Two overlapping stars can fold different members into the same survivor, and the second rewrite was dropped while its folds stood — so that cluster's members were suppressed without the detail the rewrite was asked for. Both reasons a rewrite cannot be applied now get the same answer: the cluster's folds go back and both findings stand. A merge carries the folded members' SourceSteps with it. The verifier builds its evidence bundle from the survivor's cited steps, so a merged claim whose step the survivor does not cite would be checked against evidence that does not contain it, and attachSourceQuality would miss the caveat that step carries. And the provider contract is positional and enforced. "Clusters may be omitted" plus reading by position silently reverses decisions: a provider that answers only the second of two clusters has its answer read against the first, where it is rejected as naming a stranger, and the second then gets the community rule — which may fold it the way the provider refused to. A response whose length is not the cluster count is discarded whole. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The projection sent the supporting figures both as their own field and appended to the body, which is prompt budget spent saying one thing twice — and it made the 2,000-character truncation cut the prose to make room for a repeat of the figures. The containment test still reads both, because a figure listed as an indicator is a figure another finding can restate. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each call's prompt is bounded by the layer's input window, but the NUMBER of calls grows with the cluster count — so nothing bounded what the pass can cost, and the feature's own acceptance target is under 1% of a run's tokens. DISCOVERY_CONSOLIDATION_MAX_CLUSTERS (default 60, non-positive for no bound) caps it. Deliberately generous: four measured runs proposed about thirty candidate pairs per RUN, so sixty per area is far above anything observed, and an area that reaches it is either a real finding about the playbook or a bug in the proposal step — which is why it is logged and recorded on the area's run-log row rather than passed over. Clusters past the bound are not dropped. They get the community rule, which is the decision the product makes today. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…gure-bearing survivor is not rewritten
Codex round 3, three findings, all valid.
The re-check that guards every fold accepts either test, and the community
fallback could satisfy it by containment — folding pairs at a cosine of 0.643
against a 0.88 floor, on a rule the deployment never opted into. Containment is
a proposal the adjudicator rules on; the fallback is now cosine-only, which is
what "the community outcome is unchanged" was supposed to mean.
A pair the adjudicator decided should stand is no longer folded by a fallback
verdict for an overlapping cluster. The clusters are stars and they share
members, so the same pair can appear in one cluster a provider answered and
another it declined — and the end-of-run check cannot correct it afterwards,
because it skips a document that already carries a mark.
And a survivor whose prose carries rendered evidence figures is not rewritten.
The figure template holds that text as authored with its {{fN}} references
intact, and a recommendation may restate a figure by naming its id — so
free-text replacement severs the link: every field stops matching its
re-render, which withholds a repaired insight's figures from the recommender
entirely, and on an unrepaired one a recommendation could restate a number the
survivor's prose no longer shows. The model is shown the RENDERED text, so it
cannot preserve the references, and re-deriving them from free text is the
numeral surgery the figure layer exists to avoid. The cluster is given back,
like the other two survivors that cannot be rewritten.
Refs #555
Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… fold the pair the community rule can see Codex round 4, five findings: four fixed, one documented. The keeps are registered before a single mark is written. Discovering them incrementally was useless in the case that matters: the clusters are overlapping stars processed in order, so a cluster the provider DECLINED can be processed first and its fallback folds a pair a provider verdict for a later, overlapping cluster explicitly keeps — and by then the mark is written and the end-of-run check will not revisit a marked document. A pair's keep is derivable from the verdicts alone, so it can be known first. That exposed a tension worth recording. Two overlapping verdicts can disagree about one pair, folded in one and simply unnamed in the other, and an explicit fold now beats an omission: the fold is a positive judgement, while a member a verdict did not mention may just be a member it did not mention. And a canonical is resolved the way the writer resolves it, so a verdict naming an already-suppressed canonical is read as folding into the end of that chain, not into the row it named. A pair whose fold the writer then REFUSES is registered at the refusal, because only the writer knows that. The end-of-run check honours a keep against the target it RESOLVED to, not only the candidate it matched. With b folded into a and the adjudicator having kept a and c apart, matching c against b resolved back to a and suppressed c anyway — the decision routed around by one hop. A rewrite is refused when any member it folds was refused: it may incorporate detail from a donor nobody folded, which would credit the survivor with a claim whose evidence it does not cite, and the provenance union cannot carry what was never folded. And the community rule applies pairwise inside the cluster. A star's seed need not be in every pair, so comparing everything against it folded nothing on a mixed star and both later members spent validation budget before the end-of-run check folded one into the other anyway. Documented rather than built: a mark into an area whose completed-checkpoint write FAILED, recorded against the surviving draft's id, cannot be resolved by a third attempt. It is cleared rather than left broken, so the finding is visible and unvalidated — which is what it would have been before this phase existed. Persisting an alias for that path is a change to the checkpoint schema for a two-failure case. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…community rule decides every group Codex round 5, three findings, all valid. An operator clicking Validate on a finding the dashboard shows as restated had their request turned into a permanent no-op: the run-wide duplicate skip applied to the single-document path, so the job returned skipped_duplicate without running either agent and overwrote whatever verdict the document had. The skip exists so a RUN's fixed budget is not spent on restatements, and someone who clicked the button has already decided to spend it. A rewrite is refused when it names a donor this pass did not fold because a previous area had ALREADY folded it. The rewrite credits the survivor with that donor's detail while the provenance union carries only what this verdict folded, so the survivor would state a claim whose evidence it does not cite. And the community rule decides every similarity group in a cluster, not just the first. One maximal star can hold two independent groups joined only through a finding that contains both by figure — and buildClusters removed the smaller stars as subsets, so the second group got no pre-validation decision at all: both its members spent validation budget and the end-of-run check folded one into the other afterwards anyway. A cluster can now carry several verdicts, which is also the shape the community path always wanted. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rotection that lifts when its fold does Codex round 6, three findings, all valid. Containment compared SUBSTRINGS: a headline stating 123 and 456 was contained in a body stating 1123 and 2456 — two different measurements reading as the same one, and on a pair below the similarity floor the fold costs a distinct finding its verdict and its place in Phase 5 on nothing at all. Both sides go through the same figure parser now and the comparison is token against token. The community rule is one pass over the run in document order rather than a decision per cluster. Two reasons, and it took three rounds to see both: the clusters are overlapping stars, so a per-cluster pass carried state that depended on which star came first — on two overlapping stars whose middle members are all similar, the first made one of them a canonical and the second then refused to fold it into the earlier member it actually restates, leaving two redundant findings standing. And the rule has to MATCH what the end-of-run check would have made, because the promise to a community deployment is that only the moment of the decision moves; that check is one global pass, so this is the same function over the documents in hand. A document is protected from suppression while this pass's marks point AT it, and the protection now lifts when they are given back. It was a flag, so a document that was briefly a canonical and then lost its only dependant stayed protected for the rest of the pass and a later valid fold of it was refused. And a cluster given back for an unapplicable rewrite now protects its members against every other member, not only the survivor the verdict named: the reason for refusing is that the member carries detail the survivor lacks, and that is just as true of any neighbour it might be folded into instead. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…only through folds that can be written Codex round 7, two findings, both valid. When EVERY fold a verdict proposed is refused by the re-check there was nothing to revert, so the protection branch never ran — and the members the verdict named still carry detail the survivor lacks, so the community pass then folded them sideways into each other and the caveat the rewrite existed for was lost through a different door. The protection is now independent of whether anything was written, and it covers every member the verdict named rather than only the ones that got as far as a mark. Applying the rewrite still requires a fold: a rewrite with nothing folded into the survivor loses detail rather than adding it. And the community chain is built only from folds this pass could actually WRITE. Recording a fold a keep forbids, or one whose document this pass cannot mark at all, made a later document's chain re-check resolve against a canonical the reader is NOT being sent to — so a document that plainly restates the visible finding was left standing and spent a validation check, before the end-of-run rule folded it there anyway. Three tests now assert the OUTCOME rather than a refusal counter, which is what they were always about: a pair the chain never proposes is not a pair that was proposed and refused. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rote Codex round 8, one finding, valid. The fallback was simulated against the marks the documents ARRIVED with, not the ones the provider verdicts had just written. With the provider folding b into the later c and declining the overlapping cluster, the simulation still thought b was visible: it folded c into b, and a later d then failed its re-check against b — while c, which d plainly restates at 0.906, had been visible the whole time. d spent a validation check and could reach recommendation generation before the end-of-run rule folded it into c anyway. Seeded from the live marks now, and a document this pass has already sent a reader to is skipped as a fold source for the same reason an already-marked one is: its mark would never be written, so nothing may resolve through it. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nd a donor's figures are not left orphaned
Codex round 9, two findings, both valid.
A rewrite is written against the text of the document the verdict NAMED. If
that document has itself been folded, the mark resolves somewhere else — and
applying the rewrite to the resolved survivor overwrites ITS caveats with a
rewrite of someone else's prose. The named document need not even have been in
that cluster's payload, and when the first fold needed no rewrite nothing else
caught it.
And a rewrite is refused when a FOLDED member declared evidence figures. They
were rendered into that member's prose against {{fN}} references the model
never saw and cannot reproduce: transferring the declarations leaves figures
with no placeholder in the survivor's text, which the figure layer withholds
anyway, and leaving them behind puts the number in the survivor's prose with
nothing a recommendation can reference it by — so ref and sum over that
measurement stop working while the member that owns it is suppressed. The
member stays visible instead, with its figures intact. A fold with no rewrite
is unaffected.
Refs #555
Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ed mark takes its log entry with it Codex round 10, two findings, both valid. The first corrects a claim I had written into a comment. A rewritten survivor now LOSES its cosine projection. The comment said keeping the pre-merge vector was the conservative direction; it was not. A rewrite can REDUCE similarity below the floor, and the community pass would go on folding against the old vector — suppressing a finding that no longer restates this one, and costing it its verdict and its place in Phase 5. The merged text cannot be embedded mid-area, so there is no cosine answer to give, and no answer is the conservative one. Containment still works over the new body, and Phase 9 embeds the merged text and decides on a real vector. And clearing a restored mark now clears the validation-log entry that existed only because of it. The step's ValidationResults are persisted by the split-log writer and served by the validation-results endpoint, so leaving the entry behind labelled a finding that is visible again, and was never checked, as Restated — on the one surface an operator goes to in order to ask what WAS checked. Only a skip-ONLY entry is removed: one carrying agent detail is a verdict that was actually produced. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Important Superseded. This was written when the review loop stood at its 10-round Final reportWhat. The community half of pre-validation consolidation: a new Phase 4.4 inside the Phase 4 area loop, the pairing rule extracted so one derivation serves both callers, figure containment as a second proposal test, the Why. Measured over four consecutive runs of one project, same playbook, same build: within-run restatement is 7.1% with no recorded history and then 30.9% → 48.3% → 42.6% once there is any. A step, not a ramp, and untouched by the prior-context work — the ten areas are prompted independently. Cross-run restatement is 0–3 folds a run and can be considered done. How, and the four things that are load-bearing rather than defensive.
With no provider registered the outcome is unchanged — and that claim is now exact rather than approximate. The community rule is one pass over the run in document order, which is the end-of-run check's own rule run early rather than an imitation of it. It took three rounds to get there: per cluster, then pairwise within the cluster, then every group within the cluster, before accepting that overlapping stars make any per-cluster state order-dependent. The global pass is also less code. Verified.
Review. 10 Codex rounds — the loop's hard cap — 29 findings, 28 fixed and 1 documented. A third of the later ones were my own earlier fixes overshooting or stopping one step short, which is worth saying plainly. The ones that mattered most: the dashboard was changed to offer Validate on a restated finding and the run-wide duplicate skip then made that request a permanent no-op that overwrote the document's verdict; a merged description silently invalidated the figure→prose link the recommender reads, so a repaired insight's figures were all withheld; provider keeps had to be registered before any mark was written, because a fallback cluster processed first could fold a pair a later provider verdict kept; and the API's analysis-step mirror dropped the whole nested counter block, which its own class guard could not see until it was taught to. No Copilot pass: this repo has no Copilot auto-review configured. The loop reached its cap rather than a clean round, and that matters for how to read this. Round 10 found two defects and both are fixed, but those two fixes were not themselves put through another round. They are worth a human eye specifically:
Every round produced real defects rather than theory, which is why the loop ran to the cap rather than stopping early. Round-by-round, the share that were my own earlier fixes overshooting or stopping one step short rose toward the end — the shape worth recording is that a guard patched per case grows a new variant each time, and the rule that ends it is transitive. Documented rather than built. A consolidation mark into an area whose completed-checkpoint write failed is recorded against the surviving draft's id, and a third attempt cannot resolve it. Merge order. This merges first; decisionbox-io/decisionbox-enterprise#467 pins Project board 4 could not be updated: — Co-coded with Jale 🤖 |
…rop a fold's justification Codex round 11, two findings, both valid — and both an invariant already enforced elsewhere not being enforced on one path, which is why they are worth fixing rather than guarding around. The magnitude guard reached only the cosine acceptance. sameClaim refuses a fold where the two texts are IDENTICAL and the figures differ — 100 affected entities hidden behind 1,000 — and containment accepted without passing it, so two otherwise identical findings were folded whenever the headline's figures appeared in the other's body. It now gates both, which refuses only the identical-text case and leaves every ordinary containment fold alone. And a rewrite is refused when it drops the figures a fold already written into that survivor rests on. Only a CONTAINMENT fold can be broken that way — it is justified by figures in the survivor's body, while a cosine fold rests on the vectors — and the closure check cannot catch it, because it verifies where marks POINT rather than whether the claim still holds, and the end-of-run check skips a document that already carries a mark. The finding would stay suppressed and unvalidated with its claim stated nowhere. Also collapses the merge guard, which had grown to six conditions each written TWICE — once to select the branch and again to name it, which is how a guard this long stops being checkable against itself. One function returns the reason or none, and the reasons group into three: the rewrite is about text that is not what this survivor says; the text cannot be replaced because something else derives from it; or the rewrite would lose something it claims to carry. No behaviour change — and the test suite caught the one thing that did change, since the old `case a, b, c:` short-circuited before dereferencing a slot that is -1 on an earlier area's document. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… just this cluster's Codex round 12, one finding, valid — and it corrects the INPUT to last round's guard rather than adding a condition to it. The survivor can already carry folds from an overlapping cluster of the same pass, or from a previous area's. A rewrite that keeps THIS cluster's figures while dropping an earlier dependant's left that finding hidden behind a survivor which no longer states its claim — the same defect, one cluster further out. The guard now reads the live mark set, which holds both the arrival marks and everything written since, so 'every fold into this survivor' is answerable rather than approximated by the cluster it happens to be looking at. That also removed a parameter: the check no longer needs the insight slice or the id index, only the docs and the marks. Refs #555 Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… model-supplied id is not provenance Three findings from review round 13, two of which rest on the same premise: an insight's id is the MODEL's when the model supplies one — the parse mints a UUID only for an insight that arrived without one — so ids are not unique across areas or across attempts. - A merge rewrite is refused outright for any dependant the verdict does not itself fold. The rewrite was written from this cluster's payload, which does not contain that finding's text, so there is no sense in which it carries its claim and no way to check: a cosine justification cannot be re-established against prose that has not been embedded. For the verdict's OWN dependants a similarity justification is trusted (the adjudicator saw the text; that is what MergedDescription is for) and a containment justification is re-checked. This replaces a weaker rule that skipped every cosine-justified dependant, which let a rewrite drop an earlier cluster's claim and leave that finding hidden behind a survivor that no longer stated it. - Two restored areas carrying the same stored insight id no longer resolve last-writer-wins, which pointed a mark at whichever area restored second — a reader sent to an unrelated finding while the one they asked for stayed hidden. The entry is poisoned instead: the fold is lost and both findings stay visible, which is the recoverable direction. - A restored mark now needs provenance. Matching a live id is evidence for a mark this attempt's consolidation pass wrote, and a coincidence for one that came out of a checkpoint: a re-analysed area can hand back a previous attempt's id for a different finding. A restored insight's mark survives a live-id match only if the target is also an id this attempt minted for a restored area; any other match is cleared and the finding is visible again. Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…kpoint's does The id mapping has two write sites, and the previous commit only hardened one. A draft-restored area's remap still merged last-writer-wins, so a draft id colliding with a completed checkpoint's silently overwrote it — the same hazard, for the same reason: an insight id is the model's when the model supplies one. A call-site routing; the behaviour it routes to is covered by TestRegisterRestoredIDs_ASharedStoredIDResolvesNowhere. Reaching this line from a test means standing up the whole Phase 4 loop, which needs a warehouse and an LLM. Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d mark Round 14, and the finding is the previous round's fix overshooting in the way this guard keeps overshooting: I derived "did this attempt mint this id for a restored area" from the values of the id remap, to avoid threading a second map through every restore. The remap poisons a stored id two areas claim — which replaces those two areas' fresh ids in the value set, so the insights most likely to carry a colliding mark were exactly the ones the trust rule stopped protecting. A stale mark from either then rode a live-id coincidence into a reader's hidden finding, with its skipped_duplicate verdict intact and the finding excluded from recommendations. The two questions are now two maps on a `restoredIDs` value, which is what the restores and the end-of-run settle thread between them instead of a bare map: `freshByStored` resolves a mark's target and is poisoned on a collision, `minted` records every id this attempt minted for a restored area and is not. Neither derives from the other, so there is no hole to find. The regression is pinned twice: fromRestore must survive a collision on the stored id, and a settle over a poisoned pair must clear the mark rather than hide the finding. Both fail against the derived-from-values version, the second with the exact symptom above. Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…h, once Round 15, two P2s, both the same structural flaw and both mine: the mark is translated eagerly at restore time, and then the settle reasoned about what the mark SAYS instead of what it ARRIVED as. The translation runs against a mapping the loop is still writing — a restore happens in the same pass over the areas as a fresh analysis, so areas later in that order have not registered yet. So it can resolve a mark through an entry a later area then poisons, which left a finding suppressed behind an unrelated canonical that was live and minted and therefore passed every check the settle made. And it can fail for a canonical that registers a moment later, whose stored target a freshly analysed area may happen to carry as a live id — on which the settle cleared a mark the final mapping resolves perfectly well, costing a real restatement its skip verdict and sending it to the recommender unvalidated. One rule for both: `repointRestoredMarks` records the stored target each mark arrived with, and `settleRunMarks` re-decides every such mark from that value against the final mapping. The translation becomes what it always was — a convenience for the mid-run consolidation pass, which needs live ids to tell which earlier-area documents are already canonicals, and whose worst case is a fold it declines to make. This REPLACES round 14's minted set rather than adding to it: the arrival target answers both "did this mark come out of a checkpoint" and "what did it mean", so there is one source of truth instead of two that can disagree. Safe because the pass never writes a mark onto a restored insight: a completed area is `continue`d long before Phase 4.4 and a draft-restored area's marks arrive with the draft, so a restored insight's mark is always the checkpoint's. Four tests now drive the real restore path rather than hand-writing a mark, because the arrival target recorded on the way past is the thing under test. Both new cases fail against the round-14 code with the exact symptoms above. Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round 16 found no correctness issues and one documentation gap, which was real: neither switch appeared in the configuration reference, so an operator had no way to discover either the master switch or the cost ceiling. Both rows state the default, what the setting buys, and what a non-obvious value does — for the ceiling, that a non-positive value means no ceiling and is treated as a deliberate choice rather than a typo, and that clusters past it fall through to the same rule Phase 9 uses rather than being dropped. The enabled row also says plainly what leaving it on costs: a mark here takes the finding's verdict and its place in recommendation generation, where before this feature it only hid a row behind an expandable canonical. Not added to docker-compose.yml, the other half of the finding: no DISCOVERY_* setting appears there, so the convention in this repo is that the discovery switches live in the reference alone. Adding two of eighteen would make the compose file look authoritative about a family it does not cover. Co-authored-by: Jale <jale@decisionbox.io> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Final report — the review loop closed cleanThis supersedes the 16:16 report, which was written at the loop's old 10-round cap. The cap was lifted; the loop ran to round 17, which raised nothing. What the feature is and why, the four load-bearing design calls, and the measured numbers behind them are unchanged and still read correctly there — this report covers what the extra seven rounds found and where the PR now stands. Where it stands. Rounds 11–17: seven rounds, five findings, every one of them in a fix of mine rather than in the original code. All five were in the resume path's consolidation-mark bookkeeping, and they came in a chain — which is the part worth a reviewer's attention, because it says something about that file specifically. The premise under four of them: an insight id is the model's when the model supplies one. The parse mints a UUID only for an insight that arrived without an
The version that stands removes the mechanism instead of guarding it again. Honest note on that chain. Four consecutive rounds found defects in my own guard, each a per-case patch that exposed the next variant. The signal was there by round 13 and I kept patching until round 15. The same lesson had already been recorded earlier in this PR about the write guard, where one transitive rule ended three rounds of per-case checks; I did not apply it to the resume path until the reviewer had found the same shape twice more. Nothing is left outstanding from it, but if one file in this diff deserves a slow read, it is Verification added, not just assertions changed. Every one of these fixes has a test that fails against the code it replaced, checked by running the new tests against the old product file rather than by reasoning about them. Two of those reproductions print the exact symptom the reviewer described — a mark left naming an unrelated canonical, and a good mark cleared along with its skip verdict. The four mark tests now drive a real restore (register, then re-point) instead of hand-writing a mark, because the arrival target recorded on the way past is the thing under test; the old fixtures encoded the mechanism that just went away. Round 16's finding was a documentation gap and it was real: neither Still outstanding, unchanged.
— Co-coded with Jale 🤖 |
Closes #555. The community half of pre-validation consolidation: the hook, the extracted pairing rule, the two proposal tests, the marks, and validation + Phase 5 honouring them. The adjudicating agent is enterprise and registers through the seam added here.
What changed
The hook.
orchestrator.go, Phase 4 area loop, betweenstep, insights = out.step, out.insightsandcheckpointAreaDraft. There and not a line either side of it, for three reasons:step.Insightsandinsightsare the same slice, so a mark written there is in the draft the next line checkpoints — a resumed attempt restores the marks instead of re-deriving them and never pays the provider twice; it is before thestopAfterDraftpause seam, the first point a run can stop without having paid for validation; and it is inside the!fromDraftbranch, so a restored draft's marks arrive with the draft and a completed area wascontinued long before.One embedding-text derivation (
models.InsightEmbeddingText).BuildEmbeddingTextwas aSprintfover five fields that are all also on the in-flight insight. Two derivations of what a document is is how the mid-run pass and the end-of-run check come to disagree about one — the same disciplinemagnitudeOfexists for on the quantitative side.The pairing rule, extracted into
consolidate_cluster.go: the threshold, thesameClaimmagnitude guard, and the canonical-chain re-check.markDuplicatesnow calls it. The existingmarkDuplicatessuite — includingChainsDoNotAssumeTransitivity,ChainCannotRouteAroundTheMagnitudeGuardandSameWordingDifferentMagnitude— passes unchanged through the new path, which is the equivalence guard.Figure containment, the second proposal test: every distinctive figure (three or more digits) of one headline appearing in another's body. Two or more figures required — a single shared three-digit number between two findings about the same warehouse is coincidence often enough to matter, and the cost of a wrong fold here is a finding that loses its verdict.
The seam (
agentplugin.ConsolidationProvider), modelled onDiscoveryPolicyProvider: single provider, last registrant wins, panics recovered, and an error returns no verdicts rather than the partial set — a half-applied adjudication is the one state the write guard cannot reason about.skipped_duplicate, its own verdict rather than reusingskipped_budget_cap. "There was nothing here the canonical does not already carry" is not "the run ran out of budget", and conflating them would make the measurement this feature exists to move unreadable from the stored verdicts. Added through the dashboard's five verdict surfaces too; it renders as Restated.Also:
llmlayer.LayerConsolidation, the per-areaconsolidationblock in the run log (folds counted by which test proposed them), and the marks carried onto the standalone copy at denormalisation — the standalone document is what the API filters on, so a mark that stayed on the embedded copy would hide a finding on the dashboard and return it from the API.Why the measurements point here
Four consecutive runs on one project, same playbook, same build:
Cross-run restatement is solved — #458/#545 did their job. Within-run is not, and it is a step rather than a ramp: ~7% with no history, 43–48% once there is any. The ten areas are prompted independently, so two of them reporting the same claim is the dominant shape on any project with a ledger behind it.
The three guards, and why none of them is theoretical
Clusters are stars over direct relations, never connected components. Building them by transitive closure over containment — A contains B, B contains C, therefore one cluster — proposed suppressing 23 of 55 insights on real run output where the direct relations yield 9. A chain of four documents each 25° apart is the case that tells the two apart, and it is the test.
Every fold is re-verified against the canonical the mark will actually point at. Neither test is transitive: 0°, 25° and 50° score 0.906 between neighbours and 0.643 end to end, so matching the middle says nothing about the head. Two of nine hand-applied folds on one run were dropped by exactly this re-check.
The write guard is one transitive rule, not a set of per-case checks. Three hand-run passes produced three variants of the same bug — a fold whose canonical the same proposal also suppressed; a fold whose canonical a previous pass had already suppressed, which produced a mutual cycle in which neither finding had a visible canonical; and a pre-existing mark pointing at a document the pass suppresses. Each time the case just seen was guarded, the next variant appeared. After writing, every mark in the run is resolved to a final canonical and re-verified, and a pass that cannot prove its own marks gives them all back.
Limitations, stated
Verification
check-gate.sh(every module independently,-count=1): green apart from the three known-environmental failures on this box —libs/go-common/mongodbagainst the occupied:27099, and the two config tests against an exportedMONGODB_URI.make lint-go(golangci-lint v2.11.3): 0 issues across every module.make test-ui807 tests / 54 suites,make lint-uiclean apart from the pre-existing_propswarning,tsc --noEmitclean,make lint-docsclean. CI green.markDuplicatessuite —ChainsDoNotAssumeTransitivity,ChainCannotRouteAroundTheMagnitudeGuard,SameWordingDifferentMagnitude— passes unchanged through the extracted rule. That is the equivalence guard for the extraction.confirmedduplicate, a verdict-less one and askipped_duplicateone; the end-of-run check not re-deciding a consolidated document, not folding one another points at, and honouring a keep against the resolved target; the community pass seeing the provider's marks, chaining only through folds it could write, and deciding every group; the five cases where a rewrite cannot be applied; the vector cache paid once per text; and the restore re-pointing one mark, clearing another, and taking that mark's log entry with it. Plus, from the later rounds, the resume path's mark bookkeeping driven through a real restore rather than a hand-written mark: a stored id two areas both claim resolving nowhere; a translation that resolved through an entry a later area poisons; an untranslated mark resolved from the final mapping before any live-id coincidence is consulted; and a collision that must not launder a restored mark into a reader's hidden finding.len(a)+len(b)capacity hints — re-raised with new alert numbers each time the lines moved. Reviewed and dismissed as a diff-scan artefact, every thread resolved: both operands are lengths of one run's own in-memory slices, bounded by what an LLM returned for one area, and the identical pattern atphase_embed_index.go:242is unflagged on main.Merge before the enterprise PR, which pins
.platform-refto this branch while it is open.— Co-coded with Jale 🤖
🤖 Generated with Claude Code