Skip to content

Consolidate a run's findings before validation spends its budget on them (#555) - #556

Open
jale-dbox wants to merge 24 commits into
mainfrom
jale/issue-555
Open

jale-dbox wants to merge 24 commits into
mainfrom
jale/issue-555

Conversation

@jale-dbox

@jale-dbox jale-dbox commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Closes #555. The community half of pre-validation consolidation: the hook, the extracted pairing rule, the two proposal tests, the marks, and validation + Phase 5 honouring them. The adjudicating agent is enterprise and registers through the seam added here.

What changed

The hook. orchestrator.go, Phase 4 area loop, between step, insights = out.step, out.insights and checkpointAreaDraft. There and not a line either side of it, for three reasons: step.Insights and insights are the same slice, so a mark written there is in the draft the next line checkpoints — a resumed attempt restores the marks instead of re-deriving them and never pays the provider twice; it is before the stopAfterDraft pause seam, the first point a run can stop without having paid for validation; and it is inside the !fromDraft branch, so a restored draft's marks arrive with the draft and a completed area was continued long before.

One embedding-text derivation (models.InsightEmbeddingText). BuildEmbeddingText was a Sprintf over five fields that are all also on the in-flight insight. Two derivations of what a document is is how the mid-run pass and the end-of-run check come to disagree about one — the same discipline magnitudeOf exists for on the quantitative side.

The pairing rule, extracted into consolidate_cluster.go: the threshold, the sameClaim magnitude guard, and the canonical-chain re-check. markDuplicates now calls it. The existing markDuplicates suite — including ChainsDoNotAssumeTransitivity, ChainCannotRouteAroundTheMagnitudeGuard and SameWordingDifferentMagnitude — passes unchanged through the new path, which is the equivalence guard.

Figure containment, the second proposal test: every distinctive figure (three or more digits) of one headline appearing in another's body. Two or more figures required — a single shared three-digit number between two findings about the same warehouse is coincidence often enough to matter, and the cost of a wrong fold here is a finding that loses its verdict.

The seam (agentplugin.ConsolidationProvider), modelled on DiscoveryPolicyProvider: single provider, last registrant wins, panics recovered, and an error returns no verdicts rather than the partial set — a half-applied adjudication is the one state the write guard cannot reason about.

skipped_duplicate, its own verdict rather than reusing skipped_budget_cap. "There was nothing here the canonical does not already carry" is not "the run ran out of budget", and conflating them would make the measurement this feature exists to move unreadable from the stored verdicts. Added through the dashboard's five verdict surfaces too; it renders as Restated.

Also: llmlayer.LayerConsolidation, the per-area consolidation block in the run log (folds counted by which test proposed them), and the marks carried onto the standalone copy at denormalisation — the standalone document is what the API filters on, so a mark that stayed on the embedded copy would hide a finding on the dashboard and return it from the API.

Why the measurements point here

Four consecutive runs on one project, same playbook, same build:

run prior findings within-run folds ≥ 0.88 folds against an earlier run
1 0 4 / 56 — 7.1% —
2 56 17 / 55 — 30.9% 1
3 109 28 / 58 — 48.3% 0
4 166 20 / 47 — 42.6% 3

Cross-run restatement is solved — #458/#545 did their job. Within-run is not, and it is a step rather than a ramp: ~7% with no history, 43–48% once there is any. The ten areas are prompted independently, so two of them reporting the same claim is the dominant shape on any project with a ledger behind it.

The three guards, and why none of them is theoretical

Clusters are stars over direct relations, never connected components. Building them by transitive closure over containment — A contains B, B contains C, therefore one cluster — proposed suppressing 23 of 55 insights on real run output where the direct relations yield 9. A chain of four documents each 25° apart is the case that tells the two apart, and it is the test.

Every fold is re-verified against the canonical the mark will actually point at. Neither test is transitive: 0°, 25° and 50° score 0.906 between neighbours and 0.643 end to end, so matching the middle says nothing about the head. Two of nine hand-applied folds on one run were dropped by exactly this re-check.

The write guard is one transitive rule, not a set of per-case checks. Three hand-run passes produced three variants of the same bug — a fold whose canonical the same proposal also suppressed; a fold whose canonical a previous pass had already suppressed, which produced a mutual cycle in which neither finding had a visible canonical; and a pre-existing mark pointing at a document the pass suppresses. Each time the case just seen was guarded, the next variant appeared. After writing, every mark in the run is resolved to a final canonical and re-verified, and a pass that cannot prove its own marks gives them all back.

Limitations, stated

  • The pass marks only the area it is consolidating. An earlier area's finding is a legitimate canonical — usually the right one — but not a member that can be suppressed: its verdict is already paid for, its checkpoint is written, and a restore remints its ids. When the adjudicator prefers the later phrasing both findings are left standing, and the pair is remembered for the run so the end-of-run check does not then fold it in the direction the adjudicator rejected. That memory is per-attempt: an area restored from a completed checkpoint does not re-run its pass, so a resumed run can lose a refusal and get the end-of-run rule for that pair — today's behaviour, not a new failure.
  • A merged description lands only on a canonical in the area being consolidated. An earlier area's text is already recorded in that area's checkpoint, and rewriting it here would leave the record and the result disagreeing about what the finding says.
  • A restore re-points the marks it can and clears the ones it cannot. A mark carried over verbatim names a row this attempt does not hold, which renders the finding as suppressed behind a canonical the reader cannot reach — strictly worse than no mark. Marks inside the restored area always resolve, and those are the ones that cost the validation budget.

Verification

  • check-gate.sh (every module independently, -count=1): green apart from the three known-environmental failures on this box — libs/go-common/mongodb against the occupied :27099, and the two config tests against an exported MONGODB_URI.
  • make lint-go (golangci-lint v2.11.3): 0 issues across every module. make test-ui 807 tests / 54 suites, make lint-ui clean apart from the pre-existing _props warning, tsc --noEmit clean, make lint-docs clean. CI green.
  • The existing markDuplicates suite — ChainsDoNotAssumeTransitivity, ChainCannotRouteAroundTheMagnitudeGuard, SameWordingDifferentMagnitude — passes unchanged through the extracted rule. That is the equivalence guard for the extraction.
  • New tests on the failure cases, not the happy path: the non-transitive chain directly and through a paraphrase; the chain-of-four clustering; containment's whole-token comparison and its two-figure floor; the mark closure's mutual cycle, longer cycle, self-mark and broken pointer; provider error, panic, an id outside its own cluster, a canonical that is itself folded, a verdict that suppresses its own canonical, a non-positional response; a marked duplicate costing no budget with the counts reconciling, and a single-document request ignoring that skip; exclusion from Phase 5 for a confirmed duplicate, a verdict-less one and a skipped_duplicate one; the end-of-run check not re-deciding a consolidated document, not folding one another points at, and honouring a keep against the resolved target; the community pass seeing the provider's marks, chaining only through folds it could write, and deciding every group; the five cases where a rewrite cannot be applied; the vector cache paid once per text; and the restore re-pointing one mark, clearing another, and taking that mark's log entry with it. Plus, from the later rounds, the resume path's mark bookkeeping driven through a real restore rather than a hand-written mark: a stored id two areas both claim resolving nowhere; a translation that resolved through an entry a later area poisons; an untranslated mark resolved from the final mapping before any live-id coincidence is consulted; and a collision that must not launder a restored mark into a reader's hidden finding.
  • CodeQL raised five "size computation may overflow" alerts across the rounds, all on the same len(a)+len(b) capacity hints — re-raised with new alert numbers each time the lines moved. Reviewed and dismissed as a diff-scan artefact, every thread resolved: both operands are lengths of one run's own in-memory slices, bounded by what an LLM returned for one area, and the identical pattern at phase_embed_index.go:242 is unflagged on main.
  • 17 Codex review rounds, ending in a clean one. The logs raise 40 findings across rounds 1–16 and none in round 17; all are fixed bar one, which is stated as a WONTFIX with its reason in the final report. Rounds 12–16 found nothing in the original code: every finding was in an earlier fix of mine to the resume path's mark bookkeeping, which is worth knowing when reading that file — the version that stands replaces the mechanism rather than guarding it again. No Copilot pass: this repo has no Copilot auto-review configured.
  • Not yet verified on real data. Deliberate, and it is the next step: validation coverage of distinct findings (≥ 60%, from 31.0%), fold count by source, and token cost against the <1% target.

Merge before the enterprise PR, which pins .platform-ref to this branch while it is open.

— Co-coded with Jale 🤖

🤖 Generated with Claude Code

…s budget on them

The product already detected that half of a run's insights restate another
insight from the same run. It detected it last — at 97% progress, after the
validation agent had spent its fixed budget checking those restatements, and
after the recommender had made its selection on the result.

The decision now happens at the end of each analysis area, before that area is
validated. A marked restatement is skipped with no charge against the run cap,
and excluded from Phase 5 on the mark rather than on its verdict: a nil verdict
fails open, so filtering on the verdict alone would make consolidating a
finding promote it.

Two proposal tests run and are unioned, because measured on real run output
they find largely different duplicates — 18 similarity pairs, 16 containment
pairs, 3 in common. Containment is the one similarity cannot see: a finding
whose headline is one row of another's table scores 0.71-0.87, and four of
seven hand-applied folds on one run sat below the 0.88 floor.

Which member survives goes through a provider seam. Scored against a reviewer
over three runs, document order picks the wrong survivor most of the time:
applying one run's agreed folds required unmarking four of five chosen
survivors. With no provider registered the community rule is the one the
end-of-run check already applies, so a community deployment gets today's
marks, decided earlier.

Three guards that are load-bearing rather than defensive. Clusters are built
from direct pairwise relations: transitive closure over containment proposed
suppressing 23 of 55 insights where the direct relations yield 9. Every fold
is re-verified against the canonical the mark will actually point at. And the
write guard is one transitive rule, not a set of per-case checks — three
hand-run passes produced three variants of the same bug, each guarded in turn
and each followed by the next, the worst a mutual cycle in which neither
finding had a visible canonical.

A run still embeds each insight exactly once: the pass fills a per-run vector
cache keyed on the embedding text and the end-of-run index reads it.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

jale-dbox and others added 6 commits November 14, 2026 13:35
…ads one struct deeper

Four from a self-review before the review loop.

A merged description was gated on the RUN's fold counter, so once any cluster
had folded something, every later cluster's rewrite was applied to a canonical
nothing had been folded into. Gated on its own cluster's folds now.

Which test proposed a cluster was read from every relation touching any
member, not from the relations inside it. A member of a containment cluster
usually also resembles documents in a neighbouring star, so a
containment-only cluster was reported as "both" — and that field is the one
thing that says whether the similarity threshold could have seen the duplicate
at all. Two overlapping maximal stars is the case that tells the two readings
apart, and it is the test.

A restore that clears an unresolvable mark now clears the verdict that existed
only because of it. skipped_duplicate is stamped BY the mark, so a finding
visible again with that stamp still on it rendered as a restatement of
nothing. A verdict the verifier actually produced is untouched.

And the API's AnalysisStep mirror declares the consolidation summary. Its
class guard reads top-level int counters, so a whole nested block of them
could be persisted by the agent and dropped by the mirror with every test
green — the same failure the guard exists for, one struct deeper. The guard
now also requires every pointer-to-struct field, verified by removing the
field and watching it fail.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… and marks settle once

Codex round 1, four findings, all valid.

The end-of-run check could fold a document another finding already points at:
a mid-run mark B -> A is written and persisted, so folding A into an earlier
run's X left B pointing at a hidden row — and B cannot be re-pointed there,
because X belongs to another run and its vector is not in hand to re-verify
against. A document with dependants is now left alone and its cross-run fold
deferred.

A member the PROVIDER omitted from Duplicates is an explicit judgement that it
states a different claim, and nothing recorded it — so the end-of-run check
folded the pair anyway on document order, overriding the one decision this
feature exists to take. Those pairs are recorded now, and only for clusters
the provider decided: the community fallback's non-folds ARE the end-of-run
rule, and freezing those would change the community outcome rather than
preserve it.

Clusters are overlapping stars, so two verdicts can fold different members
into the same survivor — and each merged description replaces its text, so the
second erased what the first preserved while both folds stood. First rewrite
wins. And a verdict that asks to rewrite an EARLIER area's survivor now gives
its folds back rather than applying them without the rewrite: the rewrite was
asked for because the survivor is missing something the folded members carry,
so folding anyway is exactly how a fold loses a caveat.

Marks now settle once, after the loop. The run's analysis areas come out of a
map, so their order is not stable between attempts and an area that folded
into another can restore BEFORE it — clearing on the spot threw away a good
mark whose target had not registered its ids yet, and nothing revisited it.
settleRunMarks re-points those and clears only what the run genuinely does not
hold, on both the accumulator and each area's step.

Also adds DISCOVERY_CONSOLIDATION_ENABLED (default on), forwarded to the
agent. The similarity floor is a threshold and cannot express "do not
consolidate", and the containment test does not read it — so an operator who
wanted the previous behaviour had nothing to set.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found re-reading the round-1 fix. Reverting a cluster's folds cleared the
mark off the insight but left it in the chain map, so a later cluster
resolving its canonical through that map would resolve through a mark that
no longer exists and point a real fold at the wrong document — and
keepStanding read the pair as folded and recorded no keep for it, so the
end-of-run check would fold it the way the revert had just refused.

Also records the embedding request-count caveat and the new configuration in
the changelog: the token count is unchanged, the number of batches is not.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… a merged survivor

Two more from re-reading. A cluster given back decremented Folded but not the
per-test bucket it had been counted in, so the log line could read folded=0
beside by_containment=1 — and the fold rate per test is the one number that
line exists to make legible. Both directions go through one function now.

And a merged survivor is re-projected, so a later cluster folding into it runs
its containment re-check against what the survivor now says rather than
against the text it said before the rewrite. The vector is necessarily stale,
which is the conservative direction: a cosine re-check can refuse a fold that
would now hold, never accept one that would not.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e either

Codex round 2, four findings, all valid.

A mark into a restored area is now STORED by the id the run's checkpoints hold
rather than the one this attempt minted. A third attempt restores that area
from its own checkpoint, so it can only translate the ORIGINAL id — a mark
naming the second attempt's id resolved against nothing, and settleRunMarks
correctly cleared it, costing the fold and returning an unvalidated finding to
the recommender. Same discipline as Phase 5's stored citations, and only the
stored copy is translated: the live mark must keep naming the live id or the
dashboard renders a suppression pointing at an id no row carries.

Two overlapping stars can fold different members into the same survivor, and
the second rewrite was dropped while its folds stood — so that cluster's
members were suppressed without the detail the rewrite was asked for. Both
reasons a rewrite cannot be applied now get the same answer: the cluster's
folds go back and both findings stand.

A merge carries the folded members' SourceSteps with it. The verifier builds
its evidence bundle from the survivor's cited steps, so a merged claim whose
step the survivor does not cite would be checked against evidence that does
not contain it, and attachSourceQuality would miss the caveat that step
carries.

And the provider contract is positional and enforced. "Clusters may be
omitted" plus reading by position silently reverses decisions: a provider that
answers only the second of two clusters has its answer read against the first,
where it is rejected as naming a stranger, and the second then gets the
community rule — which may fold it the way the provider refused to. A response
whose length is not the cluster count is discarded whole.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jale-dbox jale-dbox added type: feature New feature or capability component: agent Agent service priority: P1 Critical — must do labels Nov 14, 2026
@jale-dbox jale-dbox added this to the v1.0.0 milestone Nov 14, 2026
The projection sent the supporting figures both as their own field and
appended to the body, which is prompt budget spent saying one thing twice —
and it made the 2,000-character truncation cut the prose to make room for a
repeat of the figures. The containment test still reads both, because a figure
listed as an indicator is a figure another finding can restate.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jale-dbox and others added 7 commits November 14, 2026 14:12
Each call's prompt is bounded by the layer's input window, but the NUMBER of
calls grows with the cluster count — so nothing bounded what the pass can
cost, and the feature's own acceptance target is under 1% of a run's tokens.

DISCOVERY_CONSOLIDATION_MAX_CLUSTERS (default 60, non-positive for no bound)
caps it. Deliberately generous: four measured runs proposed about thirty
candidate pairs per RUN, so sixty per area is far above anything observed, and
an area that reaches it is either a real finding about the playbook or a bug
in the proposal step — which is why it is logged and recorded on the area's
run-log row rather than passed over.

Clusters past the bound are not dropped. They get the community rule, which is
the decision the product makes today.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…gure-bearing survivor is not rewritten

Codex round 3, three findings, all valid.

The re-check that guards every fold accepts either test, and the community
fallback could satisfy it by containment — folding pairs at a cosine of 0.643
against a 0.88 floor, on a rule the deployment never opted into. Containment is
a proposal the adjudicator rules on; the fallback is now cosine-only, which is
what "the community outcome is unchanged" was supposed to mean.

A pair the adjudicator decided should stand is no longer folded by a fallback
verdict for an overlapping cluster. The clusters are stars and they share
members, so the same pair can appear in one cluster a provider answered and
another it declined — and the end-of-run check cannot correct it afterwards,
because it skips a document that already carries a mark.

And a survivor whose prose carries rendered evidence figures is not rewritten.
The figure template holds that text as authored with its {{fN}} references
intact, and a recommendation may restate a figure by naming its id — so
free-text replacement severs the link: every field stops matching its
re-render, which withholds a repaired insight's figures from the recommender
entirely, and on an unrepaired one a recommendation could restate a number the
survivor's prose no longer shows. The model is shown the RENDERED text, so it
cannot preserve the references, and re-deriving them from free text is the
numeral surgery the figure layer exists to avoid. The cluster is given back,
like the other two survivors that cannot be rewritten.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… fold the pair the community rule can see

Codex round 4, five findings: four fixed, one documented.

The keeps are registered before a single mark is written. Discovering them
incrementally was useless in the case that matters: the clusters are
overlapping stars processed in order, so a cluster the provider DECLINED can
be processed first and its fallback folds a pair a provider verdict for a
later, overlapping cluster explicitly keeps — and by then the mark is written
and the end-of-run check will not revisit a marked document. A pair's keep is
derivable from the verdicts alone, so it can be known first.

That exposed a tension worth recording. Two overlapping verdicts can disagree
about one pair, folded in one and simply unnamed in the other, and an explicit
fold now beats an omission: the fold is a positive judgement, while a member a
verdict did not mention may just be a member it did not mention. And a
canonical is resolved the way the writer resolves it, so a verdict naming an
already-suppressed canonical is read as folding into the end of that chain,
not into the row it named. A pair whose fold the writer then REFUSES is
registered at the refusal, because only the writer knows that.

The end-of-run check honours a keep against the target it RESOLVED to, not
only the candidate it matched. With b folded into a and the adjudicator having
kept a and c apart, matching c against b resolved back to a and suppressed c
anyway — the decision routed around by one hop.

A rewrite is refused when any member it folds was refused: it may incorporate
detail from a donor nobody folded, which would credit the survivor with a
claim whose evidence it does not cite, and the provenance union cannot carry
what was never folded.

And the community rule applies pairwise inside the cluster. A star's seed need
not be in every pair, so comparing everything against it folded nothing on a
mixed star and both later members spent validation budget before the
end-of-run check folded one into the other anyway.

Documented rather than built: a mark into an area whose completed-checkpoint
write FAILED, recorded against the surviving draft's id, cannot be resolved by
a third attempt. It is cleared rather than left broken, so the finding is
visible and unvalidated — which is what it would have been before this phase
existed. Persisting an alias for that path is a change to the checkpoint
schema for a two-failure case.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…community rule decides every group

Codex round 5, three findings, all valid.

An operator clicking Validate on a finding the dashboard shows as restated had
their request turned into a permanent no-op: the run-wide duplicate skip
applied to the single-document path, so the job returned skipped_duplicate
without running either agent and overwrote whatever verdict the document had.
The skip exists so a RUN's fixed budget is not spent on restatements, and
someone who clicked the button has already decided to spend it.

A rewrite is refused when it names a donor this pass did not fold because a
previous area had ALREADY folded it. The rewrite credits the survivor with
that donor's detail while the provenance union carries only what this verdict
folded, so the survivor would state a claim whose evidence it does not cite.

And the community rule decides every similarity group in a cluster, not just
the first. One maximal star can hold two independent groups joined only
through a finding that contains both by figure — and buildClusters removed the
smaller stars as subsets, so the second group got no pre-validation decision
at all: both its members spent validation budget and the end-of-run check
folded one into the other afterwards anyway. A cluster can now carry several
verdicts, which is also the shape the community path always wanted.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rotection that lifts when its fold does

Codex round 6, three findings, all valid.

Containment compared SUBSTRINGS: a headline stating 123 and 456 was contained
in a body stating 1123 and 2456 — two different measurements reading as the
same one, and on a pair below the similarity floor the fold costs a distinct
finding its verdict and its place in Phase 5 on nothing at all. Both sides go
through the same figure parser now and the comparison is token against token.

The community rule is one pass over the run in document order rather than a
decision per cluster. Two reasons, and it took three rounds to see both: the
clusters are overlapping stars, so a per-cluster pass carried state that
depended on which star came first — on two overlapping stars whose middle
members are all similar, the first made one of them a canonical and the second
then refused to fold it into the earlier member it actually restates, leaving
two redundant findings standing. And the rule has to MATCH what the end-of-run
check would have made, because the promise to a community deployment is that
only the moment of the decision moves; that check is one global pass, so this
is the same function over the documents in hand.

A document is protected from suppression while this pass's marks point AT it,
and the protection now lifts when they are given back. It was a flag, so a
document that was briefly a canonical and then lost its only dependant stayed
protected for the rest of the pass and a later valid fold of it was refused.

And a cluster given back for an unapplicable rewrite now protects its members
against every other member, not only the survivor the verdict named: the
reason for refusing is that the member carries detail the survivor lacks, and
that is just as true of any neighbour it might be folded into instead.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…only through folds that can be written

Codex round 7, two findings, both valid.

When EVERY fold a verdict proposed is refused by the re-check there was
nothing to revert, so the protection branch never ran — and the members the
verdict named still carry detail the survivor lacks, so the community pass
then folded them sideways into each other and the caveat the rewrite existed
for was lost through a different door. The protection is now independent of
whether anything was written, and it covers every member the verdict named
rather than only the ones that got as far as a mark. Applying the rewrite
still requires a fold: a rewrite with nothing folded into the survivor loses
detail rather than adding it.

And the community chain is built only from folds this pass could actually
WRITE. Recording a fold a keep forbids, or one whose document this pass cannot
mark at all, made a later document's chain re-check resolve against a canonical
the reader is NOT being sent to — so a document that plainly restates the
visible finding was left standing and spent a validation check, before the
end-of-run rule folded it there anyway.

Three tests now assert the OUTCOME rather than a refusal counter, which is
what they were always about: a pair the chain never proposes is not a pair
that was proposed and refused.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rote

Codex round 8, one finding, valid.

The fallback was simulated against the marks the documents ARRIVED with, not
the ones the provider verdicts had just written. With the provider folding b
into the later c and declining the overlapping cluster, the simulation still
thought b was visible: it folded c into b, and a later d then failed its
re-check against b — while c, which d plainly restates at 0.906, had been
visible the whole time. d spent a validation check and could reach
recommendation generation before the end-of-run rule folded it into c anyway.

Seeded from the live marks now, and a document this pass has already sent a
reader to is skipped as a fold source for the same reason an already-marked
one is: its mark would never be written, so nothing may resolve through it.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jale-dbox and others added 2 commits November 14, 2026 15:59
…nd a donor's figures are not left orphaned

Codex round 9, two findings, both valid.

A rewrite is written against the text of the document the verdict NAMED. If
that document has itself been folded, the mark resolves somewhere else — and
applying the rewrite to the resolved survivor overwrites ITS caveats with a
rewrite of someone else's prose. The named document need not even have been in
that cluster's payload, and when the first fold needed no rewrite nothing else
caught it.

And a rewrite is refused when a FOLDED member declared evidence figures. They
were rendered into that member's prose against {{fN}} references the model
never saw and cannot reproduce: transferring the declarations leaves figures
with no placeholder in the survivor's text, which the figure layer withholds
anyway, and leaving them behind puts the number in the survivor's prose with
nothing a recommendation can reference it by — so ref and sum over that
measurement stop working while the member that owns it is suppressed. The
member stays visible instead, with its figures intact. A fold with no rewrite
is unaffected.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ed mark takes its log entry with it

Codex round 10, two findings, both valid. The first corrects a claim I had
written into a comment.

A rewritten survivor now LOSES its cosine projection. The comment said keeping
the pre-merge vector was the conservative direction; it was not. A rewrite can
REDUCE similarity below the floor, and the community pass would go on folding
against the old vector — suppressing a finding that no longer restates this
one, and costing it its verdict and its place in Phase 5. The merged text
cannot be embedded mid-area, so there is no cosine answer to give, and no
answer is the conservative one. Containment still works over the new body, and
Phase 9 embeds the merged text and decides on a real vector.

And clearing a restored mark now clears the validation-log entry that existed
only because of it. The step's ValidationResults are persisted by the
split-log writer and served by the validation-results endpoint, so leaving the
entry behind labelled a finding that is visible again, and was never checked,
as Restated — on the one surface an operator goes to in order to ask what WAS
checked. Only a skip-ONLY entry is removed: one carrying agent detail is a
verdict that was actually produced.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jale-dbox

jale-dbox commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Important

Superseded. This was written when the review loop stood at its 10-round
cap. The cap was lifted and the loop ran to a clean round at 17; the
paragraph below headed "The loop reached its cap rather than a clean round"
no longer holds — both fixes it asks a human to look at were reviewed in the
rounds that followed, and so were the rounds after that. Read the later
final report instead.

Final report

What. The community half of pre-validation consolidation: a new Phase 4.4 inside the Phase 4 area loop, the pairing rule extracted so one derivation serves both callers, figure containment as a second proposal test, the agentplugin.ConsolidationProvider seam, and validation plus Phase 5 honouring the marks. Plus llmlayer.LayerConsolidation, a skipped_duplicate verdict through the dashboard's verdict surfaces, and two switches.

Why. validateInsights charges one run-wide counter against MaxInsightsPerRun while Phase 4.5 runs inside the area loop, so the first areas spend the budget and every area after them is stamped skipped_budget_cap — and filterEligibleInsights then drops a capped insight from recommendation generation too, because that status is neither validation_disabled nor in the eligibility set. The end-of-run duplicate check finds the restatements at 97% progress, when both decisions have been made.

Measured over four consecutive runs of one project, same playbook, same build: within-run restatement is 7.1% with no recorded history and then 30.9% → 48.3% → 42.6% once there is any. A step, not a ramp, and untouched by the prior-context work — the ten areas are prompted independently. Cross-run restatement is 0–3 folds a run and can be considered done.

How, and the four things that are load-bearing rather than defensive.

  • Two proposal tests, unioned. On one run the shipped similarity rule proposed 18 pairs and a figure-containment scan 16, and only 3 were the same pair. Containment exists because these findings carry markdown tables and the commonest duplicate is a finding whose headline is one ROW of another's — 0.71–0.87 to cosine, invisible to a 0.88 rule. It compares whole numeric tokens through the same parser on both sides; the first version compared substrings and read a headline's 123 and 456 as contained in a body's 1123 and 2456.
  • Clusters are stars over direct relations. Transitive closure over containment proposed suppressing 23 of 55 insights where the direct relations yield 9. A chain of four documents 25° apart is the case that tells the two apart, and it is the test.
  • Every fold is re-verified against the canonical the mark will actually point at, by the test that proposed it — and containment only for a cluster an adjudicator decided, because containment is a proposal and not a community rule.
  • The write guard is one transitive rule. Three hand-run passes produced three variants of one bug, each guarded in turn and each followed by the next, the worst a mutual cycle in which neither finding had a visible canonical. After writing, every mark in the run resolves to a final canonical and is re-verified; a pass that cannot prove its own marks gives them all back.

With no provider registered the outcome is unchanged — and that claim is now exact rather than approximate. The community rule is one pass over the run in document order, which is the end-of-run check's own rule run early rather than an imitation of it. It took three rounds to get there: per cluster, then pairwise within the cluster, then every group within the cluster, before accepting that overlapping stars make any per-cluster state order-dependent. The global pass is also less code.

Verified.

  • check-gate.sh (every module independently, -count=1): green apart from the three known-environmental failures on this box — libs/go-common/mongodb against the occupied :27099, and the two config tests against an exported MONGODB_URI.
  • make lint-go (golangci-lint v2.11.3): 0 issues across every module. make test-ui 807 tests / 54 suites, make lint-ui clean apart from the pre-existing _props warning, tsc --noEmit clean, make lint-docs clean.
  • The existing markDuplicates suite — ChainsDoNotAssumeTransitivity, ChainCannotRouteAroundTheMagnitudeGuard, SameWordingDifferentMagnitude — passes unchanged through the extracted rule, which is the equivalence guard for the extraction.
  • New tests on the failure cases: the non-transitive chain directly and through a paraphrase; the chain-of-four clustering; containment's whole-token comparison and its two-figure floor; the mark closure's mutual cycle, longer cycle, self-mark and broken pointer; provider error, panic, an id outside its own cluster, a canonical that is itself folded, a verdict that suppresses its own canonical, a non-positional response; a marked duplicate costing no budget with the counts reconciling, and a single-document request ignoring that skip; exclusion from Phase 5 for a confirmed duplicate, a verdict-less one and a skipped_duplicate one; the end-of-run check not re-deciding a consolidated document, not folding a document another points at, and honouring a keep against the resolved target; the community pass seeing the provider's marks, chaining only through folds it could write, and deciding every group; the vector cache paid once per text; and the restore re-pointing one mark and clearing another.
  • CodeQL raised two "size computation may overflow" alerts on len(a)+len(b) in a capacity hint. Reviewed and dismissed as a diff-scan artefact: both operands are the lengths of one run's own in-memory insight slices, and the identical pattern at phase_embed_index.go:242 is unflagged on main.
  • Not yet verified on real data. Deliberate, and it is the next step: validation coverage of distinct findings (≥ 60%, from 31.0%), fold count by source, and token cost against the <1% target.

Review. 10 Codex rounds — the loop's hard cap — 29 findings, 28 fixed and 1 documented. A third of the later ones were my own earlier fixes overshooting or stopping one step short, which is worth saying plainly. The ones that mattered most: the dashboard was changed to offer Validate on a restated finding and the run-wide duplicate skip then made that request a permanent no-op that overwrote the document's verdict; a merged description silently invalidated the figure→prose link the recommender reads, so a repaired insight's figures were all withheld; provider keeps had to be registered before any mark was written, because a fallback cluster processed first could fold a pair a later provider verdict kept; and the API's analysis-step mirror dropped the whole nested counter block, which its own class guard could not see until it was taught to. No Copilot pass: this repo has no Copilot auto-review configured.

The loop reached its cap rather than a clean round, and that matters for how to read this. Round 10 found two defects and both are fixed, but those two fixes were not themselves put through another round. They are worth a human eye specifically:

  1. A rewritten survivor now loses its cosine projection (consolidate_phase.go, the merge branch). The comment there previously claimed that keeping the pre-merge vector was conservative. It was not: a rewrite can reduce similarity below the floor, and the community pass would go on folding against the old vector. The merged text cannot be embedded mid-area, so there is no cosine answer to give; containment still works over the new body, and Phase 9 embeds the merged text and decides on a real vector.
  2. Clearing a restored mark now drops the skip-only validation-log entry that existed because of it (analysis_resume.go, dropSkippedDuplicateResult). Those entries are persisted by the split-log writer and served by the validation-results endpoint, so one left behind labelled a finding that is visible again, and was never checked, as Restated. Only a skip-only entry is removed; one carrying agent detail is a verdict that was actually produced.

Every round produced real defects rather than theory, which is why the loop ran to the cap rather than stopping early. Round-by-round, the share that were my own earlier fixes overshooting or stopping one step short rose toward the end — the shape worth recording is that a guard patched per case grows a new variant each time, and the rule that ends it is transitive.

Documented rather than built. A consolidation mark into an area whose completed-checkpoint write failed is recorded against the surviving draft's id, and a third attempt cannot resolve it. settleRunMarks clears it rather than leaving it broken, so the finding is visible and unvalidated — which is what it would have been before this phase existed. Persisting an alias for that path is a checkpoint-schema change for a two-failure case.

Merge order. This merges first; decisionbox-io/decisionbox-enterprise#467 pins .platform-ref to this branch and sets it back to main in its last commit before merge.

Project board 4 could not be updated: addProjectV2ItemById is 403 for this token.

— Co-coded with Jale 🤖

jale-dbox and others added 3 commits November 14, 2026 16:33
…rop a fold's justification

Codex round 11, two findings, both valid — and both an invariant already
enforced elsewhere not being enforced on one path, which is why they are worth
fixing rather than guarding around.

The magnitude guard reached only the cosine acceptance. sameClaim refuses a
fold where the two texts are IDENTICAL and the figures differ — 100 affected
entities hidden behind 1,000 — and containment accepted without passing it, so
two otherwise identical findings were folded whenever the headline's figures
appeared in the other's body. It now gates both, which refuses only the
identical-text case and leaves every ordinary containment fold alone.

And a rewrite is refused when it drops the figures a fold already written into
that survivor rests on. Only a CONTAINMENT fold can be broken that way — it is
justified by figures in the survivor's body, while a cosine fold rests on the
vectors — and the closure check cannot catch it, because it verifies where
marks POINT rather than whether the claim still holds, and the end-of-run
check skips a document that already carries a mark. The finding would stay
suppressed and unvalidated with its claim stated nowhere.

Also collapses the merge guard, which had grown to six conditions each written
TWICE — once to select the branch and again to name it, which is how a guard
this long stops being checkable against itself. One function returns the
reason or none, and the reasons group into three: the rewrite is about text
that is not what this survivor says; the text cannot be replaced because
something else derives from it; or the rewrite would lose something it claims
to carry. No behaviour change — and the test suite caught the one thing that
did change, since the old `case a, b, c:` short-circuited before dereferencing
a slot that is -1 on an earlier area's document.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… just this cluster's

Codex round 12, one finding, valid — and it corrects the INPUT to last
round's guard rather than adding a condition to it.

The survivor can already carry folds from an overlapping cluster of the same
pass, or from a previous area's. A rewrite that keeps THIS cluster's figures
while dropping an earlier dependant's left that finding hidden behind a
survivor which no longer states its claim — the same defect, one cluster
further out. The guard now reads the live mark set, which holds both the
arrival marks and everything written since, so 'every fold into this
survivor' is answerable rather than approximated by the cluster it happens to
be looking at.

That also removed a parameter: the check no longer needs the insight slice or
the id index, only the docs and the marks.

Refs #555

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… model-supplied id is not provenance

Three findings from review round 13, two of which rest on the same premise:
an insight's id is the MODEL's when the model supplies one — the parse mints
a UUID only for an insight that arrived without one — so ids are not unique
across areas or across attempts.

- A merge rewrite is refused outright for any dependant the verdict does not
  itself fold. The rewrite was written from this cluster's payload, which
  does not contain that finding's text, so there is no sense in which it
  carries its claim and no way to check: a cosine justification cannot be
  re-established against prose that has not been embedded. For the verdict's
  OWN dependants a similarity justification is trusted (the adjudicator saw
  the text; that is what MergedDescription is for) and a containment
  justification is re-checked. This replaces a weaker rule that skipped every
  cosine-justified dependant, which let a rewrite drop an earlier cluster's
  claim and leave that finding hidden behind a survivor that no longer stated
  it.

- Two restored areas carrying the same stored insight id no longer resolve
  last-writer-wins, which pointed a mark at whichever area restored second —
  a reader sent to an unrelated finding while the one they asked for stayed
  hidden. The entry is poisoned instead: the fold is lost and both findings
  stay visible, which is the recoverable direction.

- A restored mark now needs provenance. Matching a live id is evidence for a
  mark this attempt's consolidation pass wrote, and a coincidence for one
  that came out of a checkpoint: a re-analysed area can hand back a previous
  attempt's id for a different finding. A restored insight's mark survives a
  live-id match only if the target is also an id this attempt minted for a
  restored area; any other match is cleared and the finding is visible again.

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jale-dbox and others added 4 commits November 14, 2026 17:10
…kpoint's does

The id mapping has two write sites, and the previous commit only hardened
one. A draft-restored area's remap still merged last-writer-wins, so a draft
id colliding with a completed checkpoint's silently overwrote it — the same
hazard, for the same reason: an insight id is the model's when the model
supplies one.

A call-site routing; the behaviour it routes to is covered by
TestRegisterRestoredIDs_ASharedStoredIDResolvesNowhere. Reaching this line
from a test means standing up the whole Phase 4 loop, which needs a
warehouse and an LLM.

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d mark

Round 14, and the finding is the previous round's fix overshooting in the way
this guard keeps overshooting: I derived "did this attempt mint this id for a
restored area" from the values of the id remap, to avoid threading a second
map through every restore. The remap poisons a stored id two areas claim —
which replaces those two areas' fresh ids in the value set, so the insights
most likely to carry a colliding mark were exactly the ones the trust rule
stopped protecting. A stale mark from either then rode a live-id coincidence
into a reader's hidden finding, with its skipped_duplicate verdict intact and
the finding excluded from recommendations.

The two questions are now two maps on a `restoredIDs` value, which is what
the restores and the end-of-run settle thread between them instead of a bare
map: `freshByStored` resolves a mark's target and is poisoned on a collision,
`minted` records every id this attempt minted for a restored area and is
not. Neither derives from the other, so there is no hole to find.

The regression is pinned twice: fromRestore must survive a collision on the
stored id, and a settle over a poisoned pair must clear the mark rather than
hide the finding. Both fail against the derived-from-values version, the
second with the exact symptom above.

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…h, once

Round 15, two P2s, both the same structural flaw and both mine: the mark is
translated eagerly at restore time, and then the settle reasoned about what
the mark SAYS instead of what it ARRIVED as.

The translation runs against a mapping the loop is still writing — a restore
happens in the same pass over the areas as a fresh analysis, so areas later
in that order have not registered yet. So it can resolve a mark through an
entry a later area then poisons, which left a finding suppressed behind an
unrelated canonical that was live and minted and therefore passed every check
the settle made. And it can fail for a canonical that registers a moment
later, whose stored target a freshly analysed area may happen to carry as a
live id — on which the settle cleared a mark the final mapping resolves
perfectly well, costing a real restatement its skip verdict and sending it to
the recommender unvalidated.

One rule for both: `repointRestoredMarks` records the stored target each mark
arrived with, and `settleRunMarks` re-decides every such mark from that value
against the final mapping. The translation becomes what it always was — a
convenience for the mid-run consolidation pass, which needs live ids to tell
which earlier-area documents are already canonicals, and whose worst case is
a fold it declines to make. This REPLACES round 14's minted set rather than
adding to it: the arrival target answers both "did this mark come out of a
checkpoint" and "what did it mean", so there is one source of truth instead
of two that can disagree.

Safe because the pass never writes a mark onto a restored insight: a
completed area is `continue`d long before Phase 4.4 and a draft-restored
area's marks arrive with the draft, so a restored insight's mark is always
the checkpoint's.

Four tests now drive the real restore path rather than hand-writing a mark,
because the arrival target recorded on the way past is the thing under test.
Both new cases fail against the round-14 code with the exact symptoms above.

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Round 16 found no correctness issues and one documentation gap, which was
real: neither switch appeared in the configuration reference, so an operator
had no way to discover either the master switch or the cost ceiling.

Both rows state the default, what the setting buys, and what a non-obvious
value does — for the ceiling, that a non-positive value means no ceiling and
is treated as a deliberate choice rather than a typo, and that clusters past
it fall through to the same rule Phase 9 uses rather than being dropped. The
enabled row also says plainly what leaving it on costs: a mark here takes the
finding's verdict and its place in recommendation generation, where before
this feature it only hid a row behind an expandable canonical.

Not added to docker-compose.yml, the other half of the finding: no
DISCOVERY_* setting appears there, so the convention in this repo is that the
discovery switches live in the reference alone. Adding two of eighteen would
make the compose file look authoritative about a family it does not cover.

Co-authored-by: Jale <jale@decisionbox.io>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jale-dbox

Copy link
Copy Markdown
Contributor Author

Final report — the review loop closed clean

This supersedes the 16:16 report, which was written at the loop's old 10-round cap. The cap was lifted; the loop ran to round 17, which raised nothing. What the feature is and why, the four load-bearing design calls, and the measured numbers behind them are unchanged and still read correctly there — this report covers what the extra seven rounds found and where the PR now stands.

Where it stands. check-gate.sh green apart from the three known-environmental failures on this box (libs/go-common/mongodb against the occupied :27099, both config tests against an exported MONGODB_URI). make build clean, make lint-go 0 issues across all 19 targets, make lint-docs clean, CI green on f582237. The enterprise half builds and passes against this branch's HEAD.

Rounds 11–17: seven rounds, five findings, every one of them in a fix of mine rather than in the original code. All five were in the resume path's consolidation-mark bookkeeping, and they came in a chain — which is the part worth a reviewer's attention, because it says something about that file specifically.

The premise under four of them: an insight id is the model's when the model supplies one. The parse mints a UUID only for an insight that arrived without an id, so ids are not unique across areas or across attempts. I verified that before acting on the first finding rather than taking it on trust, and it holds.

  1. A merge rewrite is now refused for any dependant the verdict does not itself fold. The rewrite is written from one cluster's payload, so it cannot speak for a finding whose text was never in it — and a cosine justification cannot be re-established against prose that has not been embedded.
  2. Two restored areas handing back the same stored id stopped resolving last-writer-wins, which had pointed a mark at whichever area restored second.
  3. A restored mark gained a provenance requirement, because matching a live id is evidence for a mark this attempt wrote and a coincidence for one out of a checkpoint.
  4. …except I derived that provenance from the id remap's values, and the remap poisons a colliding stored id — so the two areas most likely to carry a colliding mark were exactly the two the rule stopped protecting.
  5. …and the eager translation itself was the real problem: it runs against a mapping the loop is still writing, so it can resolve a mark through an entry a later area poisons, and fail for a canonical that registers a moment later.

The version that stands removes the mechanism instead of guarding it again. repointRestoredMarks records the stored target each mark arrived with, and settleRunMarks re-decides every checkpoint-borne mark from that value against the final mapping. The translation becomes what it always was — a convenience for the mid-run pass, which needs live ids to tell which earlier-area documents are already canonicals, and whose worst case is a fold it declines to make. That replaced finding 3's and 4's bookkeeping rather than joining it: one source of truth instead of two that can disagree. It is safe because the pass never writes a mark onto a restored insight — a completed area is continued long before Phase 4.4, and a draft-restored area's marks arrive with the draft.

Honest note on that chain. Four consecutive rounds found defects in my own guard, each a per-case patch that exposed the next variant. The signal was there by round 13 and I kept patching until round 15. The same lesson had already been recorded earlier in this PR about the write guard, where one transitive rule ended three rounds of per-case checks; I did not apply it to the resume path until the reviewer had found the same shape twice more. Nothing is left outstanding from it, but if one file in this diff deserves a slow read, it is analysis_resume.go.

Verification added, not just assertions changed. Every one of these fixes has a test that fails against the code it replaced, checked by running the new tests against the old product file rather than by reasoning about them. Two of those reproductions print the exact symptom the reviewer described — a mark left naming an unrelated canonical, and a good mark cleared along with its skip verdict. The four mark tests now drive a real restore (register, then re-point) instead of hand-writing a mark, because the arrival target recorded on the way past is the thing under test; the old fixtures encoded the mechanism that just went away.

Round 16's finding was a documentation gap and it was real: neither DISCOVERY_CONSOLIDATION_ENABLED nor DISCOVERY_CONSOLIDATION_MAX_CLUSTERS was in the configuration reference. Both now have rows with their defaults and the non-obvious semantics — that a non-positive ceiling means no ceiling and is treated as a deliberate choice, that clusters past it fall through to the Phase 9 rule rather than being dropped, and what leaving the switch on costs, since a mark here takes the finding's verdict and its place in recommendation generation. I did not add them to docker-compose.yml, the other half of that finding: no DISCOVERY_* setting appears there at all, so adding two of eighteen would make the compose file look authoritative about a family it does not cover.

Still outstanding, unchanged.

  • Not verified on real data. Deliberate and next: validation coverage of distinct findings (≥ 60%, from 31.0%), fold count by source, token cost against the <1% target.
  • One finding documented rather than built — a mark into an area whose completed-checkpoint write failed — with its reasoning in the superseded report.
  • Project board 4 could not be updated: addProjectV2ItemById is 403 for this token.
  • Merge order: this first; decisionbox-io/decisionbox-enterprise#467 pins .platform-ref to this branch and sets it back to main in its last commit before merge.

— Co-coded with Jale 🤖

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component: agent Agent service priority: P1 Critical — must do type: feature New feature or capability

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consolidate a run's findings before validation spends its budget on them: the community seam

3 participants