QA quality round: measurement harness hardening, runtime evidence discipline, corpus truth lints

[kalepail]

What this lands

One full round of the QA quality deep dive (round ledger:
.agents/rounds/2026-08-25-qa-quality-deep-dive.md), built in three reviewed
worktree lanes and integrated with all gates green.

Measurement harness (from lane/measure-harness-20260825)

• R8: rejudge pins the collected runnerRevision by default; a new
  --cases-ref worktree mode keeps unpinned judging possible; the golden-time
  guard refuses live/scheduled goldens newer than the run unless
  --allow-golden-drift.
• R9: ungradeable answers skip rejudging. Covers both modern rows and legacy
  agent-error verdicts (regression test uses the real 2026-08-14 SSRF row).
• A1: opt-in --judge-panel 2|3 majority voting with error-as-abstention,
  worst-score ties, disagreement metadata, and full cost accounting.
• A6: deterministic trap-precedence consistency rule (narrowed refusal regex).
• A8: result metadata now reports half-credit, strict, coreAnswer share +
  null count, and mean continuous coverage.

Runtime product mechanisms (from lane/product-executor-20260825)

• B1: payload-shape guard — array-only reads on object payloads raise an
  error listing real keys (use r.data.hits); prototype chain preserved via
  an Object.create(base) bridge so instanceof keeps working.
• B2: host-owned provenance sidecar — exact-path allowlist capture
  (generatedAt, dataAsOf, roster/scf scheduling fields) emitted as a
  distinct --- SOURCE METADATA --- block that is NOT a loss boundary;
  model-returned marker collisions are escaped.
• B3: server-instructions evidence-discipline block (date volatile values,
  copy exact identifiers, scope absences, surface conflicts,
  broaden-or-abstain) within the 2,000-character budget (1,998).
• Eval consumers taught the metadata marker: packs gain an additive
  provenance: line; composition analysis parses cleanly.

Corpus truth work (from lane/corpus-goldens-20260825)

• Six authoring warning classes in lint-corpus.mjs (atomization, compound
  predicates, negative predicates, non-falsifiable avoid items,
  self-referential snapshot dates, missing symmetric cautions) —
  0 errors / 1,390 warnings, warning-severity only.
• Answering contract discloses the dating + exactness requirement.
• Symmetric cautions added where the new rule fires.
• WisdomTree CRDT golden repaired: issuer/SAC verified against the owner
  TOML and Horizon (dated receipts archived under
  research/qa-deep-dive-2026-08-25/receipts/); root cause corrected to a
  transcription defect; consistency register re-stamped with 7 dated
  cluster reopens whose reconciliation rides the same-100 rerun.

Review

Every branch was independently reviewed (Claude Fable; GPT-5.6 Sol high),
returned CHANGES-REQUESTED with verified findings, and every blocker plus
nine advisories were fixed before integration. Review reports are committed
under research/qa-deep-dive-2026-08-25/.

Gates (merged tree at 83b6ec8 + docs commit)

typecheck clean · 85 files / 1,265 tests · smoke 4 / 82 · build ok ·
eval:selftest pass · routing GATE PASS vs 2026-08-25 baseline · lint
0 errors / 1,390 warnings · secrets scan clean.

Follow-ups

Same-100-id rerun at this revision is running now; its comparison against
the 45/39/15 baseline and reconciliation of the reopened clusters land in a
follow-up round note.

[Copilot]

Pull request overview

This PR consolidates a QA quality round’s outcomes into the repo by hardening the QA measurement harness, strengthening runtime evidence/provenance handling in the executor, and adding corpus truth/authoring linting plus targeted golden repairs—supported by new/updated tests and round documentation.

Changes:

• QA harness: adds re-judge guards (pinned cases ref + golden-time drift refusal), optional judge panels, and richer measurement metrics reporting.
• Executor: introduces host-owned source-metadata sidecar handling (distinct marker), escapes host-marker collisions, and adds a payload-shape diagnostic guard for object-vs-array misuse.
• Corpus: adds warning-only authoring lints, updates answering contract phrasing, and repairs/annotates selected goldens with symmetric cautions and updated provenance.

Reviewed changes

Copilot reviewed 46 out of 46 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/source-basis.test.ts	Adds unit coverage for source-basis/metadata markers, metadata bounding/dedup, and marker-collision escaping.
test/smoke/executor.test.ts	Extends isolate smoke tests for source-metadata sidecar preservation and spoofing/escaping behavior.
test/re-judge.test.ts	Adds regression coverage for cases-ref resolution, golden-time drift refusal, and skipping ungradeable legacy rows.
test/qa-verdict-consistency.test.mjs	Adds tests for the trap-refusal consistency rule and rationale ambiguity handling.
test/qa-measure-harness.test.mjs	New tests for judge panels and measurement-metric computations/guards.
test/qa-judge-stored.test.mjs	Extends judge-stored tests for metric stamping and panel-mixing refusal behavior.
test/qa-corpus-lint.test.mjs	Adds coverage for new authoring-warning lint classes and expected positives/negatives.
test/mcp-instructions.test.ts	Verifies the bounded evidence-discipline clause is present in server instructions.
test/fixtures/qa-corpus/authoring-warnings.json	Adds fixture corpus cases that exercise the new authoring-warning lint rules.
test/executor-providers.test.ts	Adds tests ensuring only allowlisted source-metadata paths are captured.
src/policy/source-basis.ts	Defines markers, allowlisted source-metadata paths/types, collision escaping, and manifest serialization/bounding logic.
src/mcp/tools.ts	Updates BASE server instructions to include evidence-discipline guidance within the size budget.
src/executor/run.ts	Appends the appropriate host block (basis vs metadata), escapes marker collisions, and emits source-metadata sidecar when present.
src/executor/providers.ts	Captures allowlisted metadata from redacted payloads and installs the payload-shape diagnostic guard in the sandbox prelude.
research/qa-miss-analysis-2026-08-25.md	Documents miss taxonomy, evidence, and key conclusions from the QA deep dive.
research/qa-improvement-plan-2026-08-25.md	Records an evidence-backed improvement roadmap and gating strategy.
research/qa-deep-dive-2026-08-25/review-judge.md	Archives independent review findings for harness/corpus lanes and their reconciliation notes.
research/qa-deep-dive-2026-08-25/review-executor.md	Archives independent review findings for executor lane and their reconciliation notes.
research/qa-deep-dive-2026-08-25/receipts/wisdomtree-toml.headers	Captures dated HTTP headers for WisdomTree TOML receipt.
research/qa-deep-dive-2026-08-25/receipts/horizon-crdt.json	Captures Horizon asset lookup receipt used for golden verification.
research/qa-deep-dive-2026-08-25/receipts/horizon-crdt.headers	Captures dated HTTP headers for Horizon receipt.
research/qa-deep-dive-2026-08-25/kimi-k3.md	Includes external lane report (survey + cold-audit) used in synthesis.
research/qa-deep-dive-2026-08-25/build-product.md	Build/verification report for the product/executor lane including post-review fixes summary.
research/qa-deep-dive-2026-08-25/build-measure.md	Build/verification report for the measurement lane including post-review fixes summary.
research/qa-deep-dive-2026-08-25/build-corpus.md	Build/verification report for the corpus lane including post-review fixes summary.
eval/qa/verdict-consistency.mjs	Implements/refines trap-refusal consistency checks using tags and narrowed rationale parsing.
eval/qa/sample.json	Updates corpus digest for the deterministic sample.
eval/qa/run-qa.mjs	Adds judge panels, richer cost accounting, and metric computation/printing/stamping.
eval/qa/re-judge.mjs	Adds --cases-ref worktree, golden-time guard, panel support, legacy ungradeable-row detection, and cross-checkout path mapping.
eval/qa/lint-corpus.mjs	Adds warning-only golden authoring lint classes and integrates them into lint/live-contract lanes.
eval/qa/judge.mjs	Adds judge panel orchestration, shared ungradeable verdict construction, and passes tags into consistency checking.
eval/qa/evidence-pack.mjs	Teaches evidence pack splitting/serialization about SOURCE METADATA (non-loss boundary) and provenance line emission.
eval/qa/corpus/battery/tooling-infra/q-ti-rpc-gettransactions-pagination-xdr.json	Adds symmetric caution and updates verification metadata.
eval/qa/corpus/battery/tooling-infra/q-infra-horizon-vs-rpc.json	Adds symmetric caution and updates verification metadata.
eval/qa/corpus/battery/defi-ecosystem/q-defi-wisdomtree-crdt.json	Repairs issuer/SAC + provenance, adds symmetric caution, and updates root-cause attribution.
eval/qa/consistency-register.json	Re-stamps/reopens affected clusters after member content changes.
eval/qa/cases.json	Regenerated compiled QA battery with updated corpus digest and updated cases.
eval/qa/analyze-composition.mjs	Ensures JSON parsing cuts at earliest host footer (incl. SOURCE METADATA).
.agents/TODO.md	Tracks follow-up work to validate Stellar strkeys during golden imports.
.agents/rounds/2026-08-25-qa-quality-deep-dive.md	Adds the round ledger capturing timeline, lanes, decisions, and integration notes.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[kalepail]

Benchmark landed (ledger updated): same-100-id rerun at the round revision — 48 correct / 35 partial / 13 wrong / 4 errors → half-credit 65.5%, strict 48.0% (baseline 64.5/45.0). First readings from the new instruments: core-answer-correct 91.7%, mean continuous coverage 71.7%. All 17 per-case improvements match the round's predicted recoveries; regressions sit inside the known judge-noise band plus one retryable agent error_max_turns on the evidence-poor CRDT case. Run history, including two invalidated attempts with their root causes (mid-run repo write; stale dev server), is in .agents/rounds/2026-08-25-qa-quality-deep-dive.md.