[Wenjie Fan (gggdttt)]

Current state

Draft, intentionally unmerged and unpublished. User-authorized scoring and isolated-review implementation is complete at d581548dada27cbab1a54dd38602f9778b638b72. The current experiment is no BCQuality versus narrowed new BCQuality, not original knowledge versus new knowledge. Neither new full evaluation has started; exact-head CI and final launch approval remain required. No additional trials, automatic retries, merges or publication are authorized.

The dataset remains byte-identical to cd148da4dd8247d03bcd73b0f25b1db8359d3322: SHA256 42178a85e9e8ce023f32ef030a95f2c2829f76268d47cac47c8beafd577a8a30, 145 entries / 300 records = 235 semantic + 56 analyzer + 9 compiler. Semantic scope is 218 Medium-or-higher and 17 Low. No gold, severity, source fixture, or atomic split was changed by the scoring commit.

Implemented scoring contract

• Precision TP counts generated comments with any concrete required-positive match, once per comment. A correct positive part suffices even if extra claims are wrong or unverified: this is deliberately optimistic, gold-relative matched-comment precision, not whole-comment correctness.
• Recall counts unique covered required gold. One comment can cover three golds; duplicate comments can each earn precision TP but never add repeated gold coverage. Coverage-redundant output is reported separately.
• Every normalized-same-file pair is judged before scoring, without line-distance cutoff, nearest-only pruning or one-to-one filtering. Protocol specific-site-v2 requires the same concrete function/statement/site, supplies start/end ranges and the full source diff with CLI-safe escaped newlines. A CSV-only finding cannot cover the vendor loop.
• Positive matches override neutral ones. All neutral-only outputs are excluded, including duplicates; mechanical and Low scope definitions remain explicit. Generated severity/kind cannot hide unmatched output.
• Results, serialization, summaries, exports and aggregates separate comment TP from covered_gold_count and preserve complete positive/negative judge-edge evidence. Four views are retained: any-match and legacy one-to-one, each Medium-aligned and all-severity semantic.
• legacy_result is legacy assignment on the SAME NEW JUDGE EDGES, not reproduction of the historical judge prompt. Old stored results remain readable as one-to-one/same-issue-v1 and are not rewritten. Mixed scoring identities are rejected.

Gold should contain one independent repair obligation, not separately count cause/impact/remedy or repeated descriptions. The read-only 235-gold review's tentative performance-001 projection split remains unapplied because applicability/severity are unresolved. Multi-gold unit fixtures do not add speculative benchmark gold.

Two-arm execution contract

Use this exact Bench commit for both full 145-entry runs, repeat=1, test-run=false, modified-only=false, model gpt-5.6-sol, native CLI 1.0.88 with updates disabled, judge gpt-5.3-codex, and Medium output treatment/scoring scope. Always supply nonempty engine-sha=25817a39a67b9e883f1a8ea9624283ab2b41af9b for the existing no-publication guard.

Arm	Entry point	Knowledge and topology
review-integration=none	bcbench evaluate copilot --category code-review --isolated-review	No BCQuality preparation or engine execution; one generic gpt-5.6-sol reviewer invocation
review-integration=bcquality	bcbench evaluate pr-review	Engine 25817a39a67b9e883f1a8ea9624283ab2b41af9b -> BCQuality 41b0b7a39f890d904fd00f3a5c61582573c9adf8, version 1.9; 19 serial gpt-5.6-luna leaves plus gpt-5.6-sol root

This is a whole-integration ablation, not proof of the effect of five article edits alone. The audit-derived dataset is not held out; one pair establishes neither significance nor generalization. Dependencies remain unmerged microsoft/BC-ALAgents#83 and microsoft/BCQuality#250. The former nine-article candidate is superseded by the narrowed candidate.

Genuine plain-review isolation

The opt-in path uses disposable HOME/configuration and an environment allowlist, disables custom instructions/hooks/IDE connection, disables all discovered skills, requires an empty plugin inventory and excludes skill/task tools. CLI discovery/disable/version commands were exercised with actual 1.0.88 without model calls. Standard harness defaults are unchanged.

All ten BCApps fixture bases contain .github/agents. With explicit user approval, only the GitHub-hosted job's disposable GITHUB_WORKSPACE/testbed may quarantine that exact clean tracked directory outside the checkout. Exact index paths receive temporary skip-worktree flags so ordinary git diff HEAD remains byte-identical, including the final CLI prompt's tool-command path; this is not a display filter. finally restores bytes/paths/index flags. Configuration/source manifests and before/isolated/after diff hashes are recorded. Dirty config, unrelated conflicting configuration, live/local checkouts, source contamination and restoration failure fail closed. No user configuration or AL/JSON fixture is deleted.

Before/after source inventories and bounded status/usage/provenance are uploaded separately from result JSONL. No raw prompt, environment, transcript or failure text is added. These observations establish configured/discovered sources, not omniscient proof of every model file read. Mock summary is retained only one day: download promptly.

Both runs remain unpublished to Braintrust, Kusto and leaderboard. The plain arm cannot requeue; enabled repeat=1 adds no next round. Report all-selected official metrics and symmetric complete-case sensitivity separately, preserving complete/partial/failed states and unknown failure reasons. Inspect actual output/logs and trace improvements, regressions and granularity/location cases to comments and source before final conclusions.

Preserved dataset repair and per-row evidence

The earlier three body-only corrections remain: performance-007's conditional update-lock contention no longer claims all readers block; performance-015 permits modification with FindSet(false) without invented mandatory per-row network costs or measured speedup; style-016 removes a duplicate AA0216 citation. Counts/kinds/severity/locations/fixtures are unchanged.

• 97 original FN ledger: 22 mechanical FN remain typed analyzer records, not deleted answers (17 AA0216 and one each AA0248, AA0018, AA0013, AA0001, AA0137).
• 322-comment integration ledger: 80 missing-gold proposals = 43 accepted + 1 already covered + 36 deferred, including all eight XML/public-API policy questions. Original IDs, hashes, audit axes and limitations remain; unresolved findings are not manufactured TPs.
• Classification, mappings and before/after evidence: current positions are array indexes, not immutable IDs. The recorded nine-article boundary review is historical alignment evidence, not a statement that the narrowed candidate still changes nine articles.

Prior repairs and retirement evidence remain in those ledgers, including unsafe mandatory ModifyAll advice, internal RecordRef visibility, BOM validation wording, used Err label, Temporary/Pending O365 Contact, and duplicate false-delivery-success gold. These are data adjudications, not blanket validation of unmatched findings.

Historical results are immutable, not the new baseline

Run 39773340119 used Bench 93069371a78fbe9612fa02640b2543b6f56ddd42 and old dataset f30803905a7a1235688d151384490ce1f998214d614dea8ad117ed49bbcb39a4. Its 145 result/runtime pairs and summary are preserved: 130 completed / 15 partial engine reviews, 2890 completed / 10 failed process records. Green job/collector status is not full-review completion. Logs did not reveal recoverable failure reasons.

Old-data view	TP	FP	FN	Neutral	Micro P / R / F1
Medium-aligned	117	325	101	53	26.47% / 53.67% / 35.45%
Unscoped	127	325	108	43	28.10% / 54.04% / 36.97%

Those 495 generated comments are not rescored, overwritten, or used as the new matched baseline. The earlier candidate was never dispatched. Historical run 39616090167 likewise remains 105 TP / 322 unmatched-by-gold FP / 97 FN over 137 completed entries; eight excluded positive entries held 31 additional gold. Historical fixed-assignment sensitivities are not current scores.

Validation

• Full suite: 1351 passed, 1 skipped, 1 deselected; no paid calibration/reviewer suite was run.
• Final timeout-scope metadata correction: 44 focused tests passed.
• Strict repository type check, Ruff and all configured applicable pre-commit hooks passed; clean working tree after commit.
• Real-git quarantine tests cover normal/error restoration, unchanged reviewer-environment diff, dirty-configuration rejection, source-contamination rejection, and the actual CLI argument boundary.
• Dataset SHA256 remains exactly 42178a85e9e8ce023f32ef030a95f2c2829f76268d47cac47c8beafd577a8a30.
• Exact-head CI is pending discovery/completion; the earlier green cd148da4 CI is not validation of this implementation.

See current code-review contract and preserved history. No new evaluation results or improvement claims yet.

[github-code-quality]

Code coverage overview

Languages: Python

Python / code-coverage/pytest

main 7c98b75	gggdttt-benchmark-au... 546af05	+/-
88%	88%	0%

View coverage of the 10 most impacted files.

File	main 7c98b75	gggdttt-benchmark-au... 546af05	+/-
src/bcbench/age...opilot/agent.py	73%	68%	-5%
src/bcbench/res...s/codereview.py	100%	97%	-3%
src/bcbench/res.../leaderboard.py	99%	98%	-1%
src/bcbench/com...ands/dataset.py	56%	55%	-1%
src/bcbench/com...nds/evaluate.py	78%	77%	-1%
src/bcbench/types.py	92%	92%	0%
src/bcbench/eva...review_judge.py	97%	98%	+1%
src/bcbench/eva...e/codereview.py	90%	97%	+7%
src/bcbench/eva...view_parsing.py	70%	80%	+10%
src/bcbench/age...lated_review.py	0%	91%	+91%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.

---

_{Updated October 22, 2026 21:28 UTC}