Skip to content

Add bounded review trace and immutable knowledge comparison pins - #83

Draft
Wenjie Fan (gggdttt) wants to merge 9 commits into
mainfrom
gggdttt-audit-candidate-engine-uptake
Draft

Wenjie Fan (gggdttt) wants to merge 9 commits into
mainfrom
gggdttt-audit-candidate-engine-uptake

Conversation

@gggdttt

@gggdttt Wenjie Fan (gggdttt) commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Current scope and immutable revisions

This supersedes the earlier no-knowledge comparison proposal. Preserve this PR as draft and unmerged. Both revisions below use the same knowledge-enabled orchestration and shared trace instrumentation; only the BCQuality ref differs.

Role Engine commit BCQuality commit Content version
Shared baseline; required for the approved two-case diagnosis 73eb5c51fb7cdde453e3116778c4452c621ac9aa 41b0b7a39f890d904fd00f3a5c61582573c9adf8 1.9
Single-event-rule candidate; not the two-case diagnosis engine 0c9681369a8f9287d1d927282d8823c87fd0d001 d5aa1b8244ed04b281c45cce7c1ff598b2cffa2d 1.9

The candidate engine is a direct child of the baseline. Their entire repository diff is one ref: replacement in agents/ALReviewAgent/bcquality.config.yaml; all other Git blobs, including root prompt, trace schema, orchestration, tests and version metadata, are identical. Both retain Microsoft-only knowledge scope. Upstream microsoft/BCQuality#250 was verified open/unmerged at d5aa; its direct parent is exactly 41b.

Shared instrumentation base: 73eb5c51fb7cdde453e3116778c4452c621ac9aa. Its pre-instrumentation parent is frozen experiment 25817a39a67b9e883f1a8ea9624283ab2b41af9b; original main base is 9607dea1c84fcf660983a5e3791dece1fb2f1623. Earlier 5062/3c719/38180/ceaae/4b5c pins are historical only. No history was amended or force-pushed.

Bounded engine-owned trace

  • Independent _review-trace.json v1; manifest declares trace_schema_version: 1. No BCQuality findings-report schema extension or skill/routing changes.
  • Original leaf identity uses ordinal, original zero-based finding index and report hash. Bounded original candidate prose, supplied article refs, safe source location, severity, admission and deterministic normalizations support semantic diagnosis.
  • Optional _root-dispositions.json permits only explicit model-declared kept/merged/downgraded/removed relationships and fixed reason codes. Missing, invalid or ambiguous relationships remain unknown. No guessed lineage, free-text reasons or exception text.
  • final_index addresses _review-report.json#/findings, zero-based, before engine filtering. It is NOT a normalized finding index or a Bench F identifier. Trace does not establish downstream post-filter lineage; compare actual raw root and consumer artifacts separately.
  • Limits: 1000 candidates/finals/dispositions, 128 diagnostics, 8 refs/indices, 1024-character candidate prose, 4MiB artifact; explicit truncation and body status. Defensive credential/control-character redaction is not a guarantee of complete secret removal. Retained prose requires the same access controls as final findings.
  • The optional root prompt/output changes model input and may affect behavior. Both future comparison arms share them; this is NOT claimed literally behavior-invariant. Existing finding eligibility, severity thresholds, topology and defaults are unchanged.

Legacy engines without the manifest marker are handled as not_available_legacy by the downstream collector. Early failures before trace/manifest creation remain missing evidence. Historical frozen runs cannot be retroactively assigned leaf/root causes.

Validation and cross-repository evidence

  • Actual engine functions generated four deterministic synthetic scenarios: explicit dispositions (including unknown), absent relations, 1002-to-1000 truncation, and fixed failure codes. No model calls.
  • microsoft/BC-Bench#983 collector receipt confirms all four producer artifacts passed unchanged with schema validation. Collector revision: 546af054f12c0e0c82a8f71105b8a1c96ba3e992.
  • Receipt helper hash caccf03569c8f6ea382bd115ee9fcde1461bfcc40f7e02539965a5228f8d1606 predates only a lint rename from $matches to $declarations. Reversing that rename reproduces the old hash exactly. Current helper hash: 54ff4851ca7e43effc5146339c9e0c89d337f3e29933fd31e95a63b95da84c94.
  • Current helper regenerated all four scenarios offline into a separate directory; every output SHA exactly matches the collector's passed receipt, and all validate against the unchanged schema hash 3c6b6dd6abf0f0c32ec1201a1ecfb2996546c5151eccc6bbbbe91bbf0147c5de. Original evidence retained.
  • Related orchestrator/metrics/trace suite: 345 passed, 3 failed. The three pre-existing schema-error-message assertions were independently reproduced on the pre-change baseline. Trace-only: 18 passed; separate entry-point/filter suites: 31 passed. Parse checks and diff checks passed; helper lint has no findings.
  • Baseline exact-head CI: https://github.com/microsoft/BC-ALAgents/actions/runs/39920462701 (in progress at publication; not yet claimed green).

Existing shared CLI reproducibility fix

Optional caller-supplied COPILOT_AUTO_UPDATE is forwarded unchanged; no default is injected. Actual cleared-environment native probes with cached newer CLI retained showed unset => 1.0.102-1, explicit false => 1.0.88 on three probes. Actual startup accepted requested 1.0.88 and rejected deliberately mismatched 1.0.83. No model calls or global installation. This fix is common to both engine identities.

Execution hold

Only privacy013 and style012, once each, have diagnostic approval; they must use the 41b baseline above, including inspection of G03 versus G05 and G14-G16. Parent readiness review and explicit launch confirmation remain required. The full 145-entry knowledge comparison remains on hold. This PR preparation performs no model/evaluation dispatch, release, merge or Agent merge enablement.

wenjiefan and others added 2 commits October 22, 2026 10:59
Use a4593a1b1585a0fa663efbf39ff02c60f846d851 as the immutable knowledge baseline while retaining orchestration from 9607dea. This is preparation only; no release or evaluation dispatch.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e71ba27-fc10-4149-8e6a-8d71c295d9a2
Only the knowledge SHA differs from baseline engine 5062ffd. Both retain BCQuality 1.9 metadata and identical Microsoft-only scope and orchestration. No scoring, runtime, release, or evaluation changes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e71ba27-fc10-4149-8e6a-8d71c295d9a2
@gggdttt
Wenjie Fan (gggdttt) requested a review from a team October 22, 2026 08:59
wenjiefan and others added 3 commits October 22, 2026 11:27
Forward COPILOT_AUTO_UPDATE unchanged when supplied without introducing a default. Version-only native probes with existing newer cache confirm false retains 1.0.88; unset still selects 1.0.102-1. Keep startup mismatch rejection unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e71ba27-fc10-4149-8e6a-8d71c295d9a2
Baseline knowledge: a4593a1b1585a0fa663efbf39ff02c60f846d851. Shared engine logic: 10086e6. Supersedes baseline 5062ffd for the upcoming comparison.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e71ba27-fc10-4149-8e6a-8d71c295d9a2
Only the knowledge SHA differs from revised baseline 38180ca. Supersedes candidate 3c71999 for the upcoming comparison.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e71ba27-fc10-4149-8e6a-8d71c295d9a2
@gggdttt Wenjie Fan (gggdttt) changed the title Pin controlled BCQuality baseline and unmerged knowledge repair candidate Prepare controlled knowledge pins with explicit CLI auto-update forwarding Oct 22, 2026
@gggdttt
Wenjie Fan (gggdttt) marked this pull request as draft October 22, 2026 09:30
wenjiefan and others added 2 commits October 22, 2026 12:36
Pin 45cbe369527bbe6d4cad0b2a877bd1d61a617cdb with nine repaired articles. Retain baseline 38180ca and shared CLI environment fix; the pair differs only in the knowledge ref. Behavior has not been evaluated.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e71ba27-fc10-4149-8e6a-8d71c295d9a2
Uptake 41b0b7a39f890d904fd00f3a5c61582573c9adf8 without changing common engine logic. The requested comparison is now no BCQuality versus narrowed BCQuality; the old-knowledge engine revision is not a no-knowledge baseline.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e71ba27-fc10-4149-8e6a-8d71c295d9a2
@gggdttt Wenjie Fan (gggdttt) changed the title Prepare controlled knowledge pins with explicit CLI auto-update forwarding Uptake narrowed BCQuality repair for knowledge-enabled comparison arm Oct 22, 2026
wenjiefan and others added 2 commits October 23, 2026 09:49
Preserve knowledge41b as the shared instrumented baseline. Record bounded original candidate prose, explicit optional root dispositions and categorical failure stages without changing the findings-report contract.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e71ba27-fc10-4149-8e6a-8d71c295d9a2
Change only the BCQuality ref from 41b0b7a to its direct child d5aa1b8. Preserve baseline engine 73eb5c5 for the authorized two-case diagnostic; candidate is not its replacement.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6e71ba27-fc10-4149-8e6a-8d71c295d9a2
@gggdttt Wenjie Fan (gggdttt) changed the title Uptake narrowed BCQuality repair for knowledge-enabled comparison arm Add bounded review trace and immutable knowledge comparison pins Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant