[khanukov]

What readers get

A substantive English study of 21 exact model IDs across 44 related synthetic cases, with two original observation series, full model/per-flow tables, scored Core decision counts, two accessible static charts, and public aggregate JSON/CSV. Three sanitized semantic case summaries explain why an achieved goal can coexist with an execution-quality failure. README and the preserved historical 19-model table link to the new material.

Evidence and limits

• Original observations stay original: historical 916 scored + 4 excluded + 4 aborted; repeat 917 scored + 7 excluded. The common-scored 910 pairs contain 66 strict flips; 61 retain goal=true in both series. No pooling, confidence intervals, significance, model-learning or same-settings replication claim.
• Separate Compliance-only diagnostic: baseline parity on 1,044 records; 23 finding/retrieval changes; 11 stored strict passes become diagnostic failures. Goals and other dimensions unchanged. No corrected full 44-case ranking, authenticated replay, official historical regrade or new model trial.
• Correction fix: account for Compliance terminal read-guard refusals #37 is now merged into main; this PR remains a documentation/publication-only delta against the updated main. Original study/data bytes are unchanged.
• Costs are unpublished/null, not zero. Cases are related synthetic variants, not 44 independent real-world tasks. Official Lifecycle scope remains one Maintenance Semantic Scenario with three variants; Commerce/Compliance remain SDK incubators.
• Case summaries are from AI-assisted maintainer review of owner-retained traces, not independent human validation. Raw trajectories are not published; aggregates support arithmetic and plotting, not full independent historical-study reproduction.

The existing publication exception is extended only to this exact study and its six aggregate/figure companions. The old table, grading, pending human/claims gates, official-release status and licensing/ownership remain unchanged. No raw provider bodies, credentials, private source IDs, operational data or billing are included.

Real checks

Parent independently checked the frozen aggregate mapping, all 21 model / 63 flow / 42 diagnostic rows, sums, null costs, exact rational order and rendered charts. On the latest candidate, all 12 provenance/governance tests, Ruff check/format (577 files) and all 14 public-release checks pass. Earlier 80-test and 55-test qualification remains attributed to its original candidate; study/data/runtime content is unchanged.

CI follow-up: one pending-review sentence now uses the canonical “human semantic validation” wording. No scanner or test was weakened or changed. The article and all aggregate/figure bytes are identical, so the existing immutable study link remains valid. Full hosted CI and requested Qodo are fresh-head gates, not implied by local results.

Owner authorized landing this study and its bounded site entry after final checks. After #37 merged, this branch was updated with a signed-off upstream merge (no force/history rewrite). Parent reran 109 focused integration tests, all 14 public-release gates, Ruff check/format (578 files), and complete packet/CSV/table/chart verification. Fresh final-head hosted CI and Qodo are required before this PR merges; old green checks are not reused as current-head CI. No benchmark/provider calls, historical regrade, raw trajectory publication, package/tag/GitHub release or unrelated PR merge is authorized.

[khanukov]

/agentic_review

[qodo-code-review]

PR Summary by Qodo

Document 21-model observations and evaluator sensitivity

📝 Documentation ⚙️ Configuration changes 🧪 Tests 🕐 40+ Minutes

AI Description

• Publishes two-series observations, model and flow aggregates, and accessible charts distinguishing
 goals from strict passes.
• Separately documents a Compliance evaluator diagnostic without changing original grades or
 evaluator code.
• Extends the bounded publication exception while retaining validation and official-release gates.

Diagram

graph TD
  Historical["Historical series"] --> Aggregates[("Aggregate data")] --> Study["October study"] --> Readers["Public readers"]
  Repeat["Repeat series"] --> Aggregates
  Aggregates --> Diagnostic["Compliance diagnostic"] --> Study
  Aggregates --> Charts["Static charts"] --> Study
  Governance["Publication gates"] --> Study

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Generate public artifacts from aggregate JSON

• ➕ Makes CSV and chart regeneration independently repeatable from published data.
• ➕ Reduces the risk of drift among static representations.
• ➖ Adds tooling and maintenance to a bounded, one-off publication.
• ➖ Cannot reproduce the underlying observations without private records.

Recommendation: The checked-in artifacts fit this fixed, narrowly authorized study and keep the publication scope explicit. If studies become recurring, a generator consuming the public aggregates would be worth adding; it would not replace raw-record validation.

Files changed (12) +3412 / -5

Tests (1) +29 / -1

test_repository_governance.pyAssert the exact study publication boundary +29/-1

Assert the exact study publication boundary

• Tests that the manifest names exactly the seven study artifacts, that they exist, and that human validation and official release remain unauthorized. Adjusts an existing checklist assertion to its revised wording.

tests/test_repository_governance.py

Documentation (10) +3380 / -1

PUBLIC_RELEASE_CHECKLIST.mdRecord the study’s bounded release exception +16/-1

Record the study’s bounded release exception

• Lists the six authorized companion artifacts and reiterates that existing claims, validation, and release gates remain open.

PUBLIC_RELEASE_CHECKLIST.md

README.mdLink readers to the October study +2/-0

Link readers to the October study

• Adds an entry point to the two-series study while distinguishing it from the preserved historical table.

README.md

MODEL_RESULTS.mdCross-link the preserved historical results table +2/-0

Cross-link the preserved historical results table

• Points readers to the new study without changing the existing 19-model table or its scores.

docs/MODEL_RESULTS.md

2026-10-aggregates.jsonPublish structured study and diagnostic aggregates +3041/-0

Publish structured study and diagnostic aggregates

• Supplies model, model-by-flow, paired-transition, and Compliance diagnostic counts with provenance and limitations. Costs remain null rather than being represented as zero.

docs/studies/2026-10-aggregates.json

2026-10-case-churn.svgChart strict-pass gains and losses by flow +16/-0

Chart strict-pass gains and losses by flow

• Adds a titled, described static SVG showing gains and losses among common-scored cases.

docs/studies/2026-10-case-churn.svg

2026-10-compliance-diagnostic.csvExpose Compliance diagnostic counts by model and series +43/-0

Expose Compliance diagnostic counts by model and series

• Publishes 42 rows comparing stored and candidate evaluator findings, retrieval checks, and strict passes without presenting them as an official regrade.

docs/studies/2026-10-compliance-diagnostic.csv

2026-10-goal-strict.svgChart latest goal and original strict fractions +70/-0

Chart latest goal and original strict fractions

• Adds a titled, described two-bar SVG for all 21 models, using the report table for exact fractions.

docs/studies/2026-10-goal-strict.svg

2026-10-model-flows.csvPublish model-by-flow observation counts +64/-0

Publish model-by-flow observation counts

• Provides 63 rows of scored outcomes, exclusions, and paired goal and strict transitions across Maintenance, Commerce, and Compliance.

docs/studies/2026-10-model-flows.csv

2026-10-models.csvPublish 21-model observation counts +22/-0

Publish 21-model observation counts

• Provides per-model historical, repeat, and common-scored counts, including scored Core decision totals and empty unpublished cost fields.

docs/studies/2026-10-models.csv

2026-10-observations.mdExplain outcomes, case churn, and evaluator sensitivity +104/-0

Explain outcomes, case churn, and evaluator sensitivity

• Introduces the two-series study with full model and flow tables, three sanitized semantic examples, and a separate Compliance diagnostic. States the limits of related synthetic cases, private traces, differing run settings, and pending independent validation.

docs/studies/2026-10-observations.md

Other (1) +3 / -3

PUBLICATION_MANIFEST.jsonBound publication to the October study artifacts +3/-3

Bound publication to the October study artifacts

• Extends the descriptive-results exception to the study and six named companions. Retains pending human validation, official-release restrictions, and prohibitions on raw traces and billing disclosure.

PUBLICATION_MANIFEST.json

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Tip of the day

💡 Did you know, you can enable the Remediation agent and Qodo fixes findings in a dedicated fix PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Previous reviews

Review updated until commit f15c259 ⚖️ Balanced

Results up to commit 3672fde ⚖️ Balanced

No changes from previous review

Results up to commit ce80b2f ⏭️ Skipped

No changes from previous review

[khanukov]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit ce80b2f

[khanukov]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit f15c259