[khanukov]

Final selected-six recovery publication

The finite technical-recovery campaign is complete: every one of the 102 initially technical excluded/aborted original cells received a linked actual replacement. There are 106 retained correction attempts because four interrupted Haiku correction attempts remain in the ledger and received distinct replacements. Original denominator stays 264, one observation per model; total retained original/correction dispositions are 370. The original 162 scored results and all 21 historical model means remain unchanged.

Effective final coverage is 207 scored, 57 technical exclusions, 0 effective aborts. Remaining technical exclusions are Mistral: 20, Grok: 29 and GLM: 8, not quality failures or hidden zeros. Haiku and Kimi now have 44 scored each; this does not improve or reinterpret any recorded grade. Legacy consumed-but-unstarted entries are nonattempts, not additional observations.

Approved aggregate SHA256: a5032145578776f88e32d6235849dfb2243afefc4992827c4344bcc34817e432.
Common27-model averages SHA256: af31cc190eac3adee1aea88ea74f360d255604624f2c98728318943b5212cc0a.
Actual model observations completed on 2026-10-29; publication integration is 2026-10-30.

Original execution revision is f3bd45fb0d219929c9415b734dfeb9ef0cb6a5b8. Corrections used that revision and 989c22d9c8342df26eb78eb77f798c9b245abf6e. The schema2 correction revision is an anchor, not a claim of uniform execution. Public provenance groups describe admitted attempt contracts, not a uniform observed native-wire contract. Rates are conditional on scored coverage, use exact Fraction arithmetic and are descriptive across heterogeneous runtime/evaluator/provider settings, not a controlled ranking or independent human validation. New costs remain Unknown/null; historical River cost assumptions apply only to the original 21. No raw traces, token usage, credentials, private paths, account data or invoices are published.

Source scope and verified evidence

• Integrates the narrow single-native-decision Haiku control, historical correction admission validation, explicit River pacing and safe HTTP error telemetry while preserving fixed cases, domain/evaluator semantics, independent snapshots, original financial policy and native request bounds. No scored quality reruns, best-score selection, artificial money/call/generation caps or parser loosening.
• All 207 effective scored records were independently replayed with network/credential access blocked, with record/trial/source binding, actual wire/usage controls and all original-to-replacement mappings checked. Six full final campaign journals were reconciled exactly once; liabilities and failed attempts remain retained privately.
• Parent independently accepted the final packet bytes and all 27 model values, old21 no-loss, source/site identity and exact ordering. Parent final local checks:55 focused governance/source-pin tests, all14 public-release checks, Ruff across 587 Python files; all 32 site tests and deterministic27-model build/check. The frozen source runtime qualification and previous original-pass evidence remain retained separately, not relabelled as current publication-head CI.
• Preserves all seven historical study artifacts and official-release/human-review boundaries. Only approved aggregate artifacts and the common report are published; source revisions are explicit byte-pinned public ancestry.

Current full hosted checks and substantive current-head Qodo review are required before normal authorized landing. Owner consent covers this bounded aggregate publication. The matching site PR is DwellyOrg/operatebench-results-preview#10 and stays held until this source PR has landed. Actual deployed revision, bytes and all 27 rows will be verified separately; this PR body is not a deployment completion claim.

[khanukov]

/agentic_review

[qodo-code-review]

PR Summary by Qodo

Add selected-six native execution and bounded 27-model results

✨ Enhancement 🐞 Bug fix 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Add six exact native provider/model routes with bounded profiles and strict response validation.
• Publish a 27-model descriptive list, preserving historical rates and disclosing scored coverage
 and provenance.
• Pin two approved aggregate artifacts while retaining historical study files and release
 restrictions.

Diagram

graph TD
  R["Selected roster"] --> T["HTTP transport"] --> N["Native providers"] --> E["Private records"] --> P["Selected-six packet"] --> M["Model means"] --> D["Results page"]
  H["Historical rates"] --> M

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Generalize provider onboarding

• ➕ Could reduce duplication when adding later provider/model routes.
• ➖ Broadens the execution and admission surface beyond the six approved routes.
• ➖ Raises regression and governance risk for historical campaigns.

Recommendation: Keep the finite selected-six profile and reuse the existing campaign machinery. A general provider framework may help future work, but it is a poor fit for this explicitly bounded execution and publication approval.

Files changed (29) +2714 / -111

Enhancement (11) +1332 / -47

2026-10-model-averages.jsonAdd approved 27-model machine means +548/-0

Add approved 27-model machine means

• Publishes ordered exact rational rates, displayed percentages, observation counts, retained cost estimates or nulls, and selected-six provenance.

docs/studies/2026-10-model-averages.json

2026-10-selected-six.jsonAdd approved selected-six aggregate packet +100/-0

Add approved selected-six aggregate packet

• Records each model's assigned, scored, excluded, and aborted counts alongside goals, strict results, routes, settings, and execution revision. Leaves all six costs null.

docs/studies/2026-10-selected-six.json

mistral_chat.pyValidate Mistral reasoning-token metadata +38/-16

Validate Mistral reasoning-token metadata

• Admits a closed completion-token details object and ensures reasoning and cached token counts do not exceed their parent totals.

src/operatebench/providers/mistral_chat.py

openai_responses.pyAdmit closed OpenAI access metadata +13/-6

Admit closed OpenAI access metadata

• Advances the response extension contract to v4, accepting only null or the observed cyber/standard access shape.

src/operatebench/providers/openai_responses.py

three_flow_admission.pyAdmit closed selected-six execution profiles +16/-2

Admit closed selected-six execution profiles

• Adds the roster and credential variables, null-cap proposal handling, and selected-route published profile validation.

tools/three_flow_admission.py

three_flow_campaign.pyReport the selected-six uncapped preflight +2/-1

Report the selected-six uncapped preflight

• Sets the selected roster's preflight monetary cap to null while retaining non-authorizing campaign behavior.

tools/three_flow_campaign.py

three_flow_http.pyDispatch selected models through native HTTP routes +89/-20

Dispatch selected models through native HTTP routes

• Adds xAI and River HTTP SDK paths, selected request controls, safe connection timeouts, response checks, and route-specific settings provenance.

tools/three_flow_http.py

three_flow_live.pyExpose the selected-six roster choice +1/-0

Expose the selected-six roster choice

• Accepts the selected-six HTTP profile in the campaign command-line roster choices.

tools/three_flow_live.py

three_flow_live_factory.pyRecord selected tariff evidence in run settings +10/-1

Record selected tariff evidence in run settings

• Binds published tariff evidence and profile identity to transport settings without treating estimated usage as an invoice.

tools/three_flow_live_factory.py

three_flow_profiles.pyRegister six exact provider/model selections +40/-1

Register six exact provider/model selections

• Defines the selected roster, xAI and River HTTP endpoints, API selections, and selected reasoning-mode routing.

tools/three_flow_profiles.py

three_flow_selected.pyDefine finite selected-six wire and profile contracts +475/-0

Define finite selected-six wire and profile contracts

• Adds exact route controls, diagnostic bounds, tariff categories, and profile validation. Validates selected OpenAI, Anthropic, and River response readings against raw wire and SDK data.

tools/three_flow_selected.py

Tests (12) +1214 / -25

extension_contract_goldens.jsonPin revised provider extension contracts +11/-4

Pin revised provider extension contracts

• Updates the literal OpenAI v4 and Mistral usage-extension expectations and their digests.

tests/fixtures/extension_contract_goldens.json

test_extension_primitives.pyCheck extension oracles independently +72/-9

Check extension oracles independently

• Parametrizes contract and digest checks so one mismatch cannot mask another. Adds a complete Lifecycle Mistral settings snapshot.

tests/test_extension_primitives.py

test_http_null_metadata.pyKeep null-metadata fixtures on the legacy branch +1/-0

Keep null-metadata fixtures on the legacy branch

• Explicitly disables selected-six behavior in manually constructed transport fakes.

tests/test_http_null_metadata.py

test_mistral_usage_metadata.pyExercise closed Mistral usage metadata +97/-0

Exercise closed Mistral usage metadata

• Tests admissible reasoning/cache details and refusal of invalid counts, unknown fields, misplaced data, and raw/typed disagreement.

tests/test_mistral_usage_metadata.py

test_openai_reasoning_items.pyPreserve the legacy reasoning-items fixture +1/-0

Preserve the legacy reasoning-items fixture

• Marks its manually constructed HTTP transport as outside the selected-six branch.

tests/test_openai_reasoning_items.py

test_openai_response_extensions.pyUpdate OpenAI extension assertions to v4 +2/-2

Update OpenAI extension assertions to v4

• Pins the new contract identifier and closed access-program scalar expectation.

tests/test_openai_response_extensions.py

test_provider_kernel_composition.pyRepin reviewed response-interpretation settings +5/-4

Repin reviewed response-interpretation settings

• Updates the settings digest and OpenAI extension expectations while continuing to check the unchanged request baseline.

tests/test_provider_kernel_composition.py

test_public_release.pyTest fail-closed aggregate provenance pins +178/-0

Test fail-closed aggregate provenance pins

• Checks exact artifact paths and bytes, JSON ambiguity and mutation rejection, and preservation of other source and archive scanners.

tests/test_public_release.py

test_repository_governance.pyVerify aggregate publication invariants +169/-6

Verify aggregate publication invariants

• Reconciles selected-six counts, historical two-rate means, exact ordering, displayed percentages, costs, and coverage. Pins the seven historical study artifacts' bytes.

tests/test_repository_governance.py

test_selected_response_metadata.pyProbe selected-route metadata agreement +102/-0

Probe selected-route metadata agreement

• Tests narrow OpenAI access and usage metadata, Anthropic usage readings, and refusal of unsupported values or raw/typed disagreement.

tests/test_selected_response_metadata.py

test_selected_six_bounds.pyPin selected-six route and output bounds +105/-0

Pin selected-six route and output bounds

• Checks all six published bounds and rejects alternate Mistral limits; verifies Mistral context and output enforcement.

tests/test_selected_six_bounds.py

test_selected_six_http.pyExercise six native routes with offline SDK wires +471/-0

Exercise six native routes with offline SDK wires

• Covers the 264-assignment census, multiturn dispatch and replay, request controls, fault accounting, argument preservation, and River wire refusal.

tests/test_selected_six_http.py

Documentation (4) +95 / -33

PUBLIC_RELEASE_CHECKLIST.mdDocument the selected-six release boundary +15/-3

Document the selected-six release boundary

• Adds the two approved artifacts and specifies scored coverage, Unknown new costs, unchanged historical files, and the limits of owner approval.

PUBLIC_RELEASE_CHECKLIST.md

README.mdDescribe the expanded results list +12/-7

Describe the expanded results list

• Points readers to the 27-model aggregate, explains the differing observation counts and coverage, and retains the no-leaderboard warning. Links the response-contract test-oracle documentation.

README.md

MODEL_RESULTS.mdPublish the common 27-model descriptive table +37/-23

Publish the common 27-model descriptive table

• Replaces the historical display with ordered aggregate rates, costs, and scored coverage while linking the original public revision. Explains heterogeneous provenance, excluded assignments, and the limits of the comparison.

docs/MODEL_RESULTS.md

RESPONSE_CONTRACT_TEST_ORACLES.mdExplain frozen response-contract oracles +31/-0

Explain frozen response-contract oracles

• Documents the OpenAI and Mistral contract expectations, settings snapshots, and legacy-versus-selected fixture boundaries.

docs/RESPONSE_CONTRACT_TEST_ORACLES.md

Other (2) +73 / -6

PUBLICATION_MANIFEST.jsonBound the aggregate publication exception +3/-3

Bound the aggregate publication exception

• Authorizes exactly two additional machine artifacts and a common 27-model descriptive list. Preserves the historical study scope and pending human-validation and official-release gates.

PUBLICATION_MANIFEST.json

check_public_release.pyScope the public source-revision exception by digest +70/-3

Scope the public source-revision exception by digest

• Exempts a source revision only within the two exact, SHA-256-approved JSON artifacts after strict parsing and location checks. Other bare commit identifiers remain prohibited.

tools/check_public_release.py

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Recovery guide still says final data is missing ✓ Resolved

Description

RECOVERY_INTAKE.md calls the checked-in packet an old partial v1 result (162 scored, 98 excluded,
4 aborted), says no final packet is supplied, and claims the actual-data goldens remain pinned to
stale v1 data. Because this PR already checks in the final schema-2 packet (207 scored, 57 excluded,
0 aborted), pins its SHA256 and publishes it in MODEL_RESULTS.md, readers following the guide will
treat the published aggregate as pending and repeat completed intake steps.

Code

docs/studies/RECOVERY_INTAKE.md[R1-3]

+# Pending owner technical recovery intake
+
+Current checked-in v1 packet is the OLD PARTIAL (162 scored, 98 excluded, 4 aborted), not final technical recovery. This prep does not approve publication. Keep packet bytes and approval pins unchanged until parent approval.

Relevance

●●● Strong

Guide contradicts this PR’s published final packet and would mislead readers into repeating
completed intake steps.

PR-#31

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The guide’s lines 3, 22 and 24 describe a partial v1 packet, a pending replacement and stale
goldens. In contrast, the checked-in docs/studies/2026-10-selected-six.json is schema version 2
with full provenance, and its per-model counts total 207 scored, 57 excluded (Mistral 20, Grok 29
and GLM 8) and 0 aborted. tools/check_public_release.py lines 962–969 pin the final approved
digests a5032145… and af31cc19…, while MODEL_RESULTS.md identifies the approved aggregate and
reports 207 scored results; replacing the packet, updating the pins and publishing the result have
therefore already happened.

docs/studies/RECOVERY_INTAKE.md[1-24]
docs/studies/2026-10-selected-six.json[36-96]
tools/check_public_release.py[961-969]
docs/studies/RECOVERY_INTAKE.md[1-3]
docs/studies/RECOVERY_INTAKE.md[15-24]
docs/MODEL_RESULTS.md[5-7]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`docs/studies/RECOVERY_INTAKE.md` presents the published final aggregate as a pending partial v1 packet, says no final packet is supplied, and claims actual-data goldens await final intake, even though the approved v2 packet has been integrated and published.

## Fix Focus Areas
- docs/studies/RECOVERY_INTAKE.md[1-24]

## Recommended Fix
Update the header and status claims to state that final intake and publication are complete: the approved v2 packet has 207 scored, 57 excluded and 0 aborted results, and its SHA256 (`a5032145…`) has been pinned. Remove claims that the final packet is missing or that actual-data checks still await intake; retain the former intake steps only if clearly marked as historical, or delete or archive the guide if it was intended solely for preparation.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Mistral guidance accepts a rejected response ✓ Resolved

Description

The revised Mistral section says its response contract accepts
completion_tokens_details.reasoning_tokens and validates both token subsets, but the
mistral-small-2603 lane it describes still permits only prompt_tokens_details.cached_tokens and
has no subset check. A reader using this guide for that lane would expect a response with
reasoning-token details to be accepted, while its wire validator rejects the undeclared usage field.

Code

docs/PROVIDER_MATRIX.md[465]

+| `usage` | `completion_tokens_details` | closed object with required `reasoning_tokens` |

Relevance

●●● Strong

Accepted documentation correction: the guide must match the lane's actual wire schema and validation
behavior.

PR-#31

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The guide names mistral-small-2603 as the lane under discussion. Its boundarybench schema allows
only cached-token details, and the usage wire shape admits only that schema's extension names; the
operatebench adapter separately defines the expanded schema and subset check.

docs/PROVIDER_MATRIX.md[451-467]
docs/PROVIDER_MATRIX.md[479-484]
src/boundarybench/providers/mistral_chat.py[379-387]
src/boundarybench/providers/mistral_chat.py[603-611]
src/boundarybench/providers/mistral_chat.py[1159-1161]
src/operatebench/providers/mistral_chat.py[181-240]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Mistral section describes the historical `mistral-small-2603` lane but now assigns it the expanded response contract implemented by the selected-six adapter. The historical lane rejects reasoning-token details and does not check token subsets.
## Fix Focus Areas
- docs/PROVIDER_MATRIX.md[451-511]
- src/boundarybench/providers/mistral_chat.py[379-414]
- src/operatebench/providers/mistral_chat.py[181-240]
## Recommended Fix
Restore the historical lane's exact accepted shape and validation description in its section. Document the selected-six adapter's additional reasoning-token extension and subset checks separately, identifying the lane to which they apply.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Mistral guidance omits valid token data ✓ Resolved

Description

RESPONSE_EXTENSION_SCHEMA now accepts completion_tokens_details.reasoning_tokens, but the
provider guide still describes a closed contract containing only
prompt_tokens_details.cached_tokens. When a Mistral response contains reasoning-token details,
readers following that guide will treat an accepted response as outside the documented contract; the
guide also says extension values are not read, although the new subset check reads them.

Code

src/operatebench/providers/mistral_chat.py[R187-189]

+        "completion_tokens_details": MappingProxyType(
+            {"reasoning_tokens": EXTENSION_COUNT}
+        ),

Relevance

●●● Strong

Documentation contradicts the newly accepted and validated reasoning-token extension behavior.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The changed schema adds the reasoning-token object, and the changed subset check reads extension
values. The unchanged provider guide says the closed contract has only the cached-token object and
that the build reads nothing from it.

src/operatebench/providers/mistral_chat.py[185-203]
src/operatebench/providers/mistral_chat.py[233-239]
docs/PROVIDER_MATRIX.md[460-489]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Mistral provider guide documents only the old cached-token extension, while the validator now accepts reasoning-token details and checks both subsets against their totals.

## Fix Focus Areas
- docs/PROVIDER_MATRIX.md[460-489]
- src/operatebench/providers/mistral_chat.py[185-203]
- src/operatebench/providers/mistral_chat.py[233-239]

## Recommended Fix
Update the guide's closed field list and validation description to match both accepted extension objects and their subset checks. Keep the distinction between validating token counts and using them for pricing.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View medium (1)

4. Mistral Large run is recorded with the wrong reasoning mode ✓ Resolved

Description

For selected-six models the settings fill in reasoning_mode with http_reasoning_mode(model).
That helper calls minimum_effort, which only returns "none" for the three GPT models, so
mistral-large-4 is recorded as "MINIMUM". The request it actually sends is `reasoning_effort:
"none"`. The new override block only fixes the mode for River models, so the stored settings for the
Mistral observation contradict the real request and the published "reasoning": "none" in
docs/studies/2026-10-selected-six.json.

Code

tools/three_flow_http.py[R542-548]

+            if provider == "river_http":
+                self.settings.pop("reasoning_items_contract", None)
+                self.settings.pop("store", None)
+                self.settings["owner_output_cap_tokens"] = 30000
+                self.settings["reasoning_mode"] = (
+                    "MINIMUM" if model == "GLM-5.3" else "OFF"
+                )

Relevance

●●● Strong

Stored reasoning mode contradicts the actual Mistral request and published observation metadata.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

mode_fields sends {"reasoning_effort": "none"} for Mistral (three_flow_selected.py line 25). But
the settings block at three_flow_http.py lines 486-490 sets
reasoning_mode=http_reasoning_mode(model). http_reasoning_mode returns "OFF" only when
minimum_effort(model) == "none", and minimum_effort is "none" only for gpt-6-luna, gpt-6-sol and
gpt-6.1-terra (three_flow_profiles.py line 33). The new selected-six override (lines 542-548)
corrects reasoning_mode only when the provider is river_http. No selected-six test checks
reasoning_mode.

tools/three_flow_http.py[486-493]
tools/three_flow_http.py[535-548]
tools/three_flow_selected.py[16-29]
tools/three_flow_profiles.py[32-33]
tools/three_flow_profiles.py[172-173]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
For selected-six runs, `reasoning_mode` in the transport settings comes from `http_reasoning_mode(model)`. That labels `mistral-large-4` as MINIMUM even though the request sends `reasoning_effort: "none"`, so the recorded provenance does not match the request.

## Fix Focus Areas
- tools/three_flow_http.py[535-548]
- tools/three_flow_profiles.py[32-33]

## Recommended Fix
In the `if selected:` block, set `self.settings["reasoning_mode"]` for every selected-six model from its actual `mode_fields`: "OFF" when the effort is "none", otherwise "MINIMUM". Alternatively, add `mistral-large-4` to the models that `minimum_effort` treats as "none". Then add a test that checks `reasoning_mode` for each of the six models.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. Second selected-six profile check never runs ✓ Resolved

Description

validate_profile now returns straight after validate_published_profile whenever the roster
profile is selected-six. That makes the later `if roster_profile == SELECTED_SIX_PROFILE:
validate_selected_profile(...)` block impossible to reach. A later reader may assume that second
check, and the generic checks around it, still apply to selected-six profiles, when none of them do.

Code

tools/three_flow_admission.py[R893-896]

+    if roster_profile == SELECTED_SIX_PROFILE:
+        from tools.three_flow_selected import validate_selected_profile
+
+        validate_selected_profile(p, provider, model)

Relevance

●●● Strong

Unreachable duplicate validation is a clear control-flow defect and obscures which selected-six
checks actually run.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Lines 757-761 return early for SELECTED_SIX_PROFILE, so control never reaches lines 893-896.
validate_published_profile already calls validate_selected_profile (three_flow_selected.py line
348).

tools/three_flow_admission.py[757-761]
tools/three_flow_admission.py[893-896]
tools/three_flow_selected.py[348]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The selected-six branch near the end of `validate_profile` can never run, because the function returns earlier for that roster profile.

## Fix Focus Areas
- tools/three_flow_admission.py[893-896]

## Recommended Fix
Delete the unreachable block. Add a comment at the early return saying selected-six profiles are fully validated by `validate_published_profile`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

Review mode: Auto: ⚖️ Balanced: Security-focused tests cover independent sanitization, pacing, and refusal paths.

Tip of the day

💡 Did you know, you can route each severity your way: inline, summary, both, or drop

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Previous reviews

Review updated until commit 097d050 ⚖️ Balanced

Results up to commit 05296b6 ⚖️ Balanced

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Directory race bypasses symlink refusal ✓ Resolved

Description

scaffold_operation checks ancestors once but then creates and opens files through pathname
traversal, so another process can replace an ancestor or the new target directory with a symlink
between checks. The command can consequently create scaffold files outside the requested real
directory despite its explicit no-symlink safety contract.

Code

src/operatebench/sdk/scaffold.py[R185-190]

+        for filename, content in rendered:
+            path = target / filename
+            flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL
+            flags |= getattr(os, "O_NOFOLLOW", 0)
+            try:
+                descriptor = os.open(path, flags, 0o644)

Relevance

●●● Strong

The advertised no-symlink safety contract remains vulnerable to a concrete pathname traversal race.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Ancestor symlinks are checked before template loading and directory creation, leaving a race window.
target.mkdir, target.lstat, and os.open(target / filename, ...) subsequently traverse path
components again; O_NOFOLLOW protects only filename, so replacement of an ancestor or target
redirects all writes.

src/operatebench/sdk/scaffold.py[82-89]
src/operatebench/sdk/scaffold.py[148-181]
src/operatebench/sdk/scaffold.py[185-204]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Eliminate the TOCTOU window that lets directory components be replaced by symlinks after validation and before file creation.

## Issue Context
`O_NOFOLLOW` applies only to the final file component; it does not protect the path's parent directories. Anchor creation to verified directory descriptors and revalidate identities where platform support requires a fallback.

## Fix Focus Areas
- src/operatebench/sdk/scaffold.py[82-89]
- src/operatebench/sdk/scaffold.py[148-161]
- src/operatebench/sdk/scaffold.py[174-204]
- tests/test_operation_scaffold.py[103-143]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Lookup shadows operation identity ✓ Resolved

Description

The registry validates operation types and CLI lookup names in separate namespaces, so one pack's
canonical ID or alias may equal another pack's operation_type. That ambiguous string resolves to
the wrong pack at the CLI boundary, defeating the registry's unique-identity guarantee and relying
on each implementation to catch the mismatch later.

Code

src/operatebench/sdk/registry.py[R157-164]

+            for name in (metadata.pack_id, *metadata.aliases):
+                prior = names.get(name)
+                if prior is not None:
+                    raise _problem(
+                        f"lookup name {name!r} is claimed by both "
+                        f"{prior.metadata.pack_id!r} and {metadata.pack_id!r}"
+                    )
+                names[name] = pack

Relevance

●●● Strong

Cross-namespace identity collisions undermine the explicitly promised unique registry and CLI
selection guarantee.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The constructor checks duplicate operation types only against operation_types, while canonical IDs
and aliases are checked only against names, with no cross-map validation. The CLI resolves
--pack through this lookup map, and SDK metadata separately defines operation_type, so a
cross-pack collision makes one identity name the wrong implementation.

src/operatebench/sdk/registry.py[147-168]
src/operatebench/cli.py[172-184]
src/operatebench/sdk/api.py[90-105]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Reject collisions between every pack lookup name (canonical ID and aliases) and every other pack's `operation_type`; allow equality only when both identities belong to the same pack.

## Issue Context
The current constructor tracks lookup names and operation types independently, permitting an operation identity to dispatch another pack.

## Fix Focus Areas
- src/operatebench/sdk/registry.py[147-168]
- tests/test_operation_pack_registry.py[64-81]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Results up to commit a6051cb 🧠 Deep

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Mistral guidance omits valid token data ✓ Resolved

Description

RESPONSE_EXTENSION_SCHEMA now accepts completion_tokens_details.reasoning_tokens, but the
provider guide still describes a closed contract containing only
prompt_tokens_details.cached_tokens. When a Mistral response contains reasoning-token details,
readers following that guide will treat an accepted response as outside the documented contract; the
guide also says extension values are not read, although the new subset check reads them.

Code

src/operatebench/providers/mistral_chat.py[R187-189]

+        "completion_tokens_details": MappingProxyType(
+            {"reasoning_tokens": EXTENSION_COUNT}
+        ),

Relevance

●●● Strong

Documentation contradicts the newly accepted and validated reasoning-token extension behavior.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The changed schema adds the reasoning-token object, and the changed subset check reads extension
values. The unchanged provider guide says the closed contract has only the cached-token object and
that the build reads nothing from it.

src/operatebench/providers/mistral_chat.py[185-203]
src/operatebench/providers/mistral_chat.py[233-239]
docs/PROVIDER_MATRIX.md[460-489]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Mistral provider guide documents only the old cached-token extension, while the validator now accepts reasoning-token details and checks both subsets against their totals.

## Fix Focus Areas
- docs/PROVIDER_MATRIX.md[460-489]
- src/operatebench/providers/mistral_chat.py[185-203]
- src/operatebench/providers/mistral_chat.py[233-239]

## Recommended Fix
Update the guide's closed field list and validation description to match both accepted extension objects and their subset checks. Keep the distinction between validating token counts and using them for pricing.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Mistral Large run is recorded with the wrong reasoning mode ✓ Resolved

Description

For selected-six models the settings fill in reasoning_mode with http_reasoning_mode(model).
That helper calls minimum_effort, which only returns "none" for the three GPT models, so
mistral-large-4 is recorded as "MINIMUM". The request it actually sends is `reasoning_effort:
"none"`. The new override block only fixes the mode for River models, so the stored settings for the
Mistral observation contradict the real request and the published "reasoning": "none" in
docs/studies/2026-10-selected-six.json.

Code

tools/three_flow_http.py[R542-548]

+            if provider == "river_http":
+                self.settings.pop("reasoning_items_contract", None)
+                self.settings.pop("store", None)
+                self.settings["owner_output_cap_tokens"] = 30000
+                self.settings["reasoning_mode"] = (
+                    "MINIMUM" if model == "GLM-5.3" else "OFF"
+                )

Relevance

●●● Strong

Stored reasoning mode contradicts the actual Mistral request and published observation metadata.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

mode_fields sends {"reasoning_effort": "none"} for Mistral (three_flow_selected.py line 25). But
the settings block at three_flow_http.py lines 486-490 sets
reasoning_mode=http_reasoning_mode(model). http_reasoning_mode returns "OFF" only when
minimum_effort(model) == "none", and minimum_effort is "none" only for gpt-6-luna, gpt-6-sol and
gpt-6.1-terra (three_flow_profiles.py line 33). The new selected-six override (lines 542-548)
corrects reasoning_mode only when the provider is river_http. No selected-six test checks
reasoning_mode.

tools/three_flow_http.py[486-493]
tools/three_flow_http.py[535-548]
tools/three_flow_selected.py[16-29]
tools/three_flow_profiles.py[32-33]
tools/three_flow_profiles.py[172-173]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
For selected-six runs, `reasoning_mode` in the transport settings comes from `http_reasoning_mode(model)`. That labels `mistral-large-4` as MINIMUM even though the request sends `reasoning_effort: "none"`, so the recorded provenance does not match the request.

## Fix Focus Areas
- tools/three_flow_http.py[535-548]
- tools/three_flow_profiles.py[32-33]

## Recommended Fix
In the `if selected:` block, set `self.settings["reasoning_mode"]` for every selected-six model from its actual `mode_fields`: "OFF" when the effort is "none", otherwise "MINIMUM". Alternatively, add `mistral-large-4` to the models that `minimum_effort` treats as "none". Then add a test that checks `reasoning_mode` for each of the six models.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Second selected-six profile check never runs ✓ Resolved

Description

validate_profile now returns straight after validate_published_profile whenever the roster
profile is selected-six. That makes the later `if roster_profile == SELECTED_SIX_PROFILE:
validate_selected_profile(...)` block impossible to reach. A later reader may assume that second
check, and the generic checks around it, still apply to selected-six profiles, when none of them do.

Code

tools/three_flow_admission.py[R893-896]

+    if roster_profile == SELECTED_SIX_PROFILE:
+        from tools.three_flow_selected import validate_selected_profile
+
+        validate_selected_profile(p, provider, model)

Relevance

●●● Strong

Unreachable duplicate validation is a clear control-flow defect and obscures which selected-six
checks actually run.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Lines 757-761 return early for SELECTED_SIX_PROFILE, so control never reaches lines 893-896.
validate_published_profile already calls validate_selected_profile (three_flow_selected.py line
348).

tools/three_flow_admission.py[757-761]
tools/three_flow_admission.py[893-896]
tools/three_flow_selected.py[348]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The selected-six branch near the end of `validate_profile` can never run, because the function returns earlier for that roster profile.

## Fix Focus Areas
- tools/three_flow_admission.py[893-896]

## Recommended Fix
Delete the unreachable block. Add a comment at the early return saying selected-six profiles are fully validated by `validate_published_profile`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Results up to commit 44db884 ⚖️ Balanced

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Mistral guidance accepts a rejected response ✓ Resolved

Description

The revised Mistral section says its response contract accepts
completion_tokens_details.reasoning_tokens and validates both token subsets, but the
mistral-small-2603 lane it describes still permits only prompt_tokens_details.cached_tokens and
has no subset check. A reader using this guide for that lane would expect a response with
reasoning-token details to be accepted, while its wire validator rejects the undeclared usage field.

Code

docs/PROVIDER_MATRIX.md[465]

+| `usage` | `completion_tokens_details` | closed object with required `reasoning_tokens` |

Relevance

●●● Strong

Accepted documentation correction: the guide must match the lane's actual wire schema and validation
behavior.

PR-#31

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The guide names mistral-small-2603 as the lane under discussion. Its boundarybench schema allows
only cached-token details, and the usage wire shape admits only that schema's extension names; the
operatebench adapter separately defines the expanded schema and subset check.

docs/PROVIDER_MATRIX.md[451-467]
docs/PROVIDER_MATRIX.md[479-484]
src/boundarybench/providers/mistral_chat.py[379-387]
src/boundarybench/providers/mistral_chat.py[603-611]
src/boundarybench/providers/mistral_chat.py[1159-1161]
src/operatebench/providers/mistral_chat.py[181-240]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Mistral section describes the historical `mistral-small-2603` lane but now assigns it the expanded response contract implemented by the selected-six adapter. The historical lane rejects reasoning-token details and does not check token subsets.
## Fix Focus Areas
- docs/PROVIDER_MATRIX.md[451-511]
- src/boundarybench/providers/mistral_chat.py[379-414]
- src/operatebench/providers/mistral_chat.py[181-240]
## Recommended Fix
Restore the historical lane's exact accepted shape and validation description in its section. Document the selected-six adapter's additional reasoning-token extension and subset checks separately, identifying the lane to which they apply.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Results up to commit 57e8beb 🚀 Fast

No changes from previous review

Results up to commit 8508ad9 🧠 Deep

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Recovery guide still says final data is missing ✓ Resolved

Description

RECOVERY_INTAKE.md calls the checked-in packet an old partial v1 result (162 scored, 98 excluded,
4 aborted), says no final packet is supplied, and claims the actual-data goldens remain pinned to
stale v1 data. Because this PR already checks in the final schema-2 packet (207 scored, 57 excluded,
0 aborted), pins its SHA256 and publishes it in MODEL_RESULTS.md, readers following the guide will
treat the published aggregate as pending and repeat completed intake steps.

Code

docs/studies/RECOVERY_INTAKE.md[R1-3]

+# Pending owner technical recovery intake
+
+Current checked-in v1 packet is the OLD PARTIAL (162 scored, 98 excluded, 4 aborted), not final technical recovery. This prep does not approve publication. Keep packet bytes and approval pins unchanged until parent approval.

Relevance

●●● Strong

Guide contradicts this PR’s published final packet and would mislead readers into repeating
completed intake steps.

PR-#31

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The guide’s lines 3, 22 and 24 describe a partial v1 packet, a pending replacement and stale
goldens. In contrast, the checked-in docs/studies/2026-10-selected-six.json is schema version 2
with full provenance, and its per-model counts total 207 scored, 57 excluded (Mistral 20, Grok 29
and GLM 8) and 0 aborted. tools/check_public_release.py lines 962–969 pin the final approved
digests a5032145… and af31cc19…, while MODEL_RESULTS.md identifies the approved aggregate and
reports 207 scored results; replacing the packet, updating the pins and publishing the result have
therefore already happened.

docs/studies/RECOVERY_INTAKE.md[1-24]
docs/studies/2026-10-selected-six.json[36-96]
tools/check_public_release.py[961-969]
docs/studies/RECOVERY_INTAKE.md[1-3]
docs/studies/RECOVERY_INTAKE.md[15-24]
docs/MODEL_RESULTS.md[5-7]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`docs/studies/RECOVERY_INTAKE.md` presents the published final aggregate as a pending partial v1 packet, says no final packet is supplied, and claims actual-data goldens await final intake, even though the approved v2 packet has been integrated and published.

## Fix Focus Areas
- docs/studies/RECOVERY_INTAKE.md[1-24]

## Recommended Fix
Update the header and status claims to state that final intake and publication are complete: the approved v2 packet has 207 scored, 57 excluded and 0 aborted results, and its SHA256 (`a5032145…`) has been pinned. Remove claims that the final packet is missing or that actual-data checks still await intake; retain the former intake steps only if clearly marked as historical, or delete or archive the guide if it was intended solely for preparation.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Results up to commit ff1a7e3 ⚖️ Balanced

No changes from previous review

[khanukov]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit 44db884

[khanukov]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit 57e8beb

[khanukov]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit 8508ad9

[khanukov]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit ff1a7e3

[khanukov]

/agentic_review

[qodo-code-review]

Code review by qodo was updated up to the latest commit 097d050