Determine the benchmark type from §6.1's load pattern

[arav-agarwal2]

Four §9.1 rows apply differently to agentic benchmarks, but §8.3 carries no field naming the benchmark type. offline-point-present admitted as much in its own docstring and warned instead of erroring:

Skipped entirely for agentic benchmarks… This checker has no way to tell an agentic curve from a single-turn one yet — §3 does not surface the distinction in any file it reads — so the absent case is reported as a WARNING rather than an error.

§6.1's load pattern answers it. The reference implementation names its fixed-concurrency agentic scheduler agentic_inference, and that name is the only agentic signal in any file a submission carries:

runtime_settings:
  load_pattern: agentic_inference

The determination

PointConfig.is_agentic is the per-point signal. ModelContext.is_agentic is the curve-level answer and requires unanimity — §8.5 defines a result as one system, one benchmark model, one dataset, so a curve is one benchmark. A curve whose points disagree is reported by a new benchmark-type-consistency ERROR and read as single-turn: the stricter branch, so one mislabelled point cannot switch off §5.7.

What swings on it

Rule	Single-turn	Agentic
load-pattern	concurrency	agentic_inference — both accepted; poisson and friends stay out
point-count	8 with a dedicated Offline run, else 7	always 7; a dedicated declaration cannot lift it to 8
offline-point-present	ERROR when absent (was WARN)	ERROR when present
accuracy-coverage	N=5	N=4

offline-point-present inverts because §5.7 is explicit:

An agentic submission neither requires nor may include an Offline point.

and §9.1 asks for "exactly one … for non-agentic benchmarks; none is present for agentic benchmarks".

accuracy-coverage needed no logic change — N falls out already, since the Offline row iterates offline_points, which an agentic curve has none of. Its pass message now names which N applied so a reviewer can see the reading. offline-ordering needed nothing either: it already no-ops without a dedicated point.

concurrency          agentic=False min_points=7 offline-point-present=ERROR  No point declares `offline`…
agentic_inference    agentic=True  min_points=7 offline-point-present=INFO   Agentic benchmark: no Offline point, as §5.7 requires

Fixture fallout

Flipping the WARN to an ERROR broke 9 tests, because no fixture declared offline at all — the hedge had been covering a corpus that is non-compliant under v1.0. Fixed at the source rather than per-fixture: regenerate_fixtures.py now elects the C_max point as the Offline result (§5.7.2 Option 2), which adds no run and leaves every point count unchanged. 12 fixtures gain one line each; --check is clean, so it stays idempotent.

The builder contract test needed its curve moved rather than patched: it topped out at 1000 against max_supported_concurrency: 1024, so no point existed to elect. The builder itself is untouched — it must not invent §8.3 disclosure (#72), so the declaration lives in the test's synthetic input.

Verification

1,016 tests pass (was 976) · mypy --strict clean · ruff clean · sphinx -W clean. 40 new tests covering the determination, the unanimity rule, both offline-point-present branches, both point minimums, and each agentic gate's aggregation.

Second commit: the agentic models and their accuracy gates

§3.2 makes the reference implementation the authority for per-model specifications, so the models and the thresholds both come from its Agentic Inference example. Models, as #93 adds them: kimi-k3, qwen3.6-35b-a3b, deepseek-v4.1-flash.

The accuracy gates could not reuse §15's. That gate folds every dataset of a point into one sample-weighted score per metric; the agentic benchmarks gate three quantities aggregated three different ways, and folding them together gives a number with no meaning under either rule. So §15's gate stands down for a recognised agentic model and agentic_targets.py + _check_agentic_accuracy take over:

Rule	Quantity	Source	Aggregation
agentic-accuracy-inline	Inline accuracy	agentic_combined	per point — "Every … submitted Pareto point must satisfy all of the model-specific accuracy thresholds"
agentic-accuracy-swebench	SWE-bench	swe_bench	mean-of-4 — §4.3's multi-turn branch: "the arithmetic mean … MUST meet the quality threshold; individual results need not"
agentic-osl-range	OSL per-turn mean	result_summary.json	per point, against a range

Thresholds: Kimi K3 inline ≥ 58.32, OSL 425–520, SWE-bench ≥ 93.5. Qwen3.6-35B-A3B ≥ 55.86, 344–422, ≥ 69.

Two details worth a reviewer's eye:

• The OSL field is resolved explicitly and never falls back to the windowed output_sequence_lengths, which has the same shape and a different value. The README is emphatic about this, and a silent fallback would gate the wrong number — so an absent full-run block warns rather than substituting.
• DSV4 is recognised but every threshold is TBD upstream, so it reports as ungateable rather than passing silently. That is a distinct report from an unrecognised model, which also warns.

A short SWE-bench set is still gated, on the mean of what is present, and flagged as short — three strong points passing unremarked seemed the wrong failure mode, and the missing results are accuracy-coverage's report.

Open items

• Overlaps fix(checker): accept native reports and agentic run configurations #93. That PR also admits agentic_inference as a load pattern but does not wire the distinction to any rule, so the two will conflict on _check_load_pattern. Either let fix(checker): accept native reports and agentic run configurations #93 keep the vocabulary change and rebase this on top as the wiring, or fold this into fix(checker): accept native reports and agentic run configurations #93.
• §4.4 steady state deliberately untouched. §4.4 scopes itself to ConcurrencyScheduler points and says "single-pass agentic workloads are handled only by the ad-hoc diagnostic tool", but agentic_inference is a fixed-concurrency scheduler — whether an agentic point is "single-pass" is not something the load pattern answers, so exempting it would be a guess.
• The model list should not be in a release. §3.2 publishes it "at least 6 weeks before the submission round opens", so a new round should not need a new checker. data/seed_sets.yaml and data/approved_drafters.yaml are the pattern; noted at the list rather than done here, since fix(checker): accept native reports and agentic run configurations #93 adds the names the same way.
• Worth a WG request: an explicit §8.3 benchmark-type field, which would make all of this a declaration rather than an inference from a scheduler name.

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[anandhu-eng]

Pointing out that in rules, it seems offline load pattern is accepted: https://github.com/mlcommons/endpoints_policies/blob/v1.0_rules_dev/endpoints_rules.md#61-load-pattern

[anandhu-eng]

https://github.com/mlcommons/endpoints/tree/main/examples/10_Agentic_Inference#dsv4-1

We might have to allign on what the name is

[arav-agarwal2]

I've changed it from flash, but honestly yea. It looks like this may be something to make an issue on and check during rules today.