Skip to content

fix(harness): give a pi tree Reef's reply budget, and name a reply cut at it - #566

Merged
quao627 merged 1 commit into
mainfrom
fix/pi-reply-budget
Sep 21, 2026
Merged

quao627 merged 1 commit into
mainfrom
fix/pi-reply-budget

Conversation

@quao627

@quao627 quao627 commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Three consecutive reefine requests answered by the coding-agent proposer on anthropic/claude-sonnet-5 produced nothing, each recorded as "the agent changed no entry". The step record shows why: the agent's last model call returned usage.output_tokens = 16384, exactly its cap, with empty content and stopReason: "length". The model spent the whole reply budget on reasoning, so the turn carried no text and no tool call, pi read that as the end of the turn and exited 0, and Reef read back a workspace nobody had written to.

The cap is pi's own fallback: maxTokens: definition.maxTokens ?? 16384 in its provider composer. Reef renders pi-agent/models.json with the model id and nothing else, so every pi tree Reef produces has always taken that fallback. It never showed before because the agent proposer is days old (#538) and the model it was first run against answered within the budget.

#565 surfaces a failed model response from the trajectory, but only for stopReason error or aborted; a reply cut at length still reported as a step that changed nothing.

Changes

  • ModelBinding gains max_output_tokens (default 32000), the reply budget a harness bound to that endpoint asks for. It sits beside timeout_s, which already describes how Reef's callers talk to the endpoint, and is validated positive.
  • compose_nodes passes it to the descriptor templates, and _substitute now returns the value itself for a string that is one placeholder and nothing else, so a number stays a number in the rendered JSON.
  • The pi descriptor renders maxTokens into models.json for all three dialects, so an evaluation episode, an agent run, a trial and the installed client tree all carry Reef's budget instead of pi's 16384.
  • Reading the trajectory, stopReason: "length" now records what happened: the reply hit its token budget and carried no answer, naming maxTokens. A person reading the step page no longer sees "changed no entry" for a run that was cut off.

Compatibility and operational impact

Every pi tree Reef renders from now on carries maxTokens: 32000 where it previously carried none; a deployment whose model answers within a smaller budget is unaffected, since the field is a cap, not a request. Providers clamp a cap above a model's real limit (verified against OpenRouter with 60000 for anthropic/claude-sonnet-5). A deployment that wants another value sets max_output_tokens on the binding. No wire or persisted format changes.

Verification

.venv/bin/python -m pytest tests/reef_service -q \
  --ignore=tests/reef_service/test_sao_bridge.py \
  --ignore=tests/reef_service/test_slime_bridge.py \
  --ignore=tests/reef_service/test_multi_scenario_bridge.py
3877 passed, 78 skipped

pre-commit run --files <changed files>   # all hooks passed
.venv/bin/python -m mypy                 # Success: no issues found in 321 source files

The three ignored modules fail to collect on this machine without ray and slime, on main as well.

New tests: a pi tree carries the binding's budget as a JSON number and rejects a non-positive one; an agent whose reply was cut at the budget records that rather than "changed no entry".

AI assistance

Claude Code diagnosed the failure from the step records, wrote the change, the tests and this description; the author reviewed the change.

Checklist

  • The change is focused and contains no unrelated cleanup.
  • Tests cover behavior changes, or this pull request does not change behavior.
  • Public interface changes include contract tests, or no public interface changes are present.
  • Affected user and developer documentation is updated, or no documentation update is required.
  • Root README changes update both languages and README.i18n.yaml, or do not affect README content.
  • An accepted RFC is linked, or this change does not require an RFC.
  • Relevant pre-commit and test checks pass.
  • Non-trivial AI assistance is disclosed above, or none was used.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@quao627 quao627 added bug Reproducible incorrect or unexpected behavior area: harness Harness integration, recipes, and evolution surfaces labels Sep 21, 2026
@quao627 quao627 added bug Reproducible incorrect or unexpected behavior area: harness Harness integration, recipes, and evolution surfaces labels Sep 21, 2026
@quao627
quao627 enabled auto-merge (squash) September 21, 2026 19:50
@github-actions

Copy link
Copy Markdown

Merge Oncall: @BobbyZhouZijian

The merge oncall will route reviews and keep this pull request moving toward a clear decision.

@github-actions github-actions Bot added the area: service Service APIs, deployment, and command-line interfaces label Sep 21, 2026
@quao627
quao627 merged commit 99f4ee6 into main Sep 21, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: harness Harness integration, recipes, and evolution surfaces area: service Service APIs, deployment, and command-line interfaces bug Reproducible incorrect or unexpected behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants