Reserve output budget for the answer so reasoning never exhausts max_tokens

[randomvariable]

Problem

A reasoning model that exhausts max_tokens while still inside the reasoning block
produces nothing usable. The entire budget is spent, the request finishes with
finish_reason="length", and no answer is ever emitted. The user pays full token cost for
a zero-value response.

This is the single highest-cost failure mode observed in practice, and the literature
measures it as dominant rather than marginal:

• arXiv:2607.11317 — on GSM8K with an INT4 1.5B reasoning model, 49% of incorrect traces
  exhaust the budget without emitting an answer, versus 6% of correct traces.
  Verbatim repetition loops, the failure everyone expects, account for ≤4% of tokens.
  Incorrect answers consume 1286 tokens against 828 for correct ones.
• arXiv:2607.21433 — reasoning generations are bimodal: converged traces reach 90.3%
  accuracy on AIME, non-converged traces reach 6.6%, at an overall convergence rate of
  62.0%. A non-converged trace is worth approximately nothing.
• arXiv:2606.00206 — in up to 52% of failures the model already reached the correct
  answer in an intermediate step and then failed to commit to it.

Those combine into a specific, addressable case: the model frequently has the answer,
keeps deliberating, runs out of budget, and emits nothing. Forcing it to stop reasoning
while enough budget remains to write an answer converts a guaranteed-zero outcome into a
plausible one.

Why existing controls do not cover this

thinking_token_budget (vllm-project#20859, vllm-project#34668) forces the reasoning-end token once an absolute
thinking-token count is exceeded. It is the right machinery but the wrong trigger for
this problem:

1. It is not coupled to max_tokens. Setting max_tokens=8000 without a thinking
   budget lets the model think for all 8000 tokens and emit nothing. The two limits must
   be kept mutually consistent by hand, per request.
2. It does not account for prompt length or prior turns. The number of output tokens
   actually available varies per request; a fixed thinking budget cannot track it.
3. It expresses the wrong intent. The operator wants "always leave enough room to
   answer". Expressing that as an absolute thinking cap requires deriving a number that is
   only correct for one prompt length and one max_tokens value.

max_tokens itself is a hard truncation with no awareness of reasoning state.

Proposal

Add a per-request reserve expressed against the output budget:

# SamplingParams
reasoning_answer_reserve: int | None = None

Semantics: while the request is inside the reasoning block, if

max_tokens - len(output_tok_ids) <= reasoning_answer_reserve

then force the reasoning-end token, exactly as thinking_token_budget does today. The
model exits the reasoning block with reasoning_answer_reserve tokens still available to
produce an answer.

Properties that make this the right shape:

• One number, expressing the actual intent. "Reserve 512 tokens for the answer" is
  directly meaningful and portable across prompt lengths and max_tokens values.
• Zero cost when reasoning terminates naturally. If the model closes the block at 30%
  of budget, the reserve never fires. It is a floor, not a schedule.
• Composes with thinking_token_budget. Both are force-close triggers; whichever
  fires first wins. They are not mutually exclusive and should not be validated as such.
• Correct under min_tokens. Interaction must be validated: a reserve smaller than the
  remaining min_tokens requirement is a configuration error and should be rejected at
  request validation, not silently resolved at sample time.

Implementation sketch

ThinkingBudgetStateHolder (vllm/v1/sample/thinking_budget_state.py) already holds
everything needed except the budget numbers:

• tracks in_think per request
• maps requests to logits rows via cu_num_tokens, including bonus-token rows
• has a working force-close path (_apply_forcing_to_logits) that handles multi-token
  close markers and speculative decoding
• already keeps a live reference to output_tok_ids, so tokens-generated is len() on an
  existing field — no new tracking

Required changes:

1. SamplingParams: add reasoning_answer_reserve with validation against max_tokens
   and min_tokens.
2. Plumb through vllm/v1/engine/input_processor.py alongside thinking_token_budget.
3. sync_batch currently creates per-request state only when thinking_token_budget is not None (thinking_budget_state.py:90-99). Admit reserve-only requests, and allow
   _init_state_entry to take a None thinking budget.
4. Store max_tokens in the state entry; evaluate the reserve condition in
   _update_think_state and set the existing force flag.
5. Guard budget-dependent countdown paths so they do not fire when thinking_token_budget
   is None.

The reserve check is integer arithmetic on values already in the state dict. It adds no
device-host sync and no work in the sampling path when the parameter is unset.

Known hazard: tool calls

vllm-project#44676 (open, upstream) reports that thinking_token_budget forces reasoning-end tokens
into the middle of tool-call arguments, because ThinkingBudgetStateHolder does not
treat <tool_call> as an implicit reasoning end.

This feature shares that force-close path and therefore inherits the bug. It must not make
it worse, and must not ship without a test covering forced close during a tool call. Read
vllm-project#44676 before wiring. If the fix for that bug lands upstream first, rebase onto it.

Relationship to other work

• [Core] Add soft thinking token budget with progressive logit bias vllm-project/vllm#45727 (open, draft, upstream) adds a soft logit ramp toward the close token over the
  last 30% of the thinking budget. Directly relevant: the same ramp driven off the
  reserve deadline instead would soften this feature's hard cut, letting the model close
  naturally before the wall. Correct sequencing is to land the hard reserve first as the
  guaranteed floor, then evaluate driving [Core] Add soft thinking token budget with progressive logit bias vllm-project/vllm#45727's ramp off whichever deadline is nearer.
• Reasoning-block marker penalty to reduce overthinking under quantized serving #27 (this fork) proposes a reasoning-block marker penalty. Complementary: the penalty
  reduces the rate at which the model opens new branches, which should reduce how often the
  reserve fires at all. Both should be measured together — the reserve's trigger rate is a
  clean, cheap proxy metric for whether the penalty is working.

Evaluation

Primary metric: extractable-answer rate. The fraction of requests that produce a
parseable final answer, as distinct from accuracy. A truncated trace scores zero here even
though today it is often invisible in accuracy-only reporting because the request is simply
dropped from scoring.

Also track:

• truncation / finish_reason="length" rate
• accuracy on requests that were forced (are forced answers better than nothing, and by how
  much — arXiv:2607.21433's 6.6% non-converged baseline is the number to beat)
• accuracy on requests that were not forced, to confirm no regression when the feature is
  inactive
• reserve trigger rate as a function of reasoning_answer_reserve

Sweep reasoning_answer_reserve ∈ {128, 256, 512, 1024} against a fixed max_tokens, on
at least one quantized reasoning model, on GSM8K and AIME. GSM8K because the non-termination
result above was measured there; AIME because arXiv:2607.21433's convergence bimodality was
measured there and it has a much lower natural convergence rate.

Acceptance criteria

• Extractable-answer rate improves by ≥10 percentage points at some reserve value on a
  workload with a measured baseline truncation rate ≥15%.
• Accuracy on forced requests exceeds the non-converged baseline: strictly better than
  answering not at all, reported with a paired comparison against the unforced run.
• No accuracy change on requests where the reserve never fires: McNemar p ≥ 0.05.
• Zero measurable throughput change versus the parameter being unset, ≥3 runs, median
  with spread.
• Correct under speculative decoding: forced close occurs at the same output position
  as without spec decode.
• Test coverage for forced close during a tool call, and the interaction with
  thinking_token_budget when both are set.
• Off by default; unset adds no work to the sampling path.

Work items

1. SamplingParams field, validation against max_tokens / min_tokens.
2. Input-processor plumbing.
3. ThinkingBudgetStateHolder: admit reserve-only requests, store max_tokens, evaluate
   reserve in _update_think_state.
4. Frontend exposure on the chat-completion protocol.
5. Tests: reserve triggers at the right position; no-fire path unchanged; spec-decode
   equivalence; tool-call interaction; combined with thinking_token_budget.
6. Docs in docs/features/reasoning_outputs.md — what it does, how it differs from
   thinking_token_budget, what happens when both are set, and the tool-call caveat.
   Google-style docstrings on the config field feed the generated reference.

Duplicate check

• gh issue list --repo vllm-project/vllm --search "reserve tokens final answer" — no
  match.
• gh issue list --repo vllm-project/vllm --search "force end thinking before max_tokens"
  — no match. Closest is [RFC]: GLM-5.x thinking_token_budget defaults and regression tests vllm-project/vllm#48201 (GLM-5.x thinking_token_budget defaults), which tunes the
  existing absolute budget rather than coupling it to the output budget.
• [Feature] limit thinking tokens (hard limit) vllm-project/vllm#20859 / [Reasoning][Feature] Support for speculative decoding with thinking budget vllm-project/vllm#34668 are merged and are the foundation this builds on.
• [Core] Add soft thinking token budget with progressive logit bias vllm-project/vllm#45727 is adjacent, mechanism documented above.
• [Bug]: thinking_token_budget forces reasoning_end tokens into the middle of tool-call arguments — ThinkingBudgetStateHolder doesn't treat <tool_call> as implicit reasoning end (Qwen3.5+) vllm-project/vllm#44676 is the tool-call hazard, documented above.