Add mods/fix-anthropic-templateless-inline-system: stop hoisting inline system messages for templateless models

[yaro-tal]

What

One line, in vllm/entrypoints/anthropic/serving.py:

if not chat_template:
-    return True
+    return False

plus a VLLM_ANTHROPIC_MERGE_INLINE_SYSTEM escape hatch.

Why

_detect_merge_inline_system (added in vllm-project/vllm#46025) decides whether to merge inline role: system messages into the leading system block by rendering a probe conversation against the jinja chat template. With no template there is nothing to probe, so it returns its documented conservative default: merge.

DeepSeek-V4-Flash has no jinja template — --tokenizer-mode deepseek_v4 renders the prompt in Python instead. So the probe never runs and every request takes the merge path.

Claude Code (>= 2.1.154) sends a per-turn system reminder inline in messages. Merging puts each new one at offset ~0. New reminder each turn → new byte 0 → nothing after it is reusable. On a 200k-token conversation that is the entire prompt, every turn.

Measured on a synthetic 4-turn conversation through the real encoder:

	shared prefix between consecutive turns
stock (merge=True)	4198 chars — exactly the static system prompt, nothing else
patched (merge=False)	100% of the previous turn

On a live 2-node TP=2 DeepSeek-V4-Flash-0731 cluster (2x DGX Spark GB10) driving Claude Code: prefix cache hit rate ~78% → 98.3% (effectively 0 → 98.3% on long conversations), prompt throughput 1600–2400 tok/s of real prefill work → ~80.

The same file already fights this exact battle two functions down, where it drops Claude Code's x-anthropic-billing-header because it "contains a per-request hash that defeats prefix caching". The merge path reintroduces that wholesale, with the entire reminder text.

Scope

Only models with no jinja template at all. Templated models take the existing render-probe path and are bit-for-bit unaffected — the patch changes only the not chat_template branch.

Relationship to upstream

• [Anthropic] Support system role messages inside messages array vllm-project/vllm#44283 (merged) introduced the merge.
• fix(anthropic): preserve inline system message position for prefix caching vllm-project/vllm#44602 (merged) preserved inline position for prefix caching.
• fix(anthropic): auto-detect template support for mid-conversation system messages vllm-project/vllm#46025 (merged) added the detector and chose "no template → merge". That default is what this changes.
• fix(anthropic): resolve model chat template before deciding inline-system merge vllm-project/vllm#46196 (open) resolves the model's template before calling the detector instead of passing the raw --chat-template arg. Complementary, but it does not help here: a --tokenizer-mode model has no jinja template to resolve, so the detector still gets None and still returns True.
• [Bugfix][Frontend] Hoist trailing Anthropic system messages vllm-project/vllm#52978 (open) hoists trailing system messages to fix tool-calling breakage ([Bug]: Anthropic /v1/messages renders system-role messages inside messages positionally into the chat template (breaks Claude Code >=2.1.2xx tool calling) vllm-project/vllm#48874). If that merges it would re-hoist Claude Code's per-turn reminder and undo this win, so the two want reconciling.

If the upstream default ever changes, this mod becomes a no-op and git apply --reverse --check makes run.sh skip cleanly.

Deliberately NOT included

This was originally found alongside a second DeepSeek-V4 defect — a conversation ending in a system message got no <|Assistant|> generation prompt, so the model emitted an immediate EOS. That is now fixed upstream in vllm/tokenizers/deepseek_v4_encoding.py, with a tighter condition than our local patch used, so it is not shipped here.

⚠️ On a build that predates that upstream fix, this mod alone makes things worse — it stops the system message being hoisted, so it now actually reaches the encoder, where the EOS bug lives. The README says so explicitly.

Testing

• git apply --check passes against both vLLM e2666d9a6 (verified by md5 against the pinned base image) and current main.
• Applies, parses, and is --reverse --check idempotent, so run.sh's already-applied guard works.
• Detector re-tested standalone after patching: templateless → False, plain template → False, Qwen loop.first template → True (unchanged), env override honoured in both directions.
• In production on the cluster described above.

Provenance

Written with AI assistance and verified on the hardware described above. The patches in this and the related PR are a collaboration between Claude Opus 5 and DeepSeek-V4-Flash, each reviewing the other's work and keeping the other honest; the measurements are from a real two-node deployment, not from a model's say-so. Please apply whatever policy you have for AI-assisted contributions — flagging it so it's your call rather than my assumption.

🤖 Generated with Claude Code