Add mod: fix tool_choice enforcement for shared parser engines (Qwen3 et al.) - #354
Open
ptorsten wants to merge 1 commit into
Open
Add mod: fix tool_choice enforcement for shared parser engines (Qwen3 et al.)#354ptorsten wants to merge 1 commit into
ptorsten wants to merge 1 commit into
Conversation
vLLM silently skips xgrammar structural-tag enforcement of tool_choice="required", named tool choice, and strict tools whenever the reasoning and tool parsers resolve to the same parser engine — which is the recommended config for Qwen3 (qwen3 + qwen3_xml/qwen3_coder) and every other engine-backed pair (DeepSeek V3.2/V4, MiniMax M2, Gemma4, Kimi K2, GLM 4.7, Seed-OSS, Nemotron V3, Inkling, Mistral). ParserManager.get_parser returns the engine class directly on that path, bypassing DelegatingParser.adjust_request, the only place the structural tag was applied. The model can then answer tool_choice="required" requests in plain text with no error. Found via tool-eval-bench TC-45 on Qwen3.8-27B-NVFP4; verified on the nightly-20260821-based runtime: the mod applies, the self-check passes, the grammar forces a call even when no offered tool fits the prompt, tool_choice=auto stays unconstrained, and spec-decode acceptance is unaffected. tool-eval-bench score 87 -> 88 with TC-45 fail -> pass as the only change. Fix also being submitted to vLLM upstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AYRyJkYP9PuXXombMaPbuT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New mod
mods/fix-tool-choice-enforcementpatching a vLLM bug:tool_choice="required", named tool choice, andstricttools are silently unenforced whenever the reasoning and tool parsers resolve to the same parser engine — which is the recommended configuration for Qwen3 (--reasoning-parser qwen3 --tool-call-parser qwen3_xml/qwen3_coder) and every other engine-backed pair (DeepSeek V3.2/V4, MiniMax M2, Gemma4, Kimi K2, GLM 4.7, Seed-OSS, Nemotron V3, Inkling, Mistral).Why
ParserManager.get_parserhas a shared-engine shortcut that returns the engine class directly, bypassingDelegatingParser.adjust_request— the only place the xgrammar structural tag is applied.VLLM_ENFORCE_STRICT_TOOL_CALLINGdefaults to true and has no effect on this path. The failure is invisible whenever the model naturally wants to call a tool; it surfaces when no offered tool fits the prompt — the model then answerstool_choice="required"requests in plain prose with no error.Found via tool-eval-bench TC-45 ("tool_choice=required Compliance") on Qwen3.8-27B-NVFP4.
The mod
Standard
run.sh+ diff, following thefix-qwen3-coder-nextpattern:patch -p1, tolerant of already-fixed images)structural_tag_modelVerification (nightly-20260821-based runtime, Qwen3.8-27B-NVFP4 on GB10)
tool-choice enforcement fix active: Qwen3Parserrequirednow forces a call even when no offered tool fits the prompt (impossible without a binding grammar); named tool choice enforced;autoand plain chat unregressedtests/parser/enginetests pass with the patch appliedThe same fix is being submitted to vLLM upstream; once it lands in nightlies,
run.shdegrades gracefully ("Patch not applicable, skipping").🤖 Generated with Claude Code