feat(api): opt-in prompt-injected tool translation for OLMoE and Qwen3.6 - #1497
Open
basil-k-aji-dev wants to merge 1 commit into
Open
basil-k-aji-dev wants to merge 1 commit into
basil-k-aji-dev wants to merge 1 commit into
Conversation
OLMoE and Qwen3.6 have no tool syntax in their chat templates, so the gateway refuses tools[] and role:"tool" rather than invent a format. COLI_TOOL_FALLBACK=1 opts into a translation for those two families: the declaration, the prior assistant calls and the tool results are written as ordinary turns, in the GLM wire format parse_tool_calls() already reads back — the parser parse_arch_tool_calls() falls back to for both. Default off, so a client that does not set the flag still gets the same 400 (now naming the flag). With the flag unset the rendered prompts are byte-identical to before. Refs JustVugg#1378
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Takes option (a) from #1378: "a prompt-injected fallback in the gateway for those families, behind a flag, which I would take as a PR against
openai_server.pywith the two-turn test."One correction to the issue first
The comment groups OLMoE, Qwen3.6 and Inkling as the three families that answer
400 Unsupported role 'tool'. Inkling does not — itsrole_tokenmap inrender_chat_inklingalready contains"tool": "<|message_tool|>", so arole: "tool"turn renders natively today. Measured ondev:tools[]role: "tool"unsupported_parameterSo Inkling only refuses
tools[], and it has a native tool slot rather than "no native tool format". Since the stated reason for grouping it does not hold, I left Inkling alone — whether it should get the declaration wired to its native token is a different change and your call, not something to fold in here.This PR covers OLMoE and Qwen3.6 only.
What it does
COLI_TOOL_FALLBACK=1(naming follows the existingCOLI_TOOL_SALVAGE) opts into writing the tool protocol as ordinary turns for those two families:tool_callsare replayed inside the assistant turnrole: "tool"result rides in as a user turn, wrapped in<tool_response>The wire format is GLM's on purpose.
parse_arch_tool_callsfalls through toparse_tool_callsfor both of these families, and that parser reads<tool_call>NAME<arg_key>K</arg_key><arg_value>V</arg_value></tool_call>. Asking the model for any other shape would produce calls nothing can read back. Round-trip on the exact string the injected preamble asks for:daysarrives as an int, so the schema-driven_coerce_argpath works through the fallback too.Default off. Without the flag the 400 is unchanged, except the message now names the flag so the failure is self-explaining.
What this is not
These templates were never trained on tool syntax — that is why it is a flag and not the default. This buys protocol compatibility for an agent loop; it does not make either model good at deciding when to call a function. I have no OLMoE or Qwen3.6 checkpoint here, so the rendering and the parse round-trip are verified, the models' call quality is not. Worth saying plainly rather than implying otherwise.
Tests
New
c/tests/test_openai_tools_fallback_e2e.py, on the same mock-engine harness as the existingtest_openai_tools_*_e2e.py(no checkpoint needed). 8 tests, both families:tools[]-> parsedtool_calls->role: "tool"turn -> second completion, asserting the declaration, the replayed call and<tool_response>all reached the prompttools[]androle: "tool"both still 400, and the error names the flagNote for anyone extending it: qwen36's
config.jsonmodel_typeisqwen3_5_moe, not the--archid — the test carries an explicit mapping.Verification
test_openai_server,test_openai_tools_e2e,test_openai_tools_k3_e2e,test_openai_tools_v4_e2e,test_openai_tools_v41_e2eall pass.make checkgreen, exit 0: portable CPU build, C unit tests, and 893 Python tests OK (71 skipped).Built and checked on aarch64 Linux (GCC 14). No CUDA path touched, so no
cuda-testrun.