Skip to content

feat(api): opt-in prompt-injected tool translation for OLMoE and Qwen3.6 - #1497

Open
basil-k-aji-dev wants to merge 1 commit into
JustVugg:devfrom
basil-k-aji-dev:feat/tool-fallback-olmoe-qwen36
Open

basil-k-aji-dev wants to merge 1 commit into
JustVugg:devfrom
basil-k-aji-dev:feat/tool-fallback-olmoe-qwen36

Conversation

@basil-k-aji-dev

Copy link
Copy Markdown

Takes option (a) from #1378: "a prompt-injected fallback in the gateway for those families, behind a flag, which I would take as a PR against openai_server.py with the two-turn test."

One correction to the issue first

The comment groups OLMoE, Qwen3.6 and Inkling as the three families that answer 400 Unsupported role 'tool'. Inkling does not — its role_token map in render_chat_inkling already contains "tool": "<|message_tool|>", so a role: "tool" turn renders natively today. Measured on dev:

family tools[] role: "tool"
OLMoE 400 400
Qwen3.6 400 400
Inkling 400 unsupported_parameter renders

So Inkling only refuses tools[], and it has a native tool slot rather than "no native tool format". Since the stated reason for grouping it does not hold, I left Inkling alone — whether it should get the declaration wired to its native token is a different change and your call, not something to fold in here.

This PR covers OLMoE and Qwen3.6 only.

What it does

COLI_TOOL_FALLBACK=1 (naming follows the existing COLI_TOOL_SALVAGE) opts into writing the tool protocol as ordinary turns for those two families:

  • the declaration goes in as a system turn
  • prior assistant tool_calls are replayed inside the assistant turn
  • a role: "tool" result rides in as a user turn, wrapped in <tool_response>

The wire format is GLM's on purpose. parse_arch_tool_calls falls through to parse_tool_calls for both of these families, and that parser reads <tool_call>NAME<arg_key>K</arg_key><arg_value>V</arg_value></tool_call>. Asking the model for any other shape would produce calls nothing can read back. Round-trip on the exact string the injected preamble asks for:

[api] tool-calls: 1 total, 1 strict, 0 unclosed-recovered, 0 de-mangled [CLEAN]
  -> {"name": "get_weather", "arguments": "{\"city\": \"Paris\", \"days\": 2}"}

days arrives as an int, so the schema-driven _coerce_arg path works through the fallback too.

Default off. Without the flag the 400 is unchanged, except the message now names the flag so the failure is self-explaining.

What this is not

These templates were never trained on tool syntax — that is why it is a flag and not the default. This buys protocol compatibility for an agent loop; it does not make either model good at deciding when to call a function. I have no OLMoE or Qwen3.6 checkpoint here, so the rendering and the parse round-trip are verified, the models' call quality is not. Worth saying plainly rather than implying otherwise.

Tests

New c/tests/test_openai_tools_fallback_e2e.py, on the same mock-engine harness as the existing test_openai_tools_*_e2e.py (no checkpoint needed). 8 tests, both families:

  • the two-turn loop the issue asks for: tools[] -> parsed tool_calls -> role: "tool" turn -> second completion, asserting the declaration, the replayed call and <tool_response> all reached the prompt
  • with the flag unset, tools[] and role: "tool" both still 400, and the error names the flag

Note for anyone extending it: qwen36's config.json model_type is qwen3_5_moe, not the --arch id — the test carries an explicit mapping.

Verification

  • Flag off leaves the prompts byte-identical. I hashed both renderers over 2 thinking modes x 4 message shapes before and after the change: all 16 identical. That is the risk that mattered here, since I restructured both message loops.
  • 8 new tests pass; existing test_openai_server, test_openai_tools_e2e, test_openai_tools_k3_e2e, test_openai_tools_v4_e2e, test_openai_tools_v41_e2e all pass.
  • make check green, exit 0: portable CPU build, C unit tests, and 893 Python tests OK (71 skipped).

Built and checked on aarch64 Linux (GCC 14). No CUDA path touched, so no cuda-test run.

OLMoE and Qwen3.6 have no tool syntax in their chat templates, so the
gateway refuses tools[] and role:"tool" rather than invent a format.
COLI_TOOL_FALLBACK=1 opts into a translation for those two families: the
declaration, the prior assistant calls and the tool results are written as
ordinary turns, in the GLM wire format parse_tool_calls() already reads
back — the parser parse_arch_tool_calls() falls back to for both.

Default off, so a client that does not set the flag still gets the same
400 (now naming the flag). With the flag unset the rendered prompts are
byte-identical to before.

Refs JustVugg#1378
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant