Skip to content

feat(rollout-gateway): make renderer derender stages pluggable - #94

Merged
Linbo-Liu merged 2 commits into
awslabs:mainfrom
Linbo-Liu:parsing-seams
Jul 29, 2026
Merged

feat(rollout-gateway): make renderer derender stages pluggable#94
Linbo-Liu merged 2 commits into
awslabs:mainfrom
Linbo-Liu:parsing-seams

Conversation

@Linbo-Liu

@Linbo-Liu Linbo-Liu commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:
HfTemplateRenderer derendered model output with one hardcoded path: split reasoning on </think>, then match tool calls with a regex for the <tool_call><function=...> XML format. Any model whose tool-call format differs silently produced zero tool calls.

For example, Qwen3-0.6B has parsing format:

  <tool_call>
  {"name": <function-name>, "arguments": <args-json-object>}
  </tool_call>

while the parser can only handle

  <tool_call>\s*<function=NAME>...<parameter=K>V</parameter>...</function>\s*</tool_call>
Stage Signature Default
reasoning_parser raw_output -> (reasoning, body_text) split on </think>
tool_parser (body_text, tools_schema) -> (text, tool_uses, ill_formed) <tool_call><function=...> regex

The stages run in sequence — reasoning first, tool calls on what remains — matching the order the inference engines use. Overriding one leaves the other on its default, so a model needing engine-grade tool parsing but emitting plain </think> reasoning supplies only tool_parser. The tool stage is skipped entirely when a request carries no tools schema.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

site = web.TCPSite(runner, "127.0.0.1", 0)
await site.start()
port = site._server.sockets[0].getsockname()[1]
@contextmanager

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering why we dropped the async here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a threaded server that can start rollout gateway even in a sync function call (both slime and verl invoke the gateway in sync caller), this integration test is to test whether threaded server can work in a sync caller.

) -> None:
self.tokenizer = tokenizer
self._stop_sequences: list = list(stop_sequences) if stop_sequences else []
self.reasoning_parser: ReasoningParserFn = reasoning_parser or split_reasoning

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible that we build a mapping from supported model family to parser functions?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to implement the parser functions or import from inference engine? Our tenet is to have 0 dependency on vllm/sglang.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am ok to use some lightweight packages like renderers (https://github.com/PrimeIntellect-ai/renderers), or implement and maintain ourselves. Anyway, I think it may improve user experience if we can automatically match the correct parser functions according to the model family used by users?

@Linbo-Liu
Linbo-Liu merged commit f2030fd into awslabs:main Jul 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants