feat(rollout-gateway): make renderer derender stages pluggable - #94
Conversation
| site = web.TCPSite(runner, "127.0.0.1", 0) | ||
| await site.start() | ||
| port = site._server.sockets[0].getsockname()[1] | ||
| @contextmanager |
There was a problem hiding this comment.
wondering why we dropped the async here?
There was a problem hiding this comment.
We have a threaded server that can start rollout gateway even in a sync function call (both slime and verl invoke the gateway in sync caller), this integration test is to test whether threaded server can work in a sync caller.
| ) -> None: | ||
| self.tokenizer = tokenizer | ||
| self._stop_sequences: list = list(stop_sequences) if stop_sequences else [] | ||
| self.reasoning_parser: ReasoningParserFn = reasoning_parser or split_reasoning |
There was a problem hiding this comment.
Is it possible that we build a mapping from supported model family to parser functions?
There was a problem hiding this comment.
Do we want to implement the parser functions or import from inference engine? Our tenet is to have 0 dependency on vllm/sglang.
There was a problem hiding this comment.
I am ok to use some lightweight packages like renderers (https://github.com/PrimeIntellect-ai/renderers), or implement and maintain ourselves. Anyway, I think it may improve user experience if we can automatically match the correct parser functions according to the model family used by users?
Issue #, if available:
Description of changes:
HfTemplateRendererderendered model output with one hardcoded path: split reasoning on</think>, then match tool calls with a regex for the<tool_call><function=...>XML format. Any model whose tool-call format differs silently produced zero tool calls.For example, Qwen3-0.6B has parsing format:
while the parser can only handle
reasoning_parserraw_output -> (reasoning, body_text)</think>tool_parser(body_text, tools_schema) -> (text, tool_uses, ill_formed)<tool_call><function=...>regexThe stages run in sequence — reasoning first, tool calls on what remains — matching the order the inference engines use. Overriding one leaves the other on its default, so a model needing engine-grade tool parsing but emitting plain
</think>reasoning supplies onlytool_parser. The tool stage is skipped entirely when a request carries no tools schema.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.