[TRTLLM-17263][refactor] Split the chat handler into a typed core and lift the SSE generator into serve/_base by JunyiXu-nv · Pull Request #21515 · NVIDIA/TensorRT-LLM

JunyiXu-nv · 2026-11-03T04:59:18Z

Stacked on #21448 ([TRTLLM-17261][refactor] Add the trtllm-serve router skeleton and move /health, /version and /v1/models into ops.system), which is itself stacked on #21406 ([TRTLLM-17261][test] Add route, middleware-order and shim-identity snapshots and a CPU test seam for trtllm-serve): the first commits below belong to those PRs and disappear from this diff when they merge; this PR's own change is the last 6 commits.

@coderabbitai summary

Description

OpenAIServer.openai_chat returned an already serialized Response, so its in-process callers (the Anthropic Messages handler, the Anthropic batch runner, /health_generate) had to parse the JSON back or sniff the error body. This PR splits it in place: create_chat_completion() is the typed core, and openai_chat keeps its name, signature and route as a thin HTTP wrapper that renders the typed result exactly as before. The typed errors live in a new tensorrt_llm/serve/_base/errors.py, and the chat SSE generator, a closure inside openai_chat, is lifted unchanged into tensorrt_llm/serve/_base/sse.py. Later PRs remove the JSON round trips of the Anthropic paths and move the core into a chat serving class; nothing here changes what any client receives.

Part of the trtllm-serve refactor (epic TRTLLM-17056): typed return contract and Anthropic convergence (TRTLLM-17263). Builds on the serve router skeleton (#21448) and the CPU test seam and fixture factory (#21406).

Coordination. One test outside the handler's own tests changes: tests/unittest/llmapi/apps/test_openai_chat_multimodal_prompt.py stubbed _create_chat_response with a SimpleNamespace that only had model_dump(); the wrapper serializes a ChatCompletionResponse and streams any other result of the core, so the stub is now a real ChatCompletionResponse. That is a tightening of a test seam forced by the new dispatch, not a behavior change: the production _create_chat_response can only return a ChatCompletionResponse, and what the test checks (the rendered prompt, the decoded image and chat_template_applied reaching the engine) is unchanged. Open PRs on the same lines, which re-apply on top once this merges: #21392 (Choose the trtllm-serve post-processing path once per server) edits the lifted generator, now in tensorrt_llm/serve/_base/sse.py; #21445 (Add public runtime accessors for the remaining serving-frontend private accesses) and #21393 (Add public LLM health queries and a private-access lint for the serving frontends) record a private-access baseline entry keyed to the generator's old scope OpenAIServer.openai_chat.chat_stream_generator, which becomes tensorrt_llm/serve/_base/sse.py / chat_stream_generator if they land first; and the PRs that add lines inside openai_chat's body, which is now create_chat_completion — #21414 (warn when a requested output format cannot be enforced), #21111 (take Kimi-K3 chat prompt ids from its renderer), #21062 (fail a request whose generation degenerates into a token run), #21059 (enforce parallel_tool_calls=false by stopping after the first DSML tool call), #20977 (configure input processor workers), #20739 (add timing metrics for input processing and multimodal encoding), #20936 (Anthropic Messages: carry the engine request id in the message id), #19800 (bound multimodal request CPU memory in trtllm-serve) and #19461 (Expose image, video, and audio prompt-token counts in trtllm-serve responses) — turn any return self.create_error_response(...) they add inside that body into raise serving_error(...) with the same arguments; everything else in the body re-applies as it is. #21107 (Anthropic Messages: answer an over-length prompt with "prompt is too long", before any stream) edits the same region of test_anthropic_routes.py as the fixture retarget below (a textual conflict only).

Behavior

No behavior change.

The contract of OpenAIServer.create_chat_completion(request, raw_request):

non-streaming: returns the ChatCompletionResponse (after the optional base64 encoding of prompt_token_ids for context-only disaggregated requests), not serialized;
streaming: returns the SSE stream as strings (the post-processing worker produces them in another process), ending with data: [DONE];
errors: raises ServingError (the HTTP status apart from the OpenAI error body, whose code may be context_length_exceeded) or EngineFatalError (after the same three side effects as today: traceback logged, generator termination recorded, SIGINT); it never returns an error response and never returns None.

The wrapper openai_chat renders: JSONResponse(result.model_dump()) for the response, StreamingResponse(media_type="text/event-stream") for the stream, JSONResponse(e.body.model_dump(), status_code=e.status) for a ServingError (byte-identical to create_error_response for the same arguments), and None after an EngineFatalError. Its own ValueError branch keeps the 400 for a response that cannot be JSON-encoded (JSONResponse encodes in its constructor with allow_nan=False; that used to fail inside the handler's own try).

What an engine failure answers, per caller:

Caller	After this PR	Same as today
`/v1/chat/completions`	wrapper returns `None`: HTTP 200, body `null`	yes
`anthropic_messages`, `_run_batched_message`	unchanged code calling `openai_chat`: `None` → 500 / item `errored`	yes
`health_generate`	unchanged: `None.status_code` → `AttributeError` → 500	yes
`chat_harmony`	not touched (no `CppExecutorError` branch; 400)	yes
disaggregated frontend	not touched (`_handle_exception` returns `None`)	yes

Answering 5xx on an engine failure is a separate, later behavior change.

create_error_response now builds its body with error_body (the one copy of the rule that an over-length prompt gets code="context_length_exceeded" while the status stays 400), so a ServingError body and a rendered error body can never differ.

Moves

Pure-move check

## Move spec

```yaml
lifted:
- from: tensorrt_llm/serve/openai_server.py::OpenAIServer.openai_chat.chat_stream_generator
  to: tensorrt_llm/serve/_base/sse.py::chat_stream_generator
  free_variables:
    self: server
    raw_request: raw_request
added:
- symbol: tensorrt_llm/serve/openai_server.py::OpenAIServer.create_chat_completion
  reason: 'typed chat core: the body of openai_chat with errors raised as ServingError
    / EngineFatalError and the typed result returned'
modified:
- symbol: tensorrt_llm/serve/openai_server.py::OpenAIServer.openai_chat
  reason: thin HTTP wrapper over create_chat_completion; same route, same responses
- symbol: tensorrt_llm/serve/openai_server.py::OpenAIServer.create_error_response
  reason: body built by serve/_base/errors.py::error_body, shared with ServingError
rewrite_internal_imports: false

Pure-move check: base d3aacdd, head d8c56b5, tools 517fc468330a

Symbol	From	To	Result
`OpenAIServer.openai_chat.chat_stream_generator`	tensorrt_llm/serve/openai_server.py	tensorrt_llm/serve/_base/sse.py	lifted
`OpenAIServer.create_error_response`	tensorrt_llm/serve/openai_server.py	tensorrt_llm/serve/openai_server.py	modified (declared)
`OpenAIServer.openai_chat`	tensorrt_llm/serve/openai_server.py	tensorrt_llm/serve/openai_server.py	modified (declared)
`OpenAIServer.create_chat_completion`	—	tensorrt_llm/serve/openai_server.py	added

Added

OpenAIServer.create_chat_completion: typed chat core: the body of openai_chat with errors raised as ServingError / EngineFatalError and the typed result returned

Modified (declared)

OpenAIServer.create_error_response: body built by serve/_base/errors.py::error_body, shared with ServingError
--- base
+++ head
@@ -1,7 +1,4 @@
@staticmethod
def create_error_response(message: str, err_type: str='BadRequestError', status_code: HTTPStatus=HTTPStatus.BAD_REQUEST) -> Response:
- code: Union[int, str] = status_code.value
- if is_context_length_exceeded_message(message):
- ```
   code = CONTEXT_LENGTH_EXCEEDED_CODE
```
- error_response = ErrorResponse(message=message, type=err_type, code=code)
- error_response = error_body(message, err_type, status_code)
  return JSONResponse(content=error_response.model_dump(), status_code=status_code.value)

`OpenAIServer.openai_chat`: thin HTTP wrapper over create_chat_completion; same route, same responses
--- base
+++ head
@@ -1,154 +1,13 @@
async def openai_chat(self, request: ChatCompletionRequest, raw_request: Request) -> Response:

def get_role() -> str:
```
   if request.add_generation_prompt:
```
```
       role = 'assistant'
```
```
   else:
```

       role = request.messages[-1]['role']

```
   return role
```
try:

   render_resources = RenderResources.from_server(self)

   extension = render_resources.extension

   tool_dicts = prepare_chat_request(request, render_resources).tool_dicts

   required_mode = extension.required_tool_choice_mode()

   required_tool_grammar = request.tool_choice == 'required' and required_mode == 'grammar'

   conversation: List[ConversationMessage] = []

   vocab_size = getattr(self.tokenizer.tokenizer, 'vocab_size', None) or getattr(self.tokenizer, 'vocab_size', None)

   sampling_params = request.to_sampling_params(vocab_size=self._logit_bias_vocab_size(), gather_generation_logits=self.generator.args.gather_generation_logits, reasoning_parser=self.generator.args.reasoning_parser, backend=self.generator.args.backend)

```
   if required_tool_grammar:
```

       _validate_required_tool_grammar_request(request, sampling_params, guided_decoding_backend=self.generator.args.guided_decoding_backend, tool_parser=self.tool_parser, reasoning_parser=self.generator.args.reasoning_parser)

   add_thinking_budget_logits_processor(sampling_params, reasoning_parser=self.generator.args.reasoning_parser, tokenizer=self.tokenizer, chat_template_kwargs=request.chat_template_kwargs)

```
   forced_tool_name = None
```
```
   forced_tool_begin_prefix = None
```

   if isinstance(request.tool_choice, ChatCompletionNamedToolChoiceParam):

```
       if not request.tools:
```

           return self.create_error_response(message='tool_choice with a named function requires `tools` to be set.', err_type='BadRequestError', status_code=HTTPStatus.BAD_REQUEST)

       if request.prompt_token_ids is not None or request.prompt_token_ids_b64:

           return self.create_error_response(message='tool_choice with a named function is not supported alongside `prompt_token_ids`; pass `messages` instead.', err_type='BadRequestError', status_code=HTTPStatus.BAD_REQUEST)

       if sampling_params.guided_decoding is not None:

           return self.create_error_response(message='tool_choice with a named function cannot be combined with `response_format` or other guided-decoding settings.', err_type='BadRequestError', status_code=HTTPStatus.BAD_REQUEST)

       if self.generator.args.guided_decoding_backend is None and (not _parser_extracts_forced_tool_calls(self.tool_parser)):

           return self.create_error_response(message="tool_choice with a named function requires the server to be started with a guided decoding backend (set `guided_decoding_backend` to 'xgrammar' or 'llguidance').", err_type='BadRequestError', status_code=HTTPStatus.BAD_REQUEST)

       forced_tool_name = request.tool_choice.function.name

   reasoning_parser_name = self.generator.args.reasoning_parser

   dynamic_tools = extension.dynamic_tools(request.messages)

   dynamic_tool_params: List[ChatCompletionToolsParam] = []

```
   if dynamic_tools:
```
```
       try:
```

           dynamic_tool_params = [ChatCompletionToolsParam.model_validate(tool) for tool in dynamic_tools]

```
       except ValidationError as e:
```

           raise ValueError(f'Invalid message-level tool declaration: {e}') from e

   _configure_parser_special_token_decoding(sampling_params, reasoning_parser_name=reasoning_parser_name, tool_parser_name=self.tool_parser, has_tools=bool(request.tools) or bool(dynamic_tools))

   all_tools = (request.tools or []) + dynamic_tool_params

```
   if self.tool_parser and all_tools:
```

       if sampling_params.guided_decoding is None and (not required_tool_grammar):

           if forced_tool_name is not None and _parser_extracts_forced_tool_calls(self.tool_parser):

```
               pass
```

           elif forced_tool_name is not None:

```
               try:
```

                   forced_tool_begin_prefix, forced_guided = _build_forced_tool_call_decoding(request.tools, self.tool_parser, forced_tool_name)

```
               except ValueError as e:
```

                   return self.create_error_response(message=str(e), err_type='BadRequestError', status_code=HTTPStatus.BAD_REQUEST)

               sampling_params.guided_decoding = forced_guided

```
           else:
```

               strict_guided = _build_tool_strict_guided_decoding_params(all_tools, self.tool_parser)

               if strict_guided is not None:

                   sampling_params.guided_decoding = strict_guided

```
   elif forced_tool_name is not None:
```

       return self.create_error_response(message='tool_choice with a named function requires the server to be started with --tool_parser.', err_type='BadRequestError', status_code=HTTPStatus.BAD_REQUEST)

   postproc_args = ChatPostprocArgs.from_request(request)

   self._apply_spec_decode_stats_opt_in(postproc_args)

   postproc_args.num_prompt_tokens_offset = render_resources.prompt_tokens_excluded_from_usage(add_generation_prompt=request.add_generation_prompt, pre_tokenized=request.prompt_token_ids is not None or request.prompt_token_ids_b64 is not None, request_chat_template=request.chat_template)

```
   if dynamic_tool_params:
```

       postproc_args.tools = (postproc_args.tools or []) + dynamic_tool_params

   self._validate_internal_disagg_request(request, raw_request)

   disaggregated_params = to_llm_disaggregated_params(request.disaggregated_params)

   resolve_media = not (disaggregated_params is not None and disaggregated_params.request_type == 'generation_only' and (request.prompt_token_ids is not None or request.prompt_token_ids_b64))

```
   try:
```

       conversation, mm_coroutines, mm_placeholder_counts, mm_item_order = parse_chat_messages_coroutines(request.messages, self.model_config, self.multimodal_server_config, request_media_io_kwargs=request.media_io_kwargs, resolve_media=resolve_media)

```
   except ValidationError:
```

       raw_body = await raw_request.json()

       raw_messages = raw_body.get('messages', [])

       conversation, mm_coroutines, mm_placeholder_counts, mm_item_order = parse_chat_messages_coroutines(raw_messages, self.model_config, self.multimodal_server_config, request_media_io_kwargs=request.media_io_kwargs, resolve_media=resolve_media)

   if request.prompt_token_ids is None and request.prompt_token_ids_b64:

```
       import numpy as np
```

       request.prompt_token_ids = np.frombuffer(base64.b64decode(request.prompt_token_ids_b64), dtype=np.int32).tolist()

```
   rendered_prompt = None
```

   if request.prompt_token_ids is not None:

       prompt = request.prompt_token_ids

```
   else:
```

       render_task = render_conversation(render_resources, conversation=conversation, mm_placeholder_counts=mm_placeholder_counts, add_generation_prompt=request.add_generation_prompt, tools=tool_dicts, documents=request.documents, chat_template=request.chat_template, chat_template_kwargs=request.chat_template_kwargs, injected_chat_template_kwargs=request.injected_chat_template_kwargs, forced_prefix=forced_tool_begin_prefix)

       rendered, (mm_data, mm_embeddings) = await asyncio.gather(render_task, mm_coroutines)

```
       if rendered.text is None:
```
```
           prompt = rendered.token_ids
```
```
       else:
```
```
           prompt = rendered.text
```

           rendered_prompt = rendered.template_text

```
   prompt = prompt_inputs(prompt)
```

   if request.prompt_token_ids is not None:

       mm_data, mm_embeddings = await mm_coroutines

```
   if mm_data:
```

       self._attach_multimodal_data(prompt, request, mm_data, mm_item_order)

```
   if mm_embeddings:
```

       prompt['multi_modal_embeddings'] = mm_embeddings

```
   if mm_data and mm_embeddings:
```

       raise ValueError("Passing 'multi_modal_data' and 'multi_modal_embeddings' at the same time is not supported.")

```
   if request.mm_processor_kwargs:
```

       prompt['mm_processor_kwargs'] = request.mm_processor_kwargs

```
   thinking = None
```

   postproc_args.reasoning_parser = self.generator.args.reasoning_parser

   if postproc_args.reasoning_parser and ReasoningParserFactory.resolves_thinking_from_prompt(postproc_args.reasoning_parser):

       if rendered_prompt and request.add_generation_prompt:

           thinking = ReasoningParserFactory.resolve_prefilled_thinking(postproc_args.reasoning_parser, rendered_prompt)

       if thinking is None and request.disaggregated_params is not None:

           thinking = request.disaggregated_params.resolved_thinking

```
           if thinking is None:
```

               _warn_unresolvable_thinking_once(postproc_args.reasoning_parser)

```
       if thinking is not None:
```

           postproc_args.chat_template_kwargs = {**(request.chat_template_kwargs or {}), 'thinking': thinking, 'enable_thinking': thinking}

```
   if required_tool_grammar:
```
```
       if thinking is None:
```

           raise ValueError('Cannot resolve required-tool reasoning state from the rendered prompt.')

       parser = ToolParserFactory.create_tool_parser(self.tool_parser)

       build_required = functools.partial(parser.build_required_structural_tag, request.tools, thinking=thinking)

       required_format = await asyncio.get_running_loop().run_in_executor(self._input_proc_executor, build_required)

```
       if required_format is None:
```

           raise ValueError('The configured tool parser does not support required tools.')

       sampling_params.guided_decoding = GuidedDecodingParams(structural_tag=required_format.model_dump_json(by_alias=True))

       postproc_args.tool_parser_grammar_constrained = True

   postproc_args.tool_parser = self.tool_parser

   postproc_args.tool_call_id_type = self.tool_call_id_type

   if conversation and conversation[-1].get('content') and (conversation[-1].get('role') == get_role()):

       postproc_args.last_message_content = conversation[-1]['content']

   postproc_params = PostprocParams(post_processor=chat_stream_post_processor if request.stream else chat_response_post_processor, postproc_args=postproc_args)

   trace_headers = None if raw_request is None else tracing.extract_trace_headers(raw_request.headers)

   resolve_request_conversation_id(request, None if raw_request is None else raw_request.headers)

   conversation_params = to_llm_conversation_params(request.conversation_params)

   scheduling_params = self._get_scheduling_params(request, raw_request)

```
   generate_inputs = prompt
```

   preprocess_fn = getattr(self.generator, 'preprocess', None)

```
   if preprocess_fn is not None:
```
```
       loop = asyncio.get_event_loop()
```

       generate_inputs = await loop.run_in_executor(self._input_proc_executor, functools.partial(preprocess_fn, prompt, sampling_params, disaggregated_params))

   promise = self.generator.generate_async(inputs=generate_inputs, sampling_params=sampling_params, _postproc_params=postproc_params if self.postproc_worker_enabled else None, streaming=request.stream, lora_request=request.lora_request, disaggregated_params=disaggregated_params, conversation_params=conversation_params, cache_salt=request.cache_salt, kv_hints=request.kv_hints, trace_headers=trace_headers, scheduling_params=scheduling_params, priority=request.priority if request.priority is not None else DEFAULT_REQUEST_PRIORITY)

   asyncio.create_task(self.await_disconnected(raw_request, promise))

   if not self.postproc_worker_enabled:

       postproc_args.tokenizer = self.tokenizer

       postproc_args.num_prompt_tokens = len(promise.prompt_token_ids)

```
   if request.stream:
```

       response_generator = chat_stream_generator(promise, postproc_params)

       return StreamingResponse(content=response_generator, media_type='text/event-stream')

```
   else:
```

       response = await self._create_chat_response(promise, postproc_params, raw_request, disaggregated_params)

       if request.disaggregated_params is not None and request.disaggregated_params.return_prompt_token_ids_b64 and (response.prompt_token_ids is not None):

```
           import numpy as np
```

           response.prompt_token_ids_b64 = base64.b64encode(np.asarray(response.prompt_token_ids, dtype=np.int32).tobytes()).decode('ascii')

           response.prompt_token_ids = None

       return JSONResponse(content=response.model_dump())

except CppExecutorError:

   logger.error(traceback.format_exc())

   _record_generator_termination(self.generator)

```
   signal.raise_signal(signal.SIGINT)
```

   result = await self.create_chat_completion(request, raw_request)

   if isinstance(result, ChatCompletionResponse):

       return JSONResponse(content=result.model_dump())

   return StreamingResponse(content=result, media_type='text/event-stream')

except ServingError as e:

   return JSONResponse(content=e.body.model_dump(), status_code=e.status)

except EngineFatalError:
```
   return None
```
except ValueError as e:
return self.create_error_response(str(e))
except Exception as e:

0 definitions moved verbatim; 0 differences; 1 lifted; 1 added; 2 modified (declared); shim exports 0 names


</details>

The generator moved as it is: with its `except Exception:` clause (from #20867, "Anthropic: read the chat stream to its end") and its two `_postprocess_result` reads, which go when the post-processing path is chosen once per server (#21392). Its two free variables (`self`, `raw_request`) became the keyword-only parameters `server` and `raw_request`; the one call site passes `server=self, raw_request=raw_request`.

Old import paths: none change (the generator was a closure, never a module attribute), so there is no shim and no release-notes list.

`create_chat_completion` is the old `openai_chat` body with exactly these edits (word diff of the two bodies; the removed closure is the lifted generator):

<details><summary>Word diff: old openai_chat body → create_chat_completion</summary>

```diff
--- old body of OpenAIServer.openai_chat (before this PR)
+++ OpenAIServer.create_chat_completion (this PR)
@@ -1,5 +1,17 @@
    async def [-openai_chat(self,-]{+create_chat_completion(+}
{+        self,+} request: ChatCompletionRequest, raw_request: [-Request)-]{+Request+}
{+    )+} -> [-Response:-]{+Union[ChatCompletionResponse, AsyncIterator[str]]:+}
{+        """Run one chat completion and return a typed result.+}

{+        Returns the ChatCompletionResponse for a non-streaming request, and+}
{+        for a streaming one the SSE stream as strings, ending with+}
{+        "data: [DONE]".+}

{+        Raises:+}
{+            ServingError: The request is rejected; the HTTP status travels+}
{+                apart from the body.+}
{+            EngineFatalError: The engine failed; SIGINT has been sent.+}
{+        """+}

        def get_role() -> str:
            if request.add_generation_prompt:
@@ -8,38 +20,6 @@
                role = request.messages[-1]["role"]
            return role

[-        async def chat_stream_generator(-]
[-                promise: RequestOutput,-]
[-                postproc_params: PostprocParams) -> AsyncGenerator[str, None]:-]
[-            try:-]
[-                if not self.postproc_worker_enabled:-]
[-                    post_processor, args = postproc_params.post_processor, postproc_params.postproc_args-]
[-                first_response = await anext(promise)-]
[-                raw_request.state.server_first_token_time = self._adjusted_steady_clock.now(-]
[-                )-]
[-                pp_results = first_response.outputs[-]
[-                    0]._postprocess_result if self.postproc_worker_enabled else post_processor(-]
[-                        first_response, args)-]
[-                for pp_res in pp_results:-]
[-                    yield pp_res-]
[-                # Making sure we can handle the situation where there is only one response-]
[-                res = first_response-]
[-                async for res in promise:-]
[-                    pp_results = res.outputs[-]
[-                        0]._postprocess_result if self.postproc_worker_enabled else post_processor(-]
[-                            res, args)-]
[-                    for pp_res in pp_results:-]
[-                        yield pp_res-]
[-                yield "data: [DONE]\n\n"-]
[-                await self._extract_metrics(res, raw_request)-]
[-                nvtx_mark("generation ends")-]
[-            except Exception:-]
[-                # Only failures are logged: a consumer that stops reading closes-]
[-                # the stream with GeneratorExit, and a client that disconnects-]
[-                # cancels it, and neither is an error of this stream.-]
[-                logger.error(traceback.format_exc())-]
[-                raise-]

        try:
            # Extension preprocessing, the required-tool check and tool dumping
            # are the first stage of the shared prompt-preparation pipeline.
@@ -95,7 +75,7 @@
            if isinstance(request.tool_choice,
                          ChatCompletionNamedToolChoiceParam):
                if not request.tools:
                    [-return self.create_error_response(-]{+raise serving_error(+}
                        message=("tool_choice with a named function requires "
                                 "`tools` to be set."),
                        err_type="BadRequestError",
@@ -113,7 +93,7 @@
                    # testing ``prompt_token_ids`` alone lets a base64 request
                    # through to the pre-tokenized branch, where the begin
                    # prefix is never appended.
                    [-return self.create_error_response(-]{+raise serving_error(+}
                        message=("tool_choice with a named function is not "
                                 "supported alongside `prompt_token_ids`; pass "
                                 "`messages` instead."),
@@ -128,7 +108,7 @@
                    # instead. Honouring only one of them silently would either
                    # ignore an explicit instruction or emit a tool call whose
                    # arguments are unconstrained text, so reject instead.
                    [-return self.create_error_response(-]{+raise serving_error(+}
                        message=("tool_choice with a named function cannot be "
                                 "combined with `response_format` or other "
                                 "guided-decoding settings."),
@@ -151,14 +131,13 @@
                    # ends up reported as ``function.arguments``. Fail loudly
                    # instead of returning a tool call whose arguments are
                    # unconstrained text.
                    [-return self.create_error_response(-]
[-                        message=(-]{+raise serving_error(message=(+}
                        "tool_choice with a named function requires the "
                        "server to be started with a guided decoding "
                        "backend (set `guided_decoding_backend` to "
                        "'xgrammar' or 'llguidance')."),
                                        err_type="BadRequestError",
                                        status_code=HTTPStatus.BAD_REQUEST)
                forced_tool_name = request.tool_choice.function.name

            reasoning_parser_name = self.generator.args.reasoning_parser
@@ -211,7 +190,7 @@
                                    request.tools, self.tool_parser,
                                    forced_tool_name))
                        except ValueError as e:
                            [-return self.create_error_response(-]{+raise serving_error(+}
                                message=str(e),
                                err_type="BadRequestError",
                                status_code=HTTPStatus.BAD_REQUEST)
@@ -226,7 +205,7 @@
                            sampling_params.guided_decoding = strict_guided
            elif forced_tool_name is not None:
                # tools were provided but the server has no tool_parser configured.
                [-return self.create_error_response(-]{+raise serving_error(+}
                    message=("tool_choice with a named function requires the "
                             "server to be started with --tool_parser."),
                    err_type="BadRequestError",
@@ -442,10 +421,10 @@
                postproc_args.num_prompt_tokens = len(promise.prompt_token_ids)

            if request.stream:
[-response_generator = chat_stream_generator(-]
[-                    promise, postproc_params)-]                return [-StreamingResponse(content=response_generator,-]
[-                                         media_type="text/event-stream")-]{+chat_stream_generator(promise,+}
{+                                             postproc_params,+}
{+                                             server=self,+}
{+                                             raw_request=raw_request)+}
            else:
                response = await self._create_chat_response(
                    promise, postproc_params, raw_request, disaggregated_params)
@@ -461,14 +440,17 @@
                        np.asarray(response.prompt_token_ids,
                                   dtype=np.int32).tobytes()).decode("ascii")
                    response.prompt_token_ids = None
                return [-JSONResponse(content=response.model_dump())-]{+response+}
        except [-CppExecutorError:-]{+ServingError:+}
{+            raise+}
{+        except CppExecutorError as e:+}
            logger.error(traceback.format_exc())
            # If internal executor error is raised, shutdown the server
            _record_generator_termination(self.generator)
            signal.raise_signal(signal.SIGINT)
            {+raise EngineFatalError() from e+}
        except ValueError as e:
            [-return self.create_error_response(str(e))-]{+raise serving_error(str(e)) from e+}
        except Exception as e:
            logger.error(traceback.format_exc())
            [-return self.create_error_response(str(e))-]{+raise serving_error(str(e)) from e+}

History: git blame -C -C tensorrt_llm/serve/_base/sse.py attributes the generator's lines to their original commits.

Removed from OpenAIServer

None. The chat SSE generator was a closure inside openai_chat, not a method; every method of OpenAIServer is still there, and create_chat_completion is new.

CI bookkeeping

Multi-GPU table: none (the existing tensorrt_llm/serve/_base/ prefix covers both new files; tensorrt_llm/serve/openai_server.py is already listed, so the multi-GPU stages run and need the ci: full pre-merge approved label).
CODEOWNERS: none.
legacy-files.txt: tensorrt_llm/serve/_base/sse.py added (legacy formatting group; its code comes from openai_server.py); tensorrt_llm/serve/_base/errors.py is new code and stays in the ruff group; derived configs regenerated with scripts/legacy_utils.py gen-configs; ruff-legacy-baseline.json: one new entry tensorrt_llm/serve/_base/sse.py: {F821: 1} (the anext builtin ruff does not know, moved with the generator; the openai_server.py entry stays at 3 — never a full lint-update-violations in a move PR).
Type checking: tensorrt_llm/serve/_base/errors.py added to [tool.mypy] files; sse.py is moved legacy code and stays out until its package graduates.
Test lists: none; the three new test files under tests/unittest/serve/ are collected by the unittest/serve entry of tests/integration/test_lists/test-db/l0_cpu.yml (line 337) and carry pytest.mark.cpu_only.
Patch sites: the server_logger fixture of the four chat-stream tests in tests/unittest/llmapi/apps/test_anthropic_routes.py patched openai_server.logger to check what the generator logs; it now patches the logger of tensorrt_llm.serve._base.sse, where the generator lives (verified by making the generator's log call raise temporarily: the failure test then fails). The remaining patch("tensorrt_llm.serve.openai_server.logger") in test_responses_store_replay.py targets Responses code, which did not move.
Test seams: tests/unittest/llmapi/apps/test_openai_chat_multimodal_prompt.py stubbed _create_chat_response with a SimpleNamespace that only had model_dump(); the wrapper serializes a ChatCompletionResponse and streams any other result of the core, so the stub is now a real ChatCompletionResponse (what the test checks, the rendered prompt and decoded image reaching the engine, is unchanged). Every other _create_chat_response stub in tests/ already returns a ChatCompletionResponse.

Test Coverage

CPU, run from the worktree with the compiled artifacts linked in (tensorrt_llm.__file__ resolved to the worktree):

pytest -m cpu_only tests/unittest/serve/test_openai_server_chat_core.py — the three answer-pinning tests (test_http_wrapper_keeps_every_answer, test_unencodable_response_is_still_a_400, test_health_generate_still_fails_with_500_on_engine_failure) run on the unsplit handler before the split: 3 passed; after the split, with the six typed-core tests: 9 passed. The unencodable-response case also pins that nothing is logged at ERROR, so it fails if the wrapper's own ValueError branch is removed and the generic handler (which logs a traceback) renders the 400 instead.
pytest -m cpu_only tests/unittest/serve/_base/test_errors.py tests/unittest/serve/_base/test_sse.py — 7 passed.
pytest -m cpu_only tests/unittest/serve tests/unittest/llmapi/apps/test_serving_extensions.py tests/unittest/llmapi/apps/test_openai_chat_disagg_multimodal.py tests/unittest/llmapi/apps/test_anthropic_routes.py tests/unittest/llmapi/apps/test_render_callers.py tests/unittest/disaggregated -q — 1759 passed, 1 skipped, 2 xfailed (includes the route snapshot, the middleware-order and layering contracts and the object.__new__ ratchet: no site added or removed).
PYTHONPATH=<worktree>/tests python3 -m test_common.serve.cpu_tests (every serve CPU test, one pytest process per test-db entry) — 66 test-db entries, 4210 passed. Two failures on the first full run, neither a defect of the change: test_openai_chat_multimodal_prompt.py::test_rendered_image_request_reaches_the_engine_marked_templated (its _create_chat_response stub was a SimpleNamespace; fixed here, 4 passed on re-run) and test_render_standalone.py::TestCpuOnlyStartup::test_the_launcher_loads_a_cuda_stub_only_when_the_driver_is_missing, which asserts shutil.which("trtllm-render") and only fails when the environment's bin/ is not on PATH (1 passed when re-run with it on PATH).
LLM_MODELS_ROOT=... pytest -rs tests/unittest/llmapi/apps/test_harmony_parsing.py tests/unittest/llmapi/apps/test_harmony_channel_validation.py — 79 passed, 0 skipped; no "not importable" skip and no channel-validation guard skip.
Route snapshot, the new tests and the Anthropic route tests on both ends of the FastAPI range: fastapi 0.136.3: 108 passed; fastapi 0.139.2: 108 passed (route snapshot, the three new test files and the Anthropic route tests).
python -c "import tensorrt_llm.serve._base.sse" — imports (every name the generator uses is imported in the new module).
bash scripts/run_mypy.sh — "Compiled bindings detected — running full mypy type check. Success: no issues found in 20 source files" (errors.py included).
python scripts/legacy_utils.py lint-precommit tensorrt_llm/serve/_base/sse.py tensorrt_llm/serve/openai_server.py — passes (its tip about one resolved F821 in openai_server.py is expected); python scripts/legacy_utils.py check-configs — "All generated configs are up to date."
pre-commit run --files <every changed file> — clean on the second pass (isort and yapf converge).
Property check with Hypothesis (local, not committed): seven properties over generated messages, error types and HTTP statuses — the typed body equals what create_error_response renders; the context_length_exceeded code rule; ServingError carries status, body and message and is not a ValueError; the wrapper renders a ServingError byte-identically to create_error_response, a typed response as 200 JSON of model_dump(), a stream as text/event-stream with the same frames, an engine failure as None and a ValueError as 400 — all pass (60–200 examples each), and a mutation gate of 8 planted bugs (in errors.py and in the wrapper) is detected 8/8.

Left to CI (GPU): test_e2e.py::test_openai_misc_example[pytorch] (/health_generate 200) and test_e2e.py::test_openai_chat_example[pytorch] (l0_a10.yml), test_e2e.py::test_anthropic_messages (l0_h100.yml).

New tests: tests/unittest/serve/test_openai_server_chat_core.py (9), tests/unittest/serve/_base/test_errors.py (4), tests/unittest/serve/_base/test_sse.py (3); all under the unittest/serve entry of l0_cpu.yml.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

…apshots and a CPU test seam for trtllm-serve Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

… baselines at the merge base The route reference is generated from the live apps; the object.__new__ baseline and the gRPC import allowance record what exists on main today and only shrink. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…n the snapshot matrix An attached frontend of a multi-frontend server registers ten launcher forwarding routes before the local table; seven share a method and path with local handlers and three use a path parameter that matches every batch id, so registration order decides which handler serves them. The new configuration records all of it, a live invariant pins the forwarding-before-local order independently of the reference, and the route-class failure message now says why the msgpack route class matters. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

- AGENTS.md: the CPU test command sets LLM_MODELS_ROOT, which the runner requires, and the stage-mapping note says the mapper matches list-entry text, so a file listed through its directory maps by the directory. - check_contracts matches each from-import as <module>.<name>, so 'from pkg import sub' and 'from .. import sub' count as imports of the submodule; new test for both forms and for the allowance entry. - The VisualGen /v1/models test asserts the 200 status. - The AGENTS.md command check also recognizes commands that start with LLM_MODELS_ROOT=. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…on the current main The disaggregated frontend now registers its routes with KvHintsRoute (NVIDIA#20269), which strips kv_hints from request bodies unless the server trusts its ingress. Regenerate the route reference: the six disaggregated configurations record KvHintsRoute where they recorded APIRoute; their route sets and order are unchanged. The route-class invariant, which a regenerated reference cannot hide, now requires KvHintsRoute on every API route of the disaggregated frontend and APIRoute on the coordinator app. Dropping the KV-hint policy from the frontend fails it even after a regeneration, and both apps still reject the aggregated server's msgpack route class. The object.__new__ baseline records the sites on main at the merge base: test_kv_hints.py builds one server that way (NVIDIA#20269), and test_harmony_parsing.py no longer builds any (NVIDIA#21176). The harmony parsing tests are now registered as a whole file (NVIDIA#21411), so the node-id entry that the CPU-selection test used as its example is gone. The node-id check now runs select_entries on a one-entry list of its own, so it no longer depends on which node-id entries l0_cpu.yml holds. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…ail on silent regressions The route snapshot now requires every configuration other than the three invalid ones to build, so a configuration that starts failing at construction cannot pass through a regenerated reference. The object.__new__ ratchet resolves 'from ... import <server class> as <name>' aliases and records each baseline site as its enclosing function or class plus the server class, so swapping one site for another in the same file no longer passes. The baseline keeps the same 52 sites in the same 26 files. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…lm/serve/_base/msgspec_route.py Generated move: _msgpack_decoder, _MsgspecRequest and _MsgspecRoute leave tensorrt_llm/serve/openai_server.py byte for byte; the module imports _MsgspecRoute back for its app router and keeps no re-export (the three names are private). The two disaggregated test importers now import the new module. OpenAIServer.health, .version and .get_model are removed here and replaced by the system router added in the following commits. tensorrt_llm/serve/_base/msgspec_route.py joins legacy-files.txt (its code comes from a legacy-formatted file), and the generated lint configs are regenerated with scripts/legacy_utils.py gen-configs. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…ve the system endpoints through it New structure: tensorrt_llm/serve/_base/ (shared serving infrastructure) and tensorrt_llm/serve/api/ (endpoint packages, one router plus one serving class each; the package __init__ imports nothing so routers load lazily, by role). serve/_base/routing.py provides make_router(), which always uses the msgpack-decoding route class, and attach_routers(), which imports and attaches only the routers whose RouterSpec matches the server role. The first endpoint package, serve/api/ops/system/, serves /health, /version and /v1/models for every role. OpenAIServer builds a SystemServing with the injected _health_probe (today's probe body, SIGINT side effect included) and attaches the routers before the role dispatch; the 12 add_api_route lines for the three paths leave the four role tables. Same routes, methods, route classes, bodies and conflict-relevant order: the recorded route reference is unchanged. The layering tests gain the new packages' contracts, the route-snapshot failure message points new endpoints at the router pattern, and the serve agent docs describe where a new endpoint goes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…w serve packages tensorrt_llm/serve/_base/ and tensorrt_llm/serve/api/ops/ join the multi-GPU trigger list: the msgpack route class decodes the disaggregated frontend's worker requests, and /health is the disaggregated readiness probe, so changes under these directories keep running the multi-GPU stages that exercise that path. The three new modules join the strict mypy file set, and the type-check pre-commit hook now also triggers on changes under tensorrt_llm/serve/ and tensorrt_llm/commands/serve_cli/, so edits to typed serving modules get local type checking; the hook still checks only the declared file set. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

… tests that build routes by hand Two multi-frontend tests build an OpenAIServer without its constructor and call register_routes() themselves. Since /health, /version and /v1/models are attached through the system router by the constructor, those tests no longer saw the three routes: one asserted that /version stays local on an attached frontend (404), the other reported the three per-frontend-safe entries as removed. Both helpers now set the serving object and attach the routers before the hand registration, in the constructor's order, and the route enumeration walks included routers, which FastAPI 0.139 keeps as objects without a path in app.routes. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…n the OpenEngine metrics test The HTTP-enablement case builds a real OpenAIServer through its constructor and looked for /prometheus/metrics by reading .path off every entry of app.routes. Since the constructor attaches the system router, FastAPI 0.139 keeps an included-router object there that has no .path, and the child process failed with AttributeError. The test now matches a GET /prometheus/metrics request against the routes with the public route-matching API, which works on both ends of the pinned FastAPI range. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…d name The check that endpoint packages under tensorrt_llm/serve/api do not import each other read the package off the from-module alone, so `from tensorrt_llm.serve import api` inside an endpoint package passed and `from tensorrt_llm.serve.api.ops import system` inside ops/system failed. It now reads the package off <module>.<name> per imported name, as the layering contracts do, and a synthetic-edge test pins both forms. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…efore splitting it Three tests run OpenAIServer.openai_chat through a server built with the real constructor and the fake generator, and pin what it answers today: a request the handler rejects keeps its 400 body byte for byte, an engine failure keeps HTTP 200 with body null (and /health_generate keeps turning that into a 500), a response that cannot be JSON-encoded keeps its 400, and a stream keeps its frames and media type. They pass on the unsplit handler and must still pass once it is split into a typed core and an HTTP wrapper. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…serve/_base/sse.py Generated move: the chat_stream_generator closure of OpenAIServer.openai_chat is lifted to module level in tensorrt_llm/serve/_base/sse.py unchanged, apart from its two free variables, which become the keyword-only parameters `server` (was `self`) and `raw_request`. openai_chat imports it back and passes server=self, raw_request=raw_request at its one call. The generator keeps its `except Exception:` clause and its two `_postprocess_result` reads. tensorrt_llm/serve/_base/sse.py joins legacy-files.txt (its code comes from a legacy-formatted file) and the generated lint configs are regenerated with scripts/legacy_utils.py gen-configs; ruff-legacy-baseline.json gains the module's one recorded F821 (the `anext` builtin ruff does not know), which openai_server.py keeps counting until its baseline is regenerated. The Anthropic stream tests patched the logger of openai_server to check what the generator logs; they now patch the logger of serve/_base/sse.py, where the generator lives. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…ompletion and an HTTP wrapper OpenAIServer.create_chat_completion is the former body of openai_chat with a typed contract: it returns the ChatCompletionResponse for a non-streaming request and the SSE stream as strings for a streaming one, and it raises ServingError (the HTTP status apart from the OpenAI error body) where it used to return an error response, or EngineFatalError after the three side effects of an engine failure. Nothing else in the body changes: the six early rejections raise where they returned, the ValueError and the generic handler raise what they rendered, and the ServingError handler comes first so an early rejection is neither logged nor re-wrapped. openai_chat keeps its name, signature and route and becomes the wrapper that renders the typed result: JSONResponse for the response, a streaming response for the stream, the error body with its status for ServingError, and HTTP 200 with body null after an engine failure, as before. Its own ValueError branch keeps the 400 for a response that cannot be JSON-encoded, which used to fail inside the core. Every caller (the route, /health_generate, the Anthropic messages and batch paths) is unchanged. tensorrt_llm/serve/_base/errors.py holds error_body, ServingError, serving_error and EngineFatalError; create_error_response builds its body with error_body, so a ServingError body and a rendered error can never differ (the over-length-prompt code lives in one place). Tests: the typed core and its errors through a server built with the real constructor; the error builders against create_error_response; the lifted SSE generator with fakes. The three answer-pinning tests added before the split pass unchanged. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

The new error module is written from scratch, so it joins the mypy file list in pyproject.toml; scripts/run_mypy.sh passes with it (the strict global settings apply). tensorrt_llm/serve/_base/sse.py stays out: it is moved legacy code and joins type checking with its package later. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…se in the multimodal prompt test The chat handler now serializes a ChatCompletionResponse and streams any other result of its core, so a stand-in for _create_chat_response that only has model_dump() is no longer rendered as JSON. The test stubs a real ChatCompletionResponse instead; what it checks (the rendered prompt, the decoded image and chat_template_applied reaching the engine) is unchanged. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

…d 400 without a traceback The HTTP wrapper answers a response that cannot be JSON-encoded from its own ValueError branch, which logs nothing; the generic handler would render the same 400 but log a traceback. The answer-pinning test now also asserts that no ERROR record is emitted, so dropping that branch fails the test. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

github-actions Bot assigned JunyiXu-nv Nov 3, 2026

This was referenced Nov 3, 2026

[TRTLLM-17263][refactor] Converge the three Anthropic orchestrations into one serving class #21551

Draft

[TRTLLM-17269][refactor] Add per-dialect error renderers in serve/exception_handling #21566

Draft

JunyiXu-nv added 18 commits November 4, 2026 05:26

[TRTLLM-17261][test] Add route, middleware-order and shim-identity sn…

b291cca

…apshots and a CPU test seam for trtllm-serve Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv force-pushed the dev-junyix-feat-serve-typed-chat-core branch from 22fda4f to d8c56b5 Compare November 4, 2026 12:50

JunyiXu-nv mentioned this pull request Nov 5, 2026

[TRTLLM-17268][feat] Serve gpt-oss chat completions through the shared chat handler #21735

Draft

1 task done

Navigation Menu

Repository navigation

[TRTLLM-17263][refactor] Split the chat handler into a typed core and lift the SSE generator into serve/_base - #21515

[TRTLLM-17263][refactor] Split the chat handler into a typed core and lift the SSE generator into serve/_base#21515
JunyiXu-nv wants to merge 18 commits into
NVIDIA:mainfrom
JunyiXu-nv:dev-junyix-feat-serve-typed-chat-core

JunyiXu-nv commented Nov 3, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

JunyiXu-nv commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Behavior

Moves

Added

Modified (declared)

OpenAIServer.openai_chat: thin HTTP wrapper over create_chat_completion; same route, same responses --- base +++ head @@ -1,154 +1,13 @@ async def openai_chat(self, request: ChatCompletionRequest, raw_request: Request) -> Response:

Removed from OpenAIServer

CI bookkeeping

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

JunyiXu-nv commented Nov 3, 2026 •

edited

Loading

`OpenAIServer.openai_chat`: thin HTTP wrapper over create_chat_completion; same route, same responses
--- base
+++ head
@@ -1,154 +1,13 @@
async def openai_chat(self, request: ChatCompletionRequest, raw_request: Request) -> Response: