Skip to content

fix(trtllm): forward served model name to native workers / 向原生 worker 传递服务模型名 - #26

Draft
adibarra wants to merge 2 commits into
mainfrom
codex/trt-native-served-model
Draft

adibarra wants to merge 2 commits into
mainfrom
codex/trt-native-served-model

Conversation

@adibarra

@adibarra adibarra commented Sep 22, 2026

Copy link
Copy Markdown

An explicit engine.served_model_name was used by benchmark/eval clients but omitted from native trtllm-serve worker commands. With locally mounted weights, clients could request org/model while the server exposed /model.

Forward the existing field as --served_model_name for native workers. Unset values retain the current command. This reuses the existing backend and config field; no new schema or launcher is introduced. Update the existing example and reference text.

SGLang and vLLM already forward an explicitly configured served model name to their servers, and SRT uses the resolved name for benchmark/eval clients. This change brings native TRT-LLM into line with that behavior: configuring a name once makes it reach both sides. The fix belongs in the existing TRT backend because translating that setting into the server command is the launcher's responsibility.

This aligns explicit-name handling only. Framework-specific YAML field locations remain unchanged. When the name is omitted, native TRT-LLM still keeps its server default; this PR does not unify omitted-name defaults across backends or change model loading, precision, or performance settings.

Validation:

  • The explicit-name command-generation case fails on clean main and passes with this fix; the unset case passes on both. The tests exercise the existing command builder and assert only the model-name behavior. All 96 focused TRT, model-staging, and sidecar tests pass.
  • Ruff and generated schema-documentation checks pass.
  • Linux CI before the test cleanup passed all tests, lint, type checking, examples, golden equality, and mock/server integration. CI for the latest test/comment cleanup is running. The two-line production fix is unchanged. No GPU run was performed for this command-generation fix.
  • Verified the option against TensorRT-LLM v1.3.0rc14 source, commit 93cb6518b6d6dbd6095748189e626db731f44545.

Related: InferenceX #3352. Its current recipes already achieve the intended behavior by repeating the name in engine.served_model_name and roles.*.extra_args: [--served_model_name, ...]. The migration therefore does not depend on merging this PR. Once a dependency revision containing this fix is adopted, those duplicate CLI arguments can be removed while preserving the intended server/client names. Keep this PR draft and unmerged pending review.

中文

显式设置的 engine.served_model_name 已供 benchmark/eval 客户端使用,但原生 trtllm-serve worker 命令没有传入该值。本地挂载权重时,客户端可能请求 org/model,而服务器实际公开的是 /model

将现有字段作为 --served_model_name 传给原生 worker;未设置时保留原命令。复用现有 backend 和配置字段,不新增 schema 或启动器,并更新现有示例和参考说明。

SGLang 和 vLLM 已将显式配置的服务模型名传给服务器,SRT 也使用解析后的名称供 benchmark/eval 客户端请求。此修改让原生 TRT-LLM 具备一致的行为:名称只需配置一次,就会同时用于服务器和客户端。将该设置转换为服务器命令属于启动器的职责,因此修复放在现有 TRT backend 中。

此次仅统一显式名称的传递行为。各框架的 YAML 字段位置保持不变。未设置名称时,原生 TRT-LLM 仍保留服务器默认值;本 PR 不统一各 backend 在名称缺省时的行为,也不改变模型加载、精度或性能设置。

验证:

  • 显式名称的命令生成测试在 clean main 上复现缺少参数的问题,修复后通过;未设置名称的测试在两者上均通过。测试调用现有命令构建逻辑,只断言模型名相关行为。96 项 TRT、模型暂存和 sidecar 定向测试全部通过。
  • Ruff 和生成的 schema 文档检查通过。
  • 测试精简前的 Linux CI 已通过完整测试、lint、类型检查、示例、golden equality 和 mock/server 集成。最新测试及注释精简提交的 CI 正在运行。两行运行时代码修复保持不变。此次命令生成修复未运行 GPU 测试。
  • 已对照 TensorRT-LLM v1.3.0rc14 源码确认该参数;commit 为 93cb6518b6d6dbd6095748189e626db731f44545

关联 InferenceX #3352。当前配方通过在 engine.served_model_nameroles.*.extra_args: [--served_model_name, ...] 中重复指定名称,已实现预期行为,因此迁移不依赖本 PR 合并。采用包含此修复的依赖版本后,即可移除重复的 CLI 参数,并保留预期的服务器和客户端模型名。保持草稿,审核前不合并。

将显式配置的服务模型名传给原生 TRT-LLM worker,确保服务器与 benchmark/eval 客户端使用一致的 API 模型名。

Signed-off-by: adibarra <93070681+adibarra@users.noreply.github.com>
按现有测试风格只验证模型名参数的传递和省略,去掉无关命令细节及重复路径断言,并简化示例注释。

Signed-off-by: adibarra <93070681+adibarra@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant