Conversation
将显式配置的服务模型名传给原生 TRT-LLM worker,确保服务器与 benchmark/eval 客户端使用一致的 API 模型名。 Signed-off-by: adibarra <93070681+adibarra@users.noreply.github.com>
按现有测试风格只验证模型名参数的传递和省略,去掉无关命令细节及重复路径断言,并简化示例注释。 Signed-off-by: adibarra <93070681+adibarra@users.noreply.github.com>
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An explicit
engine.served_model_namewas used by benchmark/eval clients but omitted from nativetrtllm-serveworker commands. With locally mounted weights, clients could requestorg/modelwhile the server exposed/model.Forward the existing field as
--served_model_namefor native workers. Unset values retain the current command. This reuses the existing backend and config field; no new schema or launcher is introduced. Update the existing example and reference text.SGLang and vLLM already forward an explicitly configured served model name to their servers, and SRT uses the resolved name for benchmark/eval clients. This change brings native TRT-LLM into line with that behavior: configuring a name once makes it reach both sides. The fix belongs in the existing TRT backend because translating that setting into the server command is the launcher's responsibility.
This aligns explicit-name handling only. Framework-specific YAML field locations remain unchanged. When the name is omitted, native TRT-LLM still keeps its server default; this PR does not unify omitted-name defaults across backends or change model loading, precision, or performance settings.
Validation:
v1.3.0rc14source, commit93cb6518b6d6dbd6095748189e626db731f44545.Related: InferenceX #3352. Its current recipes already achieve the intended behavior by repeating the name in
engine.served_model_nameandroles.*.extra_args: [--served_model_name, ...]. The migration therefore does not depend on merging this PR. Once a dependency revision containing this fix is adopted, those duplicate CLI arguments can be removed while preserving the intended server/client names. Keep this PR draft and unmerged pending review.中文
显式设置的
engine.served_model_name已供 benchmark/eval 客户端使用,但原生trtllm-serveworker 命令没有传入该值。本地挂载权重时,客户端可能请求org/model,而服务器实际公开的是/model。将现有字段作为
--served_model_name传给原生 worker;未设置时保留原命令。复用现有 backend 和配置字段,不新增 schema 或启动器,并更新现有示例和参考说明。SGLang 和 vLLM 已将显式配置的服务模型名传给服务器,SRT 也使用解析后的名称供 benchmark/eval 客户端请求。此修改让原生 TRT-LLM 具备一致的行为:名称只需配置一次,就会同时用于服务器和客户端。将该设置转换为服务器命令属于启动器的职责,因此修复放在现有 TRT backend 中。
此次仅统一显式名称的传递行为。各框架的 YAML 字段位置保持不变。未设置名称时,原生 TRT-LLM 仍保留服务器默认值;本 PR 不统一各 backend 在名称缺省时的行为,也不改变模型加载、精度或性能设置。
验证:
v1.3.0rc14源码确认该参数;commit 为93cb6518b6d6dbd6095748189e626db731f44545。关联 InferenceX #3352。当前配方通过在
engine.served_model_name和roles.*.extra_args: [--served_model_name, ...]中重复指定名称,已实现预期行为,因此迁移不依赖本 PR 合并。采用包含此修复的依赖版本后,即可移除重复的 CLI 参数,并保留预期的服务器和客户端模型名。保持草稿,审核前不合并。