Test H100 ModelScope loading from empty caches / 测试 H100 ModelScope 冷缓存加载 - #3324
functionstackx wants to merge 6 commits into
Conversation
为 H100 添加 TensorRT-LLM ModelScope 覆盖，使用 Qwen3-0.6B BF16 和与 1.3.0rc27 镜像源码匹配的回移补丁。
记录 PR #3323 的 TensorRT-LLM ModelScope 运行时补丁豁免和移除计划。
改用 ModelScope 原生 glob 过滤参数，并让基准客户端复用 ModelScope 本地 tokenizer。
Document the full-split sample audit, external scale context, and conservative 0.60 model-specific floor while preserving deterministic evaluation settings. 为 Qwen3-0.6B 记录完整测试集分析、外部规模参考和保守的 0.60 GSM8K 下限，保持确定性评测设置不变。 Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
在每次 H100 任务中从独立空缓存启动 ModelScope 下载，并记录模型文件哈希、验证无 Hugging Face 回退。 Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
为 PR 3324 更新冷缓存变更记录与引擎补丁豁免链接。 Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
The PR-associated full sweep is running on head CI passed: 1,880 tests passed, one skipped, MCP compatibility passed, and lint passed. Changelog validation and sweep setup also passed. GPU outcomes are still pending. I will verify the fresh-cache reports, model/tokenizer hashes, server logs, per-job evaluation metadata and samples, and aggregate artifacts before reporting success. |
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35544218778 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35544218778 |
|
Final PR validation: passed. InferenceX PR #3324, with All five throughput points passed: c1/c4/c16/c32/c64, 1,170/1,170 requests completed, zero failures, and valid power measurements. Both full GSM8K evaluations passed the unchanged 0.60 model-specific floor:
Verified the raw results and samples, independently expected concurrency metadata, both score-validator outputs, five throughput/power audits, all seven server logs and job logs, and the collected artifacts. The throughput aggregate has exactly five matching rows; the eval aggregate has exactly two matching rows. Five-shot chat, temperature 0, top_p 1, 5,376 output tokens, and no sample limit were preserved. Every one of the seven jobs started with empty ModelScope and isolated HF caches. Each downloaded snapshot's seven inference assets match the HF reference SHA256, including weights and tokenizer. The server logs show the ModelScope weight path; the HF caches contain only the two allowed empty Transformers bookkeeping files. All seven job logs confirm the pinned image and successful runtime patch. No error-level server entries, tracebacks, or logged HF model-download fallback were found. CPU CI passed: 1,880 tests, one skipped, MCP compatibility, and lint. Python CodeQL passed. The sweep's final GitHub job records confirm 7/7 GPU jobs passed. Reporting caveat: This validates the source-matched runtime backport in TensorRT-LLM 中文最终 PR 验证通过。 InferenceX PR #3324 已添加 五个吞吐并发点全部通过，1,170/1,170 请求完成、零失败，功耗数据有效。两次完整 GSM8K 均通过未修改的 0.60 模型专用下限：c32 为 strict 870/1,319、flexible 878/1,319；c64 为 strict 884/1,319、flexible 889/1,319。均完成全部题目，无空响应。 已核验原始结果与样本、预期并发元数据、两份评分校验输出、五份吞吐和功耗记录、七份服务日志及任务日志，以及汇总产物。五条吞吐汇总和两条评测汇总均与原始数据一致。保留五样本聊天、temperature 0、top_p 1、5,376 token 输出上限及完整测试集。 七个任务均从空 ModelScope 缓存和独立空 HF 缓存开始。 每份快照的七个推理文件均与 HF 基线哈希一致。日志确认从 ModelScope 路径加载权重，HF 目录只有两个允许的空占位文件。任务日志确认固定镜像和补丁成功，未发现服务错误、traceback 或 HF 模型下载回退记录。 CPU CI 的 1,880 项测试通过，一项跳过，MCP、lint 和 Python CodeQL 通过。最终 GitHub 记录确认 7/7 GPU 任务成功。 本次验证的是 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
This automated review found the issues noted in the inline comments. Beyond those, I also checked the HF-fallback provenance exemption in runners/modelscope_snapshot.py (the version.txt allowance) — it matches only the benign cache-format stamp file that huggingface_hub/transformers write on their own, not an actual model asset, so it doesn't undermine the intended "no HF fallback download" check.
Extended reasoning...
Two confirmed bugs are already queued as inline comments (a non-idempotent llm.py patch guard in patch_trtllm_modelscope.py, and an errexit regression in the new benchmark script relative to its sibling scripts), so approval is off the table per the guidelines. I additionally read runners/modelscope_snapshot.py's HF-fallback provenance check, which exempts any file literally named version.txt anywhere under hf_home regardless of path/size — this looked suspicious at first glance but corresponds to the known cache-format marker file that huggingface_hub writes unconditionally (not a real weight/config fallback), so I did not find it to be an additional gap worth flagging inline beyond what's already reported.
Findings marked 🟡 are optional suggestions and need no follow-up push.
| def patch_llm(path: Path) -> bool: | ||
| """Load tokenizer and configuration from the downloaded snapshot.""" | ||
| source = path.read_text(encoding="utf-8") | ||
| tokenizer_marker = " model_path = self._hf_model_dir or self.args.model\n" | ||
| generation_marker = " model_dir = self._hf_model_dir or self.args.model\n" | ||
| if tokenizer_marker in source: | ||
| if source.count(generation_marker) < 3: | ||
| raise RuntimeError(f"partial ModelScope path patch found at {path}") | ||
| return False | ||
|
|
||
| tokenizer_start = source.index(" def _try_load_tokenizer(") | ||
| tokenizer_end = source.index("\n @property\n def tokenizer", tokenizer_start) | ||
| tokenizer_source = source[tokenizer_start:tokenizer_end] | ||
| anchor = ( | ||
| " if self.args.tokenizer is not None:\n" | ||
| " assert isinstance(self.args.tokenizer, TokenizerBase)\n" | ||
| " return self.args.tokenizer\n\n" | ||
| ) | ||
| if tokenizer_source.count("self.args.model") != 4: | ||
| raise RuntimeError( | ||
| f"unsupported TensorRT-LLM tokenizer loader at {path}: " | ||
| "unexpected model-path reference count" | ||
| ) | ||
| tokenizer_source = tokenizer_source.replace("self.args.model", "model_path") | ||
| tokenizer_source = replace_once( | ||
| tokenizer_source, anchor, anchor + tokenizer_marker + "\n", path | ||
| ) | ||
| source = source[:tokenizer_start] + tokenizer_source + source[tokenizer_end:] | ||
|
|
||
| source = replace_once( | ||
| source, | ||
| " return ModelLoader.load_hf_generation_config(self.args.model)\n", | ||
| generation_marker | ||
| + " return ModelLoader.load_hf_generation_config(model_dir)\n", | ||
| path, | ||
| ) | ||
| source = replace_once( | ||
| source, | ||
| " return ModelLoader.load_hf_model_config(\n" | ||
| " self.args.model, trust_remote_code=self.args.trust_remote_code)\n", | ||
| generation_marker + " return ModelLoader.load_hf_model_config(\n" | ||
| " model_dir, trust_remote_code=self.args.trust_remote_code)\n", | ||
| path, | ||
| ) | ||
| path.write_text(source, encoding="utf-8") | ||
| return True | ||
|
|
There was a problem hiding this comment.
🔴 patch_trtllm_modelscope.py claims to be idempotent, but re-running it against an already-patched tensorrt_llm/llmapi/llm.py raises RuntimeError('partial ModelScope path patch found') and exits 1 instead of cleanly no-op-returning. patch_llm's re-entrancy guard at line 173 requires source.count(generation_marker) >= 3, but the function itself only ever inserts generation_marker twice (lines 199, 207), so a fully and correctly patched file always fails its own idempotency check on the next invocation. Fix: set the guard to the actual number of generation_marker insertions this function performs (2), or verify each insertion site independently instead of a single combined count threshold.
Extended reasoning...
The benchmark script (benchmarks/single_node/fixed_seq_len/qwen3-0.6b_bf16_h100_trt.sh) invokes patch_trtllm_modelscope.py's main() every time it runs against the pinned image. First invocation on a fresh install: llm.py has no tokenizer_marker yet, so patch_llm takes the full patch branch, writes tokenizer_marker once (line 192-194) and generation_marker twice (lines 199 and 207), returns True. Second invocation against the same already-patched image/container (a rerun after an earlier step failed, a retry, or a container reused for another search-space combo in the same job) hits line 172: tokenizer_marker is now present, so it checks source.count(generation_marker) < 3 at line 173. The real count is exactly 2, never 3, so the guard always raises RuntimeError('partial ModelScope path patch found'), which main() (line 213 area) turns into ERROR output and return 1. With the script's set -eo pipefail, this aborts the whole job even though the installed tensorrt_llm source is fully, correctly patched, contradicting the 'idempotent' claim in docs/waiver/3323.md.
Verification: normal. The idempotency/re-entrancy branch of patch_llm is dead: it always raises instead of no-op returning. The guard at runners/patch_trtllm_modelscope.py:172-175 does if tokenizer_marker in source: if source.count(generation_marker) < 3: raise RuntimeError("partial ModelScope path patch found at {path}"), where generation_marker (line 171) = `" model_dir = self._hf_model_dir or…
| @@ -0,0 +1,126 @@ | |||
| #!/usr/bin/env bash | |||
| set -eo pipefail | |||
There was a problem hiding this comment.
🟡 (optional) This script adds set -eo pipefail (line 2), unlike every other single_node/fixed_seq_len benchmark script, so operators lose the GSM8K eval results and clean GPU metrics that siblings still produce after a benchmark hiccup. If run_benchmark_serving (line 106) fails transiently, the script exits immediately and never reaches run_eval (line 120) or stop_gpu_monitor/rm -f "$EXTRA_CONFIG_FILE" (lines 124-125), so the eval stage is skipped entirely and gpu_metrics.csv is left with a truncated tail instead of being finalized. Fix: keep run_benchmark_serving and run_eval failures non-fatal (e.g. capture their exit codes) so cleanup and the eval stage still run on transient failures, consistent with dsr1_fp8_h200_trt.sh and other sibling scripts that intentionally omit set -e.
Extended reasoning...
grep '^set -e' across benchmarks/single_node/fixed_seq_len/*.sh shows this is the only script with errexit; all siblings (e.g. dsr1_fp8_h200_trt.sh) omit it deliberately, relying on falling through to stop_gpu_monitor even when run_benchmark_serving fails. run_benchmark_serving (benchmark_lib.sh:793) returns non-zero on ordinary conditions (missing args, benchmark_exit_code from infx.bench_serving.benchmark_serving, capture failures) not just catastrophic errors. With set -e active here, that non-zero return terminates the script at line 106 before line 119's RUN_EVAL check, so the GSM8K accuracy eval that this PR's own validation section relies on never runs for that job. It also skips stop_gpu_monitor (benchmark_lib.sh:478), which normally appends a final nvidia-smi sample and repairs a truncated trailing CSV row via _repair_truncated_gpu_metrics_tail; skipping it leaves gpu_metrics.csv with a partial/truncated last row. The mktemp'd EXTRA_CONFIG_FILE at line 52 also leaks since rm -f at line 125 is never reached.
Verification: nit. The mechanism is real and reachable but low severity. Line 2 of the new file is set -eo pipefail (the only fixed_seq_len script with errexit; siblings such as dsr1_fp8_h200_trt.sh omit it). run_benchmark_serving (line 106) is invoked as a bare command and returns $benchmark_exit_code (benchmark_lib.sh:1018), which is non-zero when run_server_client fails, when the `server_watch…
This branch has not been deployed
1 participant
Description
Add H100 BF16 coverage for
Qwen/Qwen3-0.6Bthrough ModelScope. Every job starts with a verified-empty ModelScope cache and an independent empty HF cache.trtllm-servedownloads the remote model itself; the benchmark then uses the tokenizer from that same local snapshot. Snapshot reports record file sizes and SHA256 and reject unexpected HF fallback files.The lane uses
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc27, matching sourcev1.3.0rc27(6e1cc953c071b8a9055b03ef2ae4ee0bc4c645c4), with a source-matched runtime backport of TensorRT-LLM #2. Dependencies are pinned tomodelscope==1.40.1andmodelscope-hub==0.4.3. The engine-patch waiver records scope, provenance, and removal. This is the requested new cold-cache PR and includes the unmerged integration from #3323.Validation
Full PR sweep passed on
e78910cb90fd425ef538e244c89e5524da161598, withfull-sweep-enabled. Complete artifact audit.The final GitHub job records show 7/7 GPU jobs passed.
run-statsis an early 6/7 snapshot: it completed 46 seconds before the last eval. Separate canary jobs were skipped, while all five throughput points and both evals executed.The controlled 12-evaluation HF/ModelScope comparison also completed. It verified identical input assets, prompts and settings, but did not establish identical accuracy: ModelScope versus patched-HF strict means differ by −0.0758 percentage points at c32 and −1.5921 at c64. Same-arm repeat variation and the fixed-order, two-repeat design limit causal interpretation. Passing the integration floor is not an equivalence claim.
GPU evidence covers the release-compatible backport, not a build of the newer TensorRT-LLM PR branch. Model downloads and inference ran only on H100. This PR remains unmerged.
AI model disclosure
Earlier integration work was disclosed as GPT-5; that earlier exact identifier could not be independently verified. This continuation used GPT-6-based Codex for implementation, validation, and monitoring; the exact runtime model/version is not exposed and could not be verified. No delegated agents were used.
中文
说明
通过 ModelScope 为
Qwen/Qwen3-0.6B添加 H100 BF16 覆盖。每个任务均使用确认为空的 ModelScope 缓存和独立 HF 缓存，由trtllm-serve自行下载远程模型，随后基准客户端使用同一快照中的分词器。报告记录文件大小和 SHA256，并拒绝非预期的 HF 回退文件。镜像固定为
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc27，对应源码v1.3.0rc27（6e1cc953c071b8a9055b03ef2ae4ee0bc4c645c4），使用 TensorRT-LLM #2 的源码匹配回移补丁。依赖固定为modelscope==1.40.1和modelscope-hub==0.4.3，引擎补丁豁免位于docs/waiver/3324.md。这是用户要求的新冷缓存 PR，包含 #3323 尚未合并的集成。验证
提交
e78910cb90fd425ef538e244c89e5524da161598的完整 PR sweep 已通过。五个吞吐并发点完成 1,170/1,170 请求、零失败，功耗数据有效。完整 GSM8K：c32 为 strict 870/1,319（65.9591%）、flexible 878/1,319（66.5656%）；c64 为 strict 884/1,319（67.0205%）、flexible 889/1,319（67.3995%）。均无空响应，保留五样本聊天、贪心参数、5,376 token 输出上限及已说明的 0.60 模型专用回归下限；全局阈值未变。七个任务均从空缓存开始，七个推理文件的哈希与 HF 基线一致。已核验全部服务及任务日志、原始样本、预期并发元数据、评分校验、功耗记录，以及与原始数据一致的五条吞吐和两条评测汇总。CI 的 1,880 项测试通过，一项跳过；MCP、lint 和 Python CodeQL 均通过，本地相关检查也通过。
最终 GitHub 记录确认 7/7 GPU 任务成功。
run-stats中的 6/7 是提前 46 秒生成的快照；独立 canary 被跳过，但全部五个吞吐点和两次评测实际执行。独立的 12 次 HF/ModelScope 对比已完成，确认资产、提示词和参数一致，但不能证明准确率完全相同。ModelScope 相对 patched HF 的 strict 均值差异为 c32 −0.0758、c64 −1.5921 个百分点。同组重复波动及固定顺序、两次重复的设计限制了因果判断。通过集成回归下限不代表等价。
GPU 证据验证的是发布版本的回移补丁，并未构建较新的 TensorRT-LLM PR 分支。模型下载和推理仅在 H100 运行。PR 未合并。
AI 模型披露
早期集成工作披露使用 GPT-5，但无法独立核实该精确标识。本次使用基于 GPT-6 的 Codex 实现、验证和监控；环境未暴露精确运行时模型/版本，因此无法核实。未使用委派代理。