[LMCache] Add LMCache arm to official DSV4 FP4 B200 vLLM AgentX sweep / 将 LMCache 分支并入官方 DSV4 FP4 B200 vLLM AgentX 扫描#2231
[LMCache] Add LMCache arm to official DSV4 FP4 B200 vLLM AgentX sweep / 将 LMCache 分支并入官方 DSV4 FP4 B200 vLLM AgentX 扫描#2231ApostaC wants to merge 3 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：将 perf-changelog 条目的 pr-link 填写为 PR #2231。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ApostaC
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29460892084 |
…sweep Rebased onto main after PR #2224 merged the tuned B200 recipe. The LMCache points now live directly in the official dsv4-fp4-b200-vllm-agentic search space alongside the vllm-simple and Mooncake arms (no standalone config section). LMCache 0.5.1 MP server (lmcache_driven transfer mode) + LMCacheMPConnector per PR #2153; the lmcache arm drops --enable-cumem-allocator because cuMem/VMM allocations cannot be CUDA-IPC-exported to the LMCache server. Ladder validated in the PR #2231 bring-up sweep: peak ~25.7k total tok/s/GPU at DEP8 conc 72 with 96-98% cache hit. 中文：在 PR #2224 的调优配方合并后 rebase 到 main。LMCache 测试点直接并入 官方 dsv4-fp4-b200-vllm-agentic 搜索空间，与 vllm-simple、Mooncake 分支 并列（不再使用独立配置段）。LMCache 0.5.1 MP server（lmcache_driven 传输模式）+ LMCacheMPConnector（沿用 PR #2153）；lmcache 分支去掉 --enable-cumem-allocator（cuMem/VMM 分配无法通过 CUDA IPC 导出给 LMCache server）。测试点阶梯已在 PR #2231 调试扫描中验证：DEP8 并发 72 达到峰值约 25.7k total tok/s/GPU，缓存命中率 96–98%。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cd7b86f to
14bd83f
Compare
ApostaC
changed the title
Change the LMCache DEP8 points from the vllm-simple ladder to the Mooncake ladder ([12, 20, 28, 36, 44, 52, 60, 68, 76]) so the two external KV stores are benchmarked at identical concurrency points. TP8 stays on [8, 12, 16] (Mooncake has no TP8 arm). 中文：将 LMCache DEP8 测试点从 vllm-simple 阶梯改为 Mooncake 阶梯 （[12, 20, 28, 36, 44, 52, 60, 68, 76]），使两个外部 KV 存储在完全相同的 并发点上进行基准测试。TP8 保持 [8, 12, 16]（Mooncake 无 TP8 分支）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29532605829 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29533029190 |
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29706772949 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29706772949 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29706772949 |
1 participant
Update Jul 20
vLLM official recipe:
Description
Adds an LMCache 0.5.1 DRAM KV-offload arm to the official
dsv4-fp4-b200-vllm-agenticsweep, alongside the vllm-simple and Mooncake arms, on the tuned recipe and image that #2224 merged (vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec). Rebased onto main after #2224 merged; the LMCache points now live directly in the official config's search space (the earlier standalonedsv4-fp4-b200-vllm-agentic-lmcachesection is gone).benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.shadds anlmcacheoffload-backend case (from [LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups #2153): LMCache MP server (lmcache_driventransfer mode) +LMCacheMPConnector, L1 pool derated to 75% ofTOTAL_CPU_DRAM_GB.--enable-cumem-allocatoris dropped, becauseLMCacheMPConnectorexports the KV cache through legacy CUDA IPC handles and cuMem/VMM allocations cannot be exported that way (register_kv_cachesfails withcudaErrorInvalidValue). All other flags are identical across arms, so backends are directly comparable.Already validated on hardware in this PR's earlier bring-up sweep (run 29460892084, pre-rebase, standalone-section structure — same script/recipe): 12/12 points that received nodes passed (2 points lost to SLURM allocation revocations, never reached the container). Peak ~25.7k total tok/s/GPU and 194.5 output tok/s/GPU at DEP8 conc 72 (14.2 tok/s/user), 96–98% LMCache hit rate across the ladder.
中文说明
在官方
dsv4-fp4-b200-vllm-agentic扫描中新增 LMCache 0.5.1 DRAM KV 卸载分支，与 vllm-simple、Mooncake 分支并列，使用 #2224 合并的调优配方与镜像（vllm/vllm-openai:nightly-dev-x86_64-cu13.0.1-904e4ec）。已在 #2224 合并后 rebase 到 main；LMCache 测试点直接并入官方配置的搜索空间（不再使用之前独立的dsv4-fp4-b200-vllm-agentic-lmcache配置段）。lmcache卸载后端分支（来自 [LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups #2153）：LMCache MP server（lmcache_driven传输模式）+LMCacheMPConnector，L1 池按TOTAL_CPU_DRAM_GB的 75% 降额。--enable-cumem-allocator：LMCacheMPConnector通过传统 CUDA IPC 句柄导出 KV cache，cuMem/VMM 分配无法以这种方式导出（register_kv_caches报cudaErrorInvalidValue）。其余参数在各分支间完全一致，便于卸载后端直接对比。本 PR 早期的调试扫描（run 29460892084，rebase 前的独立配置段结构，脚本/配方相同）已在硬件上验证：拿到节点的 12 个点全部通过（2 个点因 SLURM 分配被撤销未能启动容器）。峰值约 25.7k total tok/s/GPU、194.5 output tok/s/GPU（DEP8 并发 72，14.2 tok/s/user），全阶梯 LMCache 命中率 96–98%。
Related Issue
Builds on #2224 (merged tuned B200 recipe) and #2153 (LMCache backend). / 基于 #2224（已合并的 B200 调优配方）与 #2153（LMCache 后端）。
Type of Change
Checklist
perf-changelog.yamlperf-changelog.yamlentries are appended to the end of the file (the file is chronological: oldest at top, newest at bottom)OWNER/MEMBER/COLLABORATOR) has commented/reuse-sweep-runon this PR — do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps (remove and re-add the label to force one)🤖 Generated with Claude Code