[LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups by ApostaC · Pull Request #2153 · SemiAnalysisAI/InferenceX

ApostaC · 2026-07-10T16:49:16Z

Summary

Adding LMCache configurations for dsv4-vllm launching script
Adding 2 nvidia configs with lmcache agentix

Signed-off-by: ApostaC <yihua98@uchicago.edu>

…ntry Resolve the perf-changelog.yaml tail conflict with #2127 per KLAUD_DEBUG.md §1.1: take main's changelog wholesale and re-append this PR's entry at the end, with pr-link updated from the XXX placeholder to #2153. 中文：将 origin/main 合并到 lmcache/agentix，按 KLAUD_DEBUG.md §1.1 处理 perf-changelog.yaml 末尾冲突：整体采用 main 的版本并在末尾重新追加本 PR 的条目，同时把 pr-link 占位符更新为 #2153。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

claude

Additional findings (outside current diff — PR may have been updated during review):

🟡 perf-changelog.yaml:4720-4727 — PR title and description are English-only, but AGENTS.md requires bilingual PR metadata: title format <English title> / <中文标题> and a ## 中文说明 section in the body mirroring the English summary. Please update the title (e.g. append / [LMCache] 为 dsv4 vllm b200/b300 agentix 配置添加 LMCache) and add a ## 中文说明 section to the description before merge.
Extended reasoning...

AGENTS.md line 7 states: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: ' / <中文标题>'. In the PR/issue body, follow the English content with its Chinese translation (e.g. a '## 中文说明' section mirroring the summary)." This is a repo-wide convention that applies to every PR.

The PR under review is missing both required pieces:
- Title: [LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups — English only, no / <中文标题> suffix.
- Body: two English bullets under # Summary ("Adding LMCache configurations for dsv4-vllm launching script" / "Adding 2 nvidia configs with lmcache agentix") with no ## 中文说明 section following them.
Step-by-step proof of the violation:
1. Open AGENTS.md at line 7 — the bilingual rule is explicit and unambiguous, with the exact title format and the ## 中文说明 section pattern spelled out.
2. Read the PR title as rendered on GitHub — it contains only ASCII characters and no / separator introducing a Chinese portion.
3. Read the PR body — the entire description is 3 lines of English (# Summary header plus two bullets); grep for 中文说明 in the body returns nothing.
4. Therefore both required bilingual elements are absent, directly violating the rule at AGENTS.md:7.
Impact: This is metadata-only — the benchmark scripts and configs themselves are unaffected, CI will still run, and no runtime behavior changes. But the repo convention exists so that non-English speakers on the team can review and search PR history, and every merged PR sets precedent. The rule is stated as a requirement, not a suggestion.

How to fix (no code change needed):
1. Edit the PR title to append a Chinese translation after /, e.g. [LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups / [LMCache] 为 dsv4 vllm b200/b300 agentix 配置添加 LMCache.
2. Add a ## 中文说明 section to the PR body after the English summary, mirroring the two bullets — for example:
```
## 中文说明
- 为 dsv4-vllm 启动脚本添加 LMCache 配置
- 新增 2 个使用 LMCache agentix 的 NVIDIA 配置
```
Severity is nit — this is a documentation/metadata policy issue, easy to fix in the PR description with no code impact, but should be addressed before merge since AGENTS.md declares it a requirement.

…ntry Signed-off-by: ApostaC <yihua98@uchicago.edu>

github-actions · 2026-07-10T17:00:29Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29109297629
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29109297629

Bring-up on PR #2153 showed full-budget pinned pools OOM-killing B200 nodes (host OOM-killer / cudaErrorMemoryAllocation) as the LMCache lazy allocator filled past ~2 TB, so cap --l1-size-gb at 75% of TOTAL_CPU_DRAM_GB. Run the LMCache section on vllm/vllm-openai:v0.24.0 (the LMCache-0.5.x-validated pairing) with the CUTLASS DSL cu13 reinstall gated to vLLM < 0.24, and drop the B300 LMCache section for now to focus test runs on B200. 中文：PR #2153 启动调试发现满预算固定内存池会在缓存填充超过约 2 TB 时触发节点 OOM（主机 OOM-killer / cudaErrorMemoryAllocation），故将 --l1-size-gb 上限降至 TOTAL_CPU_DRAM_GB 的 75%。LMCache 配置改用 vllm/vllm-openai:v0.24.0 镜像（与 LMCache 0.5.x 验证过的组合），CUTLASS DSL cu13 重装仅在 vLLM < 0.24 时执行；暂时移除 B300 的 LMCache 配置，将测试集中在 B200。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Signed-off-by: ApostaC <yihua98@uchicago.edu>

github-actions · 2026-07-10T19:08:20Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29109420492
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29109420492

github-actions · 2026-07-10T19:09:58Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29116938835
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29116938835

functionstackx · 2026-07-10T22:00:10Z

+# The v0.24 images ship fixed bindings; reinstalling there would downgrade.
+VLLM_INSTALLED_VERSION=$(python3 -c "from importlib.metadata import version; print(version('vllm'))")
+if [ "$(printf '%s\n' "$VLLM_INSTALLED_VERSION" 0.24.0 | sort -V | head -n1)" != "0.24.0" ]; then
+    agentic_pip_install --quiet --force-reinstall --no-deps \
+        'nvidia-cutlass-dsl-libs-cu13==4.5.2'


generally we don't allow reinstalling kernel libraries

functionstackx

thanks for the contribution @ApostaC

can u please update recipes.vllm.ai with ur new configs? https://github.com/vllm-project/recipes

vllm recipes doesn't currently support offloading tho the inferact guys said they are open to contributions to support mooncake store, lmcache, vllm native offloading support in vllm recipes

as in sglang cookbook recipes , they also have support for cpu offloading recipes

Signed-off-by: ApostaC <yihua98@uchicago.edu>

github-actions · 2026-07-10T22:46:48Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29117029396
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29117029396

functionstackx · 2026-07-10T22:47:17Z

        agentic_pip_install --quiet --no-cache-dir --no-deps \
            --force-reinstall "mooncake-transfer-engine-cuda13==$MOONCAKE_VERSION"
        python3 -c "from mooncake.store import MooncakeDistributedStore" >/dev/null
+        python3 "$(dirname "$0")/patch_vllm_pr45406.py"


generally we wanna avoid patches too

Yeah, this is what I picked from #2138. Once that is in, I will remove this.

github-actions · 2026-07-11T00:45:26Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29128619300
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29128619300

Signed-off-by: ApostaC <yihua98@uchicago.edu>

github-actions · 2026-07-11T09:20:18Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29133742844
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29133742844

Signed-off-by: ApostaC <yihua98@uchicago.edu>

github-actions · 2026-07-11T18:50:15Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29133742844
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29133742844

github-actions · 2026-07-11T19:27:08Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29164165383
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29164165383

Signed-off-by: ApostaC <yihua98@uchicago.edu>

github-actions · 2026-07-11T22:38:27Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29168226910
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29168226910

github-actions · 2026-07-12T01:08:00Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29170738763
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29170738763

Signed-off-by: ApostaC <yihua98@uchicago.edu>

github-actions · 2026-07-12T03:12:02Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29174945007
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29174945007

Signed-off-by: ApostaC <yihua98@uchicago.edu>

github-actions · 2026-07-12T05:14:36Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29177997711
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29177997711

Signed-off-by: ApostaC <yihua98@uchicago.edu>

ApostaC · 2026-07-13T19:12:24Z

-# AgentX concurrency counts live session trees, not individual requests.
-# Subagent fan-out can push instantaneous request concurrency above CONC, so
-# leave 2x headroom rather than clipping those bursts at the scheduler.
-MAX_NUM_SEQS=$((2 * CONC))
-


This is being moved to the top of the script

github-actions · 2026-07-13T19:51:24Z

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29272097721
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29272097721

Add LMCache configs for dsv4 vllm b200/b300

10e6639

Signed-off-by: ApostaC <yihua98@uchicago.edu>

ApostaC requested a review from a team July 10, 2026 16:49

ApostaC requested review from Ankur-singh, jgangani and kedarpotdar-nv as code owners July 10, 2026 16:49

github-project-automation Bot added this to InferenceMAX Board Jul 10, 2026

update perf changelog

73137f1

Signed-off-by: ApostaC <yihua98@uchicago.edu>

ApostaC added the sweep-enabled label Jul 10, 2026

cquil11 added the agentx AgentX benchmarks, recipes, and infrastructure label Jul 10, 2026 — with ChatGPT Codex Connector

claude Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread benchmarks/single_node/agentic/dsv4_fp4_b200_vllm.sh

Comment thread perf-changelog.yaml Outdated

Merge origin/main into lmcache/agentix, re-append lmcache changelog e…

951f2c8

…ntry Signed-off-by: ApostaC <yihua98@uchicago.edu>

ApostaC force-pushed the lmcache/agentix branch from 0223f79 to 951f2c8 Compare July 10, 2026 17:00

lmcache: derate L1 pool to 75%, move to vLLM v0.24.0, defer B300 section

4244f91

Signed-off-by: ApostaC <yihua98@uchicago.edu>

ApostaC force-pushed the lmcache/agentix branch from f00b597 to 4244f91 Compare July 10, 2026 19:07

ApostaC added full-sweep-enabled and removed sweep-enabled labels Jul 10, 2026

functionstackx reviewed Jul 10, 2026

View reviewed changes

functionstackx requested changes Jul 10, 2026

View reviewed changes

ApostaC added 2 commits July 10, 2026 15:37

fixing the comments and picking optimizations from #2138

380ee5a

Signed-off-by: ApostaC <yihua98@uchicago.edu>

merging upstream updates

6623385

Signed-off-by: ApostaC <yihua98@uchicago.edu>

functionstackx reviewed Jul 10, 2026

View reviewed changes

ApostaC mentioned this pull request Jul 11, 2026

Add KV cache offloading (LMCache) feature to DeepSeek-V4-Pro vllm-project/recipes#625

Closed

revert to 0.24.0

d9209dd

Signed-off-by: ApostaC <yihua98@uchicago.edu>

ApostaC added 2 commits July 11, 2026 11:33

Merge remote-tracking branch 'origin/main' into lmcache/agentix

b935b80

update vllm commands

fa1a16c

Signed-off-by: ApostaC <yihua98@uchicago.edu>

ApostaC added 2 commits July 11, 2026 14:06

update configs

ef9db5e

Signed-off-by: ApostaC <yihua98@uchicago.edu>

update gpu memory util

7439500

Signed-off-by: ApostaC <yihua98@uchicago.edu>

add gpu memory util for EP

b750702

Signed-off-by: ApostaC <yihua98@uchicago.edu>

move to 0.88 gpu mem util

0b8646f

Signed-off-by: ApostaC <yihua98@uchicago.edu>

ApostaC added 2 commits July 13, 2026 10:50

Merge remote-tracking branch 'origin/main' into lmcache/agentix

e550e72

update configure and allow vllm to log

fc97185

Signed-off-by: ApostaC <yihua98@uchicago.edu>

ApostaC commented Jul 13, 2026

View reviewed changes

cquil11 mentioned this pull request Jul 14, 2026

fix(inference): let unofficial run overlays win the cross-engine AgentX guard / 修复：让非官方 run 的 overlay 在跨引擎 AgentX 互斥中优先显示 SemiAnalysisAI/InferenceX-app#575

Merged

Uh oh!

Conversation

ApostaC commented Jul 10, 2026

Summary

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

functionstackx Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

ApostaC Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

functionstackx left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 10, 2026

Uh oh!

functionstackx Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

ApostaC Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 11, 2026

Uh oh!

github-actions Bot commented Jul 12, 2026

Uh oh!

github-actions Bot commented Jul 12, 2026

Uh oh!

github-actions Bot commented Jul 12, 2026

Uh oh!

ApostaC Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants