From 8e4158b1a6760ffa9db7a2e6b3a690019468e948 Mon Sep 17 00:00:00 2001 From: yihou Date: Fri, 7 Aug 2026 07:51:57 +0000 Subject: [PATCH 1/3] docs(patches): give every patch a schema-checked upstream status record The single patch.upstream.status.md table could not answer the question that decides whether a patch can go: does it change bytes in THIS engine, on THIS pinned base. It also had no gate, so a patch could be added without a row -- patch_mooncake_mamba_unpack.py went unindexed from 2026-08-02 to 2026-08-05. Replace it with one .upstream.status.yaml beside each patch (25 records), a patch.upstream.status.yaml index grouped by the library patched, and a single patch.archived.yaml for retired patches. Three JSON schemas validate them; scripts/validate-patch-status.py also cross-checks the set against the tree, so a patch without a record, an index whose totals drift, or an archived entry that cannot be recovered fails the lint gate. Each record carries the fields the old table lacked: per-engine op/no-op with its evidence, an enumerated reason the patch is still alive, and drop_signal -- how you find out it is dead. Writing that down surfaced four things: - Four patches can only be marked `unverified`: whether they are load-bearing depends on what the pinned base already carries and nobody has checked. - HIPFILE_GIT_REF is unpinned, so two images from the same infera commit built either side of rocm-systems#7386 (2026-07-16) hold different hipfile code. - patch_hicache_rocm_staged_write_back.py does not drop when its MERGED PR lands -- sglang#28534 is what INTRODUCED the defect; it drops on #30350. - Six patches keep applying cleanly after upstream fixes them, so only the pinned version says they are dead; drop_signal now enumerates which. Nothing is deleted: the legacy vllm-dsv4 patches move to patches/archived/ via git mv so history follows, and the old table is kept as patch.upstream.status.superseded.md. Image behaviour is unchanged. Every Dockerfile patch loop globs *.py or patch_*.py, so no .upstream.status.yaml can be executed, and nothing copies patches/ wholesale, so _schema/ and archived/ never enter an image. The gate itself had two holes, both from discovery only running patch -> record: - extra_files was a global set with no check that the claiming record sits anywhere near the file it claims, so one line in any record -- in any directory -- marked an unrelated new patch as covered and the gate passed. A record may now only name files beside it, which is what the schema's own description ("further files that are part of the same patch") already said. - Nothing walked record -> patch, so a record whose patch was renamed or deleted was never loaded, never schema-checked, and never reported. A git mv of a patch is the realistic version of this. Three tests cover those, each confirmed failing against the validator without the fix. Finally, yamllint over deploy/docker/**.yaml, which caught eleven values losing text: an unquoted ` #` opens a YAML comment, so a PR number written mid-sentence silently ends the value, and the result is still valid YAML that still passes its schema. action: Get #2725 reviewed; it has had no activity ... -> "Get" reusable_on: An aiter older than #3033, e.g. the earlier ... -> "An aiter older than" Quoting the eleven restores the text; that diff is quotes only, no wording changed. --strict, because the truncation is reported as a warning and a hook that exits 0 on warnings would not have caught it. line-length is 160 rather than the default 80: these records quote upstream issue titles and engine error strings verbatim, and those stay byte-exact to stay greppable. Scoped to deploy/docker/ -- deploy/operator/ is a much larger body of existing k8s YAML and is left alone for now. Signed-off-by: yihou --- .pre-commit-config.yaml | 28 + .yamllint.yaml | 42 ++ deploy/docker/Dockerfile.vllm | 4 +- deploy/docker/patch.upstream.status.md | 230 ------- deploy/docker/patch.upstream.status.yaml | 550 ++++++++++++++++ .../_schema/patch.archived.schema.json | 199 ++++++ .../_schema/patch.upstream.index.schema.json | 205 ++++++ .../_schema/patch.upstream.status.schema.json | 585 ++++++++++++++++++ .../patches/archived/patch.archived.yaml | 360 +++++++++++ .../legacy => archived/vllm-dsv4}/README.md | 9 +- .../vllm-dsv4}/patch_dsv4_aiter_moe.py | 0 .../vllm-dsv4}/patch_dsv4_mhc_aiter.diff | 0 .../vllm-dsv4}/patch_dsv4_mhc_aiter.py | 0 ...gdn_pd_state_transfer.upstream.status.yaml | 146 +++++ ...inimax_m2_qknorm_rope.upstream.status.yaml | 140 +++++ ...ooncake_consumer_slot.upstream.status.yaml | 123 ++++ .../hipfile_async.upstream.status.yaml | 180 ++++++ ...ma_auto_chunk_mr_2017.upstream.status.yaml | 177 ++++++ ...ransport_dmabuf_cmake.upstream.status.yaml | 159 +++++ .../transfer_engine_impl.upstream.status.yaml | 183 ++++++ ...2_nextn_quark_exclude.upstream.status.yaml | 169 +++++ ...early_send_wait_event.upstream.status.yaml | 188 ++++++ deploy/docker/patches/sglang_dsa/README.md | 7 +- ...ft_cuda_graph_dp_vote.upstream.status.yaml | 227 +++++++ ...c_and_page_table_rows.upstream.status.yaml | 196 ++++++ ...er_hip_dp_padded_rows.upstream.status.yaml | 229 +++++++ ...cache_rocm_host_alloc.upstream.status.yaml | 203 ++++++ ...ocm_staged_write_back.upstream.status.yaml | 238 +++++++ ...dsl_moe_memref_bufres.upstream.status.yaml | 129 ++++ ...dsv4_hybrid_blocksize.upstream.status.yaml | 134 ++++ ...v4_noncontig_register.upstream.status.yaml | 122 ++++ ...o_dsv4_sparse_backend.upstream.status.yaml | 128 ++++ ...tch_defer_kv_register.upstream.status.yaml | 162 +++++ ...mooncake_mamba_unpack.upstream.status.yaml | 161 +++++ .../patch_moriio_pagelen.upstream.status.yaml | 165 +++++ .../patch_moriio_write.upstream.status.yaml | 164 +++++ .../patch_sched_guard.upstream.status.yaml | 183 ++++++ ...lm_mooncake_blocksize.upstream.status.yaml | 180 ++++++ ...mooncake_prom_metrics.upstream.status.yaml | 193 ++++++ pyproject.toml | 3 + scripts/validate-patch-status.py | 288 +++++++++ tests/unit/test_patch_status_records.py | 175 ++++++ 42 files changed, 6729 insertions(+), 235 deletions(-) create mode 100644 .yamllint.yaml delete mode 100644 deploy/docker/patch.upstream.status.md create mode 100644 deploy/docker/patch.upstream.status.yaml create mode 100644 deploy/docker/patches/_schema/patch.archived.schema.json create mode 100644 deploy/docker/patches/_schema/patch.upstream.index.schema.json create mode 100644 deploy/docker/patches/_schema/patch.upstream.status.schema.json create mode 100644 deploy/docker/patches/archived/patch.archived.yaml rename deploy/docker/patches/{vllm-dsv4/legacy => archived/vllm-dsv4}/README.md (79%) rename deploy/docker/patches/{vllm-dsv4/legacy => archived/vllm-dsv4}/patch_dsv4_aiter_moe.py (100%) rename deploy/docker/patches/{vllm-dsv4/legacy => archived/vllm-dsv4}/patch_dsv4_mhc_aiter.diff (100%) rename deploy/docker/patches/{vllm-dsv4/legacy => archived/vllm-dsv4}/patch_dsv4_mhc_aiter.py (100%) create mode 100644 deploy/docker/patches/atom/patch_gdn_pd_state_transfer.upstream.status.yaml create mode 100644 deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.upstream.status.yaml create mode 100644 deploy/docker/patches/atom/patch_mooncake_consumer_slot.upstream.status.yaml create mode 100644 deploy/docker/patches/hipfile_async/hipfile_async.upstream.status.yaml create mode 100644 deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.upstream.status.yaml create mode 100644 deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.upstream.status.yaml create mode 100644 deploy/docker/patches/mooncake_cpp/transfer_engine_impl.upstream.status.yaml create mode 100644 deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.upstream.status.yaml create mode 100644 deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml create mode 100644 deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.upstream.status.yaml create mode 100644 deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml create mode 100644 deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.upstream.status.yaml create mode 100644 deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml create mode 100644 deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.upstream.status.yaml create mode 100644 deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.upstream.status.yaml create mode 100644 deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.upstream.status.yaml create mode 100644 deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.upstream.status.yaml create mode 100644 deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.upstream.status.yaml create mode 100644 deploy/docker/patches/vllm/patch_defer_kv_register.upstream.status.yaml create mode 100644 deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.upstream.status.yaml create mode 100644 deploy/docker/patches/vllm/patch_moriio_pagelen.upstream.status.yaml create mode 100644 deploy/docker/patches/vllm/patch_moriio_write.upstream.status.yaml create mode 100644 deploy/docker/patches/vllm/patch_sched_guard.upstream.status.yaml create mode 100644 deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.upstream.status.yaml create mode 100644 deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.upstream.status.yaml create mode 100644 scripts/validate-patch-status.py create mode 100644 tests/unit/test_patch_status_records.py diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 5abd08ba..b50e15fc 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -28,6 +28,19 @@ repos: - id: ruff-check args: [--fix] + # Style gate for the hand-written YAML under deploy/docker/ only. check-yaml + # above answers "does this parse"; yamllint answers "does it say what its + # author meant" — an unquoted ` #` opens a comment, so a PR number mid-sentence + # silently truncates the value. Scoped by `files:` rather than turned loose on + # the repo: deploy/operator/ holds a much larger body of existing k8s YAML and + # is left alone deliberately. Config in .yamllint.yaml. + - repo: https://github.com/adrienverge/yamllint + rev: v1.38.0 + hooks: + - id: yamllint + args: [--strict] # warnings fail too; a hook that exits 0 gets ignored + files: ^deploy/docker/.*\.ya?ml$ + # Rust formatting gate (needs a local toolchain: rustup + rustfmt). clippy and # tests are heavier, so they run in CI (.github/workflows/ci.yml, job `rust`). - repo: local @@ -39,6 +52,21 @@ repos: files: ^rust/.*\.rs$ pass_filenames: false + # Every patch under deploy/docker/patches/ must have a status record that validates + # against its schema and agrees with the index, so a patch cannot be added — or + # silently outlive its upstream fix — without saying where it stands. + # language: python (not system) so pre-commit builds an env holding the two + # deps — the CI lint job installs pre-commit alone, never the project, so a + # system hook has no jsonschema. Kept in sync with pyproject's [dev] extra, + # which is what the pytest job uses. + - id: validate-patch-status + name: patch upstream status (schema + index cross-check) + entry: python scripts/validate-patch-status.py + language: python + additional_dependencies: ["jsonschema>=4.18", "PyYAML>=6"] + files: ^deploy/docker/(patch\.upstream\.status\.yaml|patches/) + pass_filenames: false + # Refuse a commit whose git email is a machine-generated local hostname # (git's fallback when user.email is unset), so an internal build-host # name can't leak into permanent public history. Skipped in CI. diff --git a/.yamllint.yaml b/.yamllint.yaml new file mode 100644 index 00000000..cfe7da0e --- /dev/null +++ b/.yamllint.yaml @@ -0,0 +1,42 @@ +# Style gate for the hand-written YAML under deploy/docker/ (the patch status +# records and their index). check-yaml already answers "does this parse"; this +# answers "does it say what its author meant". +# +# Scoped by the pre-commit hook's `files:`, not from here — the k8s manifests and +# Helm charts under deploy/operator/ are a much larger body of existing YAML and +# are deliberately left alone for now. +# +# Run by hand over the same set: +# yamllint $(git ls-files 'deploy/docker/**/*.yaml') + +extends: default + +rules: + # The records quote upstream verbatim — issue titles, engine error strings, + # call chains — and those have to stay byte-exact to stay greppable, so the + # limit is set where it catches a runaway line without asking anyone to reflow + # a fact. 80 (the default) is unusable here; the longest line today is 156. + line-length: + max: 160 + allow-non-breakable-words: true # a bare URL should not have to wrap + + # These are single-document files by construction; the schemas have no notion + # of a multi-doc record, so a leading `---` would be noise on all 27 of them. + document-start: disable + + # This one is load-bearing, not cosmetic. In YAML an unquoted ` #` opens a + # comment, so `action: Get #2725 reviewed; ...` silently parses as `Get` and + # the rest of the sentence is discarded. Eleven values were losing text that + # way before this hook went in — every one of them a PR number mid-sentence. + comments: + require-starting-space: true + min-spaces-from-content: 2 + + # Warnings are failures: a hook that prints and exits 0 teaches people to + # ignore it. + truthy: + level: error + braces: + level: error + brackets: + level: error diff --git a/deploy/docker/Dockerfile.vllm b/deploy/docker/Dockerfile.vllm index 92a6ab0e..52bfbd67 100644 --- a/deploy/docker/Dockerfile.vllm +++ b/deploy/docker/Dockerfile.vllm @@ -33,7 +33,9 @@ RUN if [ "${BUILD_AITER}" = "1" ]; then \ # Apply patches/vllm/ (general PD-transport / scheduler fixes) then patches/ # vllm-dsv4/ (DSv4-specific: moriio_dsv4 {hybrid_blocksize, sparse_backend, # noncontig_register} + aiter flydsl). Each patch no-ops if its anchor is absent, -# so a base bump degrades gracefully. legacy/ is upstream on this base, not applied. +# so a base bump degrades gracefully. The retired legacy/ patches have moved to +# patches/archived/vllm-dsv4/ (recorded in patches/archived/patch.archived.yaml) and are +# no longer copied into the image at all. # # Removed on v0.25.1 (were no-op here): the three GLM-5.1 moriio patches # patch_moriio_dsa_write.py, patch_moriio_hetero.py and patch_vllm_moriio_blocksize.py. diff --git a/deploy/docker/patch.upstream.status.md b/deploy/docker/patch.upstream.status.md deleted file mode 100644 index 812c1226..00000000 --- a/deploy/docker/patch.upstream.status.md +++ /dev/null @@ -1,230 +0,0 @@ -# Patch ↔ upstream status - -Every patch under `deploy/docker/patches/`, and where it stands relative to the -project it patches. Kept here so "why do we still carry this?" has one answer -per row, and so a patch that upstream has since merged gets dropped instead of -quietly outliving its reason. - -**Verified with `gh` on 2026-08-01**, except the `patches/sglang_rocm/` section: -2026-08-03 for the host allocator, 2026-08-04 for the staged write-back. -State drifts; re-check before relying on a row. `gh search` -matches titles and bodies, **not diff content**, so "no upstream PR" means "none -found by search", not "none exists" — where a row could be checked by reading -upstream source instead, it says so. - -Column meanings: - -- **ours?** — was the upstream PR opened by a contributor of this repo? - `yes` = us; `no` = a third party; `—` = no PR. -- **PR state** — of the upstream PR named in the same row. - -## sglang — `patches/sglang_dsa/` (baked by `Dockerfile.sglang` and `Dockerfile.sglang.gfx942`, `APPLY_SGLANG_DSA_PATCHES=1`) - -Only patch 01 is baked by both: it is an anchor script, while 02 and 04 are -`--fuzz=0` diffs pinned to v0.5.15.post1 and cannot apply to the gfx942 image's -v0.5.16 base. That image substitutes 02b and 04 at runtime with -`--json-model-override-args '{"index_share_for_mtp_iteration":false}'` and does -not address 02a at all — `patches/sglang_dsa/README.md` carries the reasoning. - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.py` | HIP/aiter paged-MQA sizes its output from DP-padded rows while `lengths` is sized to real rows → `Expected lengths.size(0) == B` | none found | [sglang#33059](https://github.com/sgl-project/sglang/pull/33059) | **yes** (`dorado269`) | OPEN, `REVIEW_REQUIRED` | -| ″ (same bug class, other platform) | — | none found | [sglang#32762](https://github.com/sgl-project/sglang/pull/32762) `[NPU] Fix DSA eager padding mismatch` — our diff is written in its shape | no (`stellaxcpeng`) | OPEN | -| ″ (anchor collision, **not** a fix) | — | — | [sglang#32738](https://github.com/sgl-project/sglang/pull/32738) pads heads for DeepGEMM at the same two aiter call sites; [#31480](https://github.com/sgl-project/sglang/pull/31480) extracts the paged-MQA backend and restructures the `is_aiter()` dispatch | no | both OPEN (re-read 2026-08-03) | -| `sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff` (2a) | `seq_lens.max().item()` is a host sync on a branch only *some* DP ranks take → DP collectives desync → deadlock | none found | none found | — | — | -| `sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff` (2b) | page table has one row per **request**, top-k one per **token** under MTP → `assert page_table.shape[0] == topk_indices.shape[0]` | none found | [sglang#32209](https://github.com/sgl-project/sglang/pull/32209) solves the same row mismatch by **trimming q/top-k**; porting that half here fails at conc=32 and is unresolved | no (`HZY-Wade`) | OPEN, `REVIEW_REQUIRED` | -| `sglang_dsa/draft_cuda_graph_dp_vote.diff` | draft graph/eager choice is per-rank and diverges on the PD decode leg → group deadlock | [sglang#32527](https://github.com/sgl-project/sglang/issues/32527) (independent report, 8× Blackwell) | [sglang#32209](https://github.com/sgl-project/sglang/pull/32209) — same defect, same strategy; **this diff adopts its placement** | no (`HZY-Wade`) | OPEN, `REVIEW_REQUIRED` | - -Prerequisite for the set, applied earlier in the same Dockerfile and **asserted** -by `scripts/apply_sglang_dsa_patches.sh`: - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `sglang/patch_glm52_nextn_quark_exclude.py` | GLM-5.2 MTP `eh_proj` is bf16 but the quark-exclude check probes the bare layer prefix → draft weight-load dies `3072 vs 6144` | none found | [sglang#30265](https://github.com/sgl-project/sglang/pull/30265) `[AMD] Fix GLM-5.2 MTP Quark excludes` — a **superset** (dedicated `GlmMoeDsaForCausalLMNextN`); ours is a narrow backport | no (`wangjiaxin99`) | **MERGED** 2026-07-08 | - -> Our base `v0.5.15.post1` (`0b3bb0c`) predates #30265 — the release line was cut -> without it. **Drop this patch** when the base sglang carries #30265; the anchor -> disappears and the script no-ops. - -Background, already present in the base and **not** patched by us: -[sglang#30378](https://github.com/sgl-project/sglang/pull/30378) / -[#30427](https://github.com/sgl-project/sglang/pull/30427) (MERGED) clamp padded-row -seq_lens **values**; our patch 01 fixes the HIP-side row **count**. -[#30839](https://github.com/sgl-project/sglang/pull/30839) / -[#31083](https://github.com/sgl-project/sglang/pull/31083) (MERGED 2026-07-14) -introduced the guard patch 04 repairs — so that deadlock is a regression in this -baseline, not a legacy wart. -[#32722](https://github.com/sgl-project/sglang/pull/32722) (OPEN) adds a test for -PD + DP-attention + MTP, i.e. **no CI covers this topology today**. - -## sglang PD — `patches/sglang_disagg/` (baked by `Dockerfile.sglang` and `Dockerfile.sglang.gfx942`) - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `sglang_disagg/patch_mooncake_early_send_wait_event.py` | `mooncake/conn.py` never waits on the forward's completion event and the overlap path records none, so chunked prefill hands a non-final chunk to the decode leg while the forward writing those pages is still running → prompts longer than one chunk come back **partially wrong**, with nothing in any log | [sglang#25583](https://github.com/sgl-project/sglang/issues/25583) reports the same corruption shape on GLM-5, but **aggregated** — no PD, no mooncake — so a shared root cause is unestablished; closed **inactive** 2026-07-18, no follow-up | none found | — | — | - -> `prefill.py` already records the completion event this needs; only the `mori` -> backend ever read it, and the patch mirrors what `mori` does. **Drop it** once a -> base sglang synchronizes on that event itself — the script then reports "already -> present" and no-ops. -> -> Unlike the rest of this page, this row was **not** verified with `gh`: the issue -> state was read from the web UI on 2026-08-03, and no upstream PR search was run. -> "none found" here is weaker than elsewhere on the page. - -## sglang ROCm — `patches/sglang_rocm/` (baked by `Dockerfile.sglang`, `Dockerfile.sglang.gfx942`) - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `sglang_rocm/patch_hicache_rocm_host_alloc.py` | hicache allocates host pools with `mmap` + `hipHostRegister`, which on ROCm maps the pages at a device address ≠ the host VA, but the pools hand raw host `data_ptr()`s to GPU kernels via device-side pointer tables → `Memory access fault by GPU node-N on address ` on the first kvd write-back | none found | none found | — | — | -| `sglang_rocm/patch_hicache_rocm_staged_write_back.py` | `pool_host/mla.py` enables the staged write-back JIT on HIP while `DSAIndexerPoolHost` — in the same `HostPoolGroup` for any DSA model — still gates it on `_is_cuda`. The group ANDs the flag, so the controller puts the destination indices on the GPU; the anchor MLA pool then reads its own flag and launches the JIT anyway → `Tensor match failed … device=rocm:0 … allowed options: [cpu, rocm_host]`, scheduler exit −3 on the first write-back | none found | [sglang#28534](https://github.com/sgl-project/sglang/pull/28534) `[AMD] Enable JIT staged HiCache write-back and fix CPU-index crash` — added the HIP enablement and aligned `cache_controller.py`, `memory_pool_host.py`, `pool_host/mha.py`; **never touched `pool_host/mla.py`** | no (`AMD-yanfeiwang`) | **MERGED** 2026-07-09 | - -**`patch_hicache_rocm_host_alloc.py`** — the fault is **gfx950-only so far**: MI300X -(amdgpu 6.14.14, ROCm 7.2.0) measures the two addresses equal, so -`Dockerfile.sglang.gfx942` carries this one preventively rather than to fix a crash. -Don't read its row above as evidence the fault was seen on both arches. - -> Verified on 2026-08-03 by **reading upstream `main` directly** (contents API, -> `pool_host/common.py`): `ALLOC_MEMORY_FUNCS` still overrides only `"npu"` and -> `"musa"`, with no HIP entry — so main is affected too, and this is stronger -> than the usual "no search hit". [sglang#23361](https://github.com/sgl-project/sglang/pull/23361) -> (**MERGED**, MUSA) is the same one-line dispatch override for the same reason -> and is the shape this patch copies. -> -> **No PR of ours has been filed** — it should be. **Drop this patch** when a -> base sglang routes HIP to `alloc_with_pin_memory`; the anchor stops matching -> and the script exits non-zero, so the drop is not silent. Note -> [#32503](https://github.com/sgl-project/sglang/pull/32503) / -> [#32792](https://github.com/sgl-project/sglang/pull/32792) (OPEN, Intel XPU -> HiCache) touch this same dict — expect an anchor conflict, not a fix. - -**`patch_hicache_rocm_staged_write_back.py`** — **not** preventive: without it the -v0.5.16 gfx942 base kills the prefill scheduler on the first reused prefix, so that -image needs it to run kvd at all. A **no-op on the mi35x image**, which has nothing to -fix — every `can_use_write_back_jit` gate at v0.5.15.post1 is still `_is_cuda` (MHA, -MLA, both V4 pools and `DSAIndexerPoolHost`, all in `memory_pool_host.py` before -MLA/MHA moved into `pool_host/`), and `_is_hip` appears only in the kernel import -guard, so the group's AND and its anchor agree on False. #28534 introduced the -disagreement after that tag; the absent `pool_host/mla.py` is only how the script -notices. Both `_is_cuda or _is_hip` in `pool_host/mla.py` and the CUDA-only gates on -the other pools are still on upstream `main` (read from the raw files, so stronger -than a search miss), i.e. **main is affected** — and #28534's own reasoning argues for -the opposite repair, teaching the remaining pools the JIT rather than gating MLA down, -so expect upstream to close this differently than we did. - -> **That repair is already in flight (checked 2026-08-04):** -> [sglang#30350](https://github.com/sgl-project/sglang/pull/30350) `Add HiCache JIT -> test and benchmark for ROCm/HIP CI support` (**OPEN**, `Emmanuel0612`) adds -> `_is_cuda_alike = _is_cuda or _is_hip` and flips exactly the three CUDA-only gates -> (`DSAIndexerPoolHost`, `DeepSeekV4PagedHostPool`, `DeepSeekV4StateHostPool`), so the -> group AND stops reading False on ROCm — **including the V4 stack our patch does not -> cover**. It also teaches `staged_write_back.cuh` to accept kDLROCM/kDLROCMHost (the -> TensorMatcher check that emits our crash) and adds an AMD CI lane for the HiCache -> JIT; the author reports 47/47 on MI355X. Stalled rather than rejected: amd-bot -> called its AMD suites green on 07-09 alongside a merge conflict, conflicts were -> cleared 07-13, nothing since 07-16, and #28534 landed in between. **Our leverage is -> a gfx942 reproduction on that thread, not a competing PR** — it has no MI300X -> datapoint. -> -> **Anchor drift cannot be the drop signal.** #30350 never touches -> `pool_host/mla.py`, so our anchor would keep matching and the patch would keep -> applying on top of the fix — not a crash, since both gates read False again, but a -> silent forfeit of the staged kernel #30350 enables. `check_group_still_poisoned()` -> checks the *precondition* instead: once `DSAIndexerPoolHost` stops gating on -> `_is_cuda` alone, the script refuses and exits 1 telling the operator to drop it. -> -> **Exercised 2026-08-04** against the v0.5.16 *and* current-`main` copies of both -> files, on throwaway trees. Stock: applies, exit 0. Re-run: "already applied", exit -> 0. #30350 simulated by flipping the three CUDA-only gates: refuses, exit 1, with -> `mla.py` byte-identical to pristine. `pool_host/mla.py` removed (the v0.5.15.post1 -> shape): tolerated, exit 0. Anchor or pool renamed: exit 1. -> -> Scope: `DSAIndexerPoolHost` is not the only CUDA-only member that can poison the -> group's AND, and `build_deepseek_v4_hicache_stack` puts `DeepSeekV4PagedHostPool` -> in a group anchored by `LogicalHostPool` (flag unconditionally True). Expect the -> same crash on a V4 hicache stack on gfx942; **this patch gates `mla.py` only**. No -> V4 stack runs on this branch, so that gate would be untested — the script's SCOPE -> section records it for whoever gets there. - -## Mooncake C++ — `patches/mooncake_cpp/` (built by `Dockerfile.sglang`, `Dockerfile.vllm`, `Dockerfile.atom`) - -Pinned to Mooncake `main @ 747003c`; `git apply` fails loudly on ref drift. - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `mooncake_cpp/rdma_auto_chunk_mr_2017.diff` | buffers over the device `max_mr_size` are silently truncated by `ibv_reg_mr` while `BufferDesc.length` advertises the full size → `IBV_WC_REM_ACCESS_ERR` past the boundary | [Mooncake#2017](https://github.com/kvcache-ai/Mooncake/issues/2017) | [Mooncake#2644](https://github.com/kvcache-ai/Mooncake/pull/2644) | **yes** (`jiejingzhangamd`) | **MERGED** 2026-07-28 | -| `mooncake_cpp/rdma_transport_dmabuf_cmake.diff` | `USE_HIP_DMABUF` is defined only on the `transfer_engine` target, so the `ibv_reg_dmabuf_mr` branch compiles **out** of `rdma_transport` — where it is actually called → GPU buffers fall back to bare `ibv_reg_mr`, which cannot pin VRAM without `ib_peer_mem` | none found | none found | — | — | -| `mooncake_cpp/transfer_engine_impl.diff` | `installTransport("hip")` runs unconditionally, so GPU buffers become intra-node HIP IPC segments a cross-node peer cannot open (`Corrupted segment descriptor hipbuffer`) | none found | none found | — | — | - -> #2644 is merged upstream but **not** in the pinned `747003c` tree, so the local -> diff is still applied. Drop it when the pin advances past the merge. - -## vLLM — `patches/vllm/` (baked by `Dockerfile.vllm`) - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `vllm/patch_defer_kv_register.py` | registering the Mooncake KV pool before warmup trips a decode-boot crash at high util (`compile_or_warm_up_model` returns None → `AttributeError: 'NoneType' … language_model`); defer to the end of warmup | none found | none found | — | — | -| `vllm/patch_moriio_pagelen.py` | MoRIIO MLA derives per-block transfer size/stride from tensor **shape**, so block-scaled fp8 MLA + DSA transfers the wrong geometry → PD output is wrong while direct prefill is correct | none found | none found | — | — | -| `vllm/patch_moriio_write.py` | in WRITE (push) mode the decode addresses **itself** (`is_producer=True`), the consumer handler asserts, the notify thread dies and the request hangs | `AMD-AGI/Infera#67` | fixed on vLLM `main` / `v0.22.1rc0` **source**, but no AMD ROCm image ships it | no | n/a (source-only) | -| `vllm/patch_sched_guard.py` | decode EngineCore dies on `assert req_id in self.requests` when a KV-xfer-finished event arrives for an already-removed request | `AMD-AGI/Infera#69` | none found | — | — | -| `vllm/patch_vllm_mooncake_blocksize.py` | backends that force a kernel block size of 1 make the connector index logical pages at kernel granularity → RDMA moves empty rows, decode attends over zeros | none found | none found | — | — | -| `vllm/patch_vllm_mooncake_prom_metrics.py` | `MooncakeConnector` lacks `build_prom_metrics`, so `MultiKVConnectorPromMetrics.observe` asserts and kills the engine under `MultiConnector` | none found | internal PR #178 (the sibling fix for `InferaKvdConnector`) | **yes** | **MERGED** | - -> The `AMD-AGI/Infera#NN` references above are quoted verbatim from the patches' -> own docstrings, which is the citation form this repo already uses. They resolve -> against an internal tracker, so treat a `404` as expected rather than as a -> stale number. - -## vLLM DSv4 — `patches/vllm-dsv4/` (baked by `Dockerfile.vllm`) - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.py` | aiter 0.1.16's `fx.ptrtoint` rejects flydsl memrefs → `MLIRError` in the 2-stage MoE GEMM (Kimi-K2.6 int4 W4A16) | none found | none found | — | — | -| `vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.py` | a global `block_size` equality check kills the prefill worker at KV registration, though DSv4 registers per-layer caches with different block sizes and offsets already use the per-layer map | none found | none found | — | — | -| `vllm-dsv4/patch_moriio_dsv4_noncontig_register.py` | `register_torch_tensor` rejects DSv4's 576B-aligned non-contiguous fp8_ds_mla KV view, and `.contiguous()` would detach from the buffer the forward writes into | none found | none found | — | — | -| `vllm-dsv4/patch_moriio_dsv4_sparse_backend.py` | the generic ROCm selector returns `ROCM_AITER_MLA_SPARSE` (no `fp8_ds_mla`) and raises, killing the prefill worker, though `backend_name` is only a P/D handshake tag | none found | none found | — | — | - -### `patches/vllm-dsv4/legacy/` — archival, **not executed by any Dockerfile** - -Kept for provenance; both are no-ops on the current verified stack -(vLLM `0.23.1rc1.dev748`, `amd-aiter 0.1.16.post2`). - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `vllm-dsv4/legacy/patch_dsv4_aiter_moe.py` | DSv4 MXFP4 MoE gate-mode plumbing, before aiter carried it | none found | [aiter#3123](https://github.com/ROCm/aiter/pull/3123) `[MoE] Align Swiglu MXFP4 fused quant paths` | no (`XiaobingSuper`) | **MERGED** 2026-05-12 | -| `vllm-dsv4/legacy/patch_dsv4_mhc_aiter.py` + `.diff` | `mhc_pre_gemm_sqrsum_kernel` store race → EngineCore dies | none found | [aiter#3033](https://github.com/ROCm/aiter/pull/3033) `Fix sqrsum store race condition` | no (`kkHuang-amd`) | **MERGED** 2026-05-06 | - -## ATOM — `patches/atom/` (baked by `Dockerfile.atom`) - -ATOM is an internal engine; there is no public upstream to file against. - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `atom/patch_gdn_pd_state_transfer.py` | hybrid GDN models never transfer the GatedDeltaNet recurrent state in PD, so decode starts from a zero state and cannot recall prompt context (5/5 mixed vs 0/5 PD) | n/a — internal engine | n/a | — | — | -| `atom/patch_minimax_m2_qknorm_rope.py` | stock `minimax_m2.py` fails to load: `get_rope()` rejects `dtype=`, and the TP fused QK-norm kernel needs a batch guard | n/a | n/a | — | — | -| `atom/patch_mooncake_consumer_slot.py` | `UnboundLocalError: consumer_staging_pool_idx` crashes the decode worker on the first PD request for models with slot state but no `slot_regions` | n/a | n/a | — | — | - -## hipFile — `patches/hipfile_async/` (baked by `Dockerfile.vllm`) - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `hipfile_async/hipfile_async.patch` (+ `file_async_fragment.py`, `patch_hipfile_async.sh`) | the ROCm hipFile binding exposes no async API; adds `write_async`/`read_async`, `Stream`, `supports_async()`. Cython stack-locals passed as `&local` die before the driver dereferences them → `bytes_done` reads 0 and intermittent `HipFileException 5022`, so the wrapper calls `libhipfile.so` via ctypes with heap-allocated slots | none found | none found | — | — | - -## Not patches - -Every file under `patches/` is covered above except these, which carry no fix of -their own: `mooncake_cpp/apply_mooncake_cpp_patches.sh` (applies the three -Mooncake diffs), `sglang_dsa/README.md`, `sglang_disagg/README.md` and -`vllm-dsv4/legacy/README.md`. - -## Maintenance - -A row is ready to delete when its upstream PR is **merged and present in the -pinned base**. Merged-but-not-in-base still needs the local patch — sglang#30265 -and Mooncake#2644 are exactly that case. The staged write-back row is the -exception: its `MERGED` PR (sglang#28534) is what *introduced* the defect, so that -patch drops on sglang#30350 instead. - -When adding a patch, add its row here in the same commit, and put the full -argument — evidence, alternatives, how it differs from our own upstream PR — in -the patch's own header. This table is the index, not the record. diff --git a/deploy/docker/patch.upstream.status.yaml b/deploy/docker/patch.upstream.status.yaml new file mode 100644 index 00000000..3e3a75bc --- /dev/null +++ b/deploy/docker/patch.upstream.status.yaml @@ -0,0 +1,550 @@ +# yaml-language-server: $schema=patches/_schema/patch.upstream.index.schema.json +# +# Every third-party patch this repo carries, grouped by the library it patches. +# +# This file is an INDEX. It holds only what you need to decide whether to open the +# per-patch record: which engines see the patch, why it is still here, and how you will +# find out when it can go. The argument — root cause, upstream state, evidence, call +# chain, verification — lives in the referenced .upstream.status.yaml, and the +# patch's own header carries the code-level detail. +# +# Validated by scripts/validate-patch-status.py against +# patches/_schema/patch.upstream.index.schema.json, which also cross-checks that every +# patch on disk has a record and that the totals below match. A patch added without a +# record fails the lint gate. +# +# Replaces patch.upstream.status.md (removed in the same change). +schema_version: 1 +status_updated: 2026-08-05 + +verification_note: >- + Every row was re-established on 2026-08-05: patch header read, `gh` queried for the + live state of each PR and issue, and for the rows where a search found nothing, upstream + `main` read directly through the contents API. What is NOT covered: no image was built + or inspected, so the four rows whose effect depends on what the pinned base already + carries are marked `unverified` in their records rather than guessed at. Treat any row + whose status_updated has drifted far from today as a prompt to re-check, not as fact. + +pinned_bases: + - surface: Dockerfile.sglang + library: sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + image: + tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: >- + gfx950 / MI355X. On release/v0.5.15, not main — so upstream main fixes do not arrive + by a patch-level bump. release/v0.5.17 already exists upstream. + - surface: Dockerfile.sglang.gfx942 + library: sglang + ref: v0.5.16 + commit: fdebc938f7f4 + pinned_ref_on_main: false + image: + tag: lmsysorg/sglang:v0.5.16-rocm720-mi30x + digest: null + digest_source: unresolved + note: >- + gfx942 / MI325X. On release/v0.5.16. The two sglang bases disagree about which + patches they need, which is why several records carry two target versions. + - surface: Dockerfile.vllm + library: vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + image: + tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: >- + The only digest-pinned base in the tree. vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + - surface: Dockerfile.atom + library: atom + ref: atom0.1.4 + commit: null + pinned_ref_on_main: null + image: + tag: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612 + digest: null + digest_source: unresolved + note: Internal engine — no public repo, so there is no commit and no notion of main. + - surface: all images (MOONCAKE_GIT_REF) + library: mooncake + ref: main @ 747003c + commit: 747003c058015c4077a266e7ccd7549bbc9baede + pinned_ref_on_main: true + image: null + note: >- + Rebuilt in place in every engine image. The only base pinned to a real upstream commit, + and the only one on main. It is 2026-06-26 and two of our own merged fixes are newer, + so advancing it retires two patches at once. + - surface: Dockerfile.vllm (BUILD_AITER=1) + library: aiter + ref: v0.1.16.post1 + commit: null + pinned_ref_on_main: null + image: null + note: >- + Built in-container from a tag. The flydsl patch was verified against 0.1.16.post2, one + patch level above this — they have drifted. + - surface: Dockerfile.vllm (BUILD_HIPFILE=1) + library: hipfile + ref: unpinned-default-branch + commit: null + pinned_ref_on_main: true + image: null + note: >- + NOT PINNED. build_hipfile.sh clones HEAD of ROCm/rocm-systems, so image content is a + function of the build DATE. Two images from the same infera commit built either side of + 2026-07-16 contain different hipfile code. Worth fixing independently of any patch. + +libraries: + sglang: + repo: sgl-project/sglang + note: >- + Two bases, and they need different patch sets. The two --fuzz=0 context diffs apply only + to v0.5.15.post1; the gfx942 image substitutes a runtime flag for one of them and does + not address the other. + patches: + - patch: deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.py + record: deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.upstream.status.yaml + component: quantization + engines: [sglang] + status: drop-candidate + alive_because: >- + sglang#30265 merged 2026-07-08 but release/v0.5.15 was cut without it + drop_signal: base-version-only + ours_upstream_pr: null + summary: >- + GLM-5.2's bf16 MTP layer is built as MXFP4 because the quark exclude probe tests the + bare layer prefix instead of the eh_proj submodule, so the draft weight-load asserts. + A narrow one-line backport of #30265, applied to the mi35x base only. WARNING: the + anchor does NOT disappear on a fixed base — #30265 left the matched string on main — + so decide the drop from the base version, not from the build log. + - patch: deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py + record: deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml + component: disaggregation + engines: [sglang] + status: carry-no-upstream-fix + alive_because: not submitted upstream; main confirmed affected by source read + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + SILENT CORRECTNESS BUG. Chunked prefill over mooncake PD RDMA-reads KV pages while the + forward is still writing them, because the barrier prefill.py records is only ever read + by mori/conn.py — mooncake/conn.py has no wait_event at all. Long prompts come back + partially wrong with no log line. Needle retrieval 5/9 -> 9/9. + - patch: deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.py + record: deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.upstream.status.yaml + component: dsa + engines: [sglang] + status: carry-upstream-pr-open + alive_because: our sglang#33059 is open and unreviewed; both bases are release-branch pins + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: sgl-project/sglang#33059 + summary: >- + The aiter/HIP paged-MQA branch sizes logits and lengths from different row counts, which + diverge in BOTH directions under DP-attention. Reconciles to min(real, padded). The only + sglang_dsa patch that runs on both bases, because it is an anchor script rather than a + --fuzz=0 diff. + - patch: deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff + record: deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml + component: dsa + engines: [sglang] + status: carry-no-upstream-fix + alive_because: >- + no upstream PR for the DP host-sync deadlock at all; the one PR touching the row + mismatch takes an approach that fails here at concurrency 32 + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + Two independent defects. A blocking device-to-host sync on a DP-divergent branch + deadlocks the group; and the decode page table is per-request while top-k is per-token + under MTP, which asserts. gfx950 only. + - patch: deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.diff + record: deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.upstream.status.yaml + component: speculative-decoding + engines: [sglang] + status: carry-upstream-pr-open + alive_because: >- + sglang#32209 carries this exact fix and is unreviewed; we deliberately opened no + competing PR + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + The draft-CUDA-graph choice is made per rank from two rank-dependent terms, so the DP + group deadlocks on the first PD request. Made a group decision via one extra int64 slot + in the existing all-gather. A regression from merged #30839/#31083, independently + reported upstream as #32527 on Blackwell — not ROCm-specific. + - patch: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py + record: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml + component: hicache + engines: [sglang] + status: carry-no-upstream-fix + alive_because: no upstream issue and no PR, ours included; main confirmed affected + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + On ROCm, hipHostRegister maps pages at a DIFFERENT device address than the host VA, but + hicache stores host VAs in a device-side pointer table a kernel dereferences — GPU + memory access fault at the host address. Routes ROCm to pin_memory, exactly as the merged + MUSA entry (#23361) does. The clearest missing-PR gap in the tree. + - patch: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.py + record: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.upstream.status.yaml + component: hicache + engines: [sglang] + status: carry-upstream-pr-open + alive_because: >- + drops on sglang#30350 (OPEN, CHANGES_REQUESTED) — note its MERGED sibling #28534 is what + INTRODUCED the defect + drop_signal: precondition-check-refuses + ours_upstream_pr: null + summary: >- + Two gates decide one thing and disagree on ROCm: the MLA pool opts HIP into the staged + JIT while the pools sharing its group do not, so the controller puts indices on the GPU + and the kernel demands them on the host. Kills the scheduler on the first write-back. + Uses a PRECONDITION CHECK rather than an anchor check, because #30350 never touches our + anchor and would let the patch silently outlive its reason. + + vllm: + repo: vllm-project/vllm + note: >- + One digest-pinned base. Patches are baked by Dockerfile.vllm and, for patches/vllm/, also + applied at container start by the overlay. Note the patch loop swallows a patch's + sys.exit(1) into "skipped", so a real failure and a benign skip look identical in the log. + patches: + - patch: deploy/docker/patches/vllm/patch_moriio_pagelen.py + record: deploy/docker/patches/vllm/patch_moriio_pagelen.upstream.status.yaml + component: moriio + engines: [vllm] + status: carry-no-upstream-fix + alive_because: no upstream issue and no PR; main confirmed affected by source read + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + SILENT CORRECTNESS BUG. MoRIIO derives MLA per-block transfer size and stride from tensor + SHAPE rather than spec.page_size_bytes, which disagrees two independent ways: DSv4's + fp8_ds_mla page is padded (the dropped tail holds the scale), and GLM-5.1 pages 16x + larger than its kernel block. PD output wrong while prefill-direct is correct. Localised + by differential against Mooncake on the same nodes. + - patch: deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.py + record: deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.upstream.status.yaml + component: mooncake-connector + engines: [vllm] + status: carry-no-upstream-fix + alive_because: no upstream issue and no PR; main confirmed affected at line 1678 + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + `conv, _ = cache_or_caches` treats Mamba2's two-tensor shape as the contract, so Kimi-K3's + KDA linear attention raises "too many values to unpack" on every rank during KV + registration. Three-line fix, no competing PR. Existed since 2026-08-02 and was never + indexed on the old status page — the gap this system closes. + - patch: deploy/docker/patches/vllm/patch_sched_guard.py + record: deploy/docker/patches/vllm/patch_sched_guard.upstream.status.yaml + component: scheduler + engines: [vllm] + status: carry-no-upstream-fix + alive_because: >- + three open upstream issues describe this exact assert and none has a PR; ours is a + symptom guard we should not upstream as-is + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + Skips a KV-xfer-finished event for an already-removed request instead of asserting, so the + decode EngineCore survives concurrent PD load. Deliberately NOT a fix — the racing request + still loses its transfer — so apply for throughput runs and leave it OFF for correctness + runs, which should surface the race. + - patch: deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.py + record: deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.upstream.status.yaml + component: mooncake-connector + engines: [vllm] + status: carry-upstream-pr-open + alive_because: "vllm#50374 open and unreviewed; #43836 would also resolve it" + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + MooncakeConnector has no build_prom_metrics, so under MultiConnector the Prometheus + registration assert kills the engine on the first request — i.e. exactly the PD + kvd-L3 + recipe with stats on. Patched at build time, not run time, because the runtime approach + loses an import-order race. Our internal #178 fixed only the sibling connector, which + fixed nothing. + - patch: deploy/docker/patches/vllm/patch_defer_kv_register.py + record: deploy/docker/patches/vllm/patch_defer_kv_register.upstream.status.yaml + component: mooncake-connector + engines: [vllm] + status: carry-no-upstream-fix + alive_because: >- + nothing upstream; also the hardest to upstream, because the crash it avoids was routed + around rather than diagnosed + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + Defers Mooncake KV registration to the end of warmup, which avoids a high-util boot crash + where one TP worker's compile_or_warm_up_model returns None. A workaround with an + unexplained mechanism — the weakest-evidence record here, and it says so. + - patch: deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.py + record: deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.upstream.status.yaml + component: mooncake-connector + engines: [vllm] + status: drop-candidate + alive_because: >- + vllm#46807 MERGED 2026-06-30 with the identical repair; only unconfirmed whether the + pinned base carries it + drop_signal: self-guard-marker-skips + ours_upstream_pr: vllm-project/vllm#46334 + summary: >- + THE NEXT ONE TO DELETE. Registers KV at logical-page granularity for backends that force + kernel block size 1. Our #46334 proposed this first and was closed as superseded by + #46807. The guard marker is the name UPSTREAM added, so on a fixed base it self-skips — + safe, but "already patched" in the log reads like success rather than redundancy. + - patch: deploy/docker/patches/vllm/patch_moriio_write.py + record: deploy/docker/patches/vllm/patch_moriio_write.upstream.status.yaml + component: moriio + engines: [vllm] + status: drop-candidate + alive_because: >- + fixed on upstream source (main and v0.22.1rc0); written when no shipped ROCm image had + it, and the base has moved twice since without anyone re-checking + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + One token: in WRITE mode the decode addressed itself (is_producer=True), the consumer + handler asserted, the notify thread died and the request hung at HTTP 000. Main has + is_producer=False at both call sites; v0.25.1 postdates every image the header lists as + affected, so this is probably already redundant. + + # ---- DSv4-specific, from patches/vllm-dsv4/ (Dockerfile.vllm's second loop) ---- + # Grouped here rather than under a directory-shaped key, because the library they patch + # is vLLM. All three were verified on vllm 0.23.x and none was re-verified on the v0.25.1 + # base — the same refactor that retired three sibling patches. This is the single largest + # unverified area in the index. + - patch: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.py + record: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.upstream.status.yaml + component: moriio + engines: [vllm] + status: carry-no-upstream-fix + alive_because: nothing upstream; effect on the current base unverified + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + Drops a global block_size equality check that kills the prefill worker at KV registration, + although DSv4 legitimately registers per-layer caches at different block sizes and the + offset path already consults the per-layer map. + - patch: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.py + record: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.upstream.status.yaml + component: moriio + engines: [vllm] + status: carry-no-upstream-fix + alive_because: nothing upstream; effect on the current base unverified + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + Registers DSv4's 576B-aligned non-contiguous fp8_ds_mla KV view by storage span, as + sglang's mori connector already does. The trap worth knowing: .contiguous() would have + "worked" and silently moved stale KV, since it copies away from the buffer the forward + writes into. + - patch: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.py + record: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.upstream.status.yaml + component: moriio + engines: [vllm] + status: carry-no-upstream-fix + alive_because: nothing upstream; effect on the current base unverified + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + The generic ROCm selector returns a backend that lacks fp8_ds_mla and raises, killing the + prefill worker — for a value used only as a P/D handshake tag. Sets it directly for DSv4 + sparse MLA. The whole fix rests on backend_name still being cosmetic. + + aiter: + repo: ROCm/aiter + note: Built in-container by Dockerfile.vllm from AITER_GIT_REF. + patches: + - patch: deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.py + record: deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.upstream.status.yaml + component: flydsl-moe + engines: [vllm] + status: carry-no-upstream-fix + alive_because: >- + nothing upstream; and verified against aiter 0.1.16.post2 while the Dockerfile now builds + v0.1.16.post1 + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + aiter 0.1.16's fx.ptrtoint rejects the flydsl memrefs vLLM's tensors become, so the + 2-stage MoE GEMM dies with an MLIRError. Reverts gemm1/gemm2 to the memref-friendly + buffer_ops calls. Needed by Kimi-K2.6 int4 W4A16; no-op for DSv4 MXFP4. + + mooncake: + repo: kvcache-ai/Mooncake + note: >- + Rebuilt in place in every engine image from MOONCAKE_GIT_REF and patched with three C++ + diffs. The pin is on main but two of our own merged fixes are newer than it, so ONE pin + advance retires two of these three — sequence them together, because the same advance also + pulls #2682, which is what makes the third patch necessary. + patches: + - patch: deploy/docker/patches/mooncake_cpp/transfer_engine_impl.diff + record: deploy/docker/patches/mooncake_cpp/transfer_engine_impl.upstream.status.yaml + component: transfer-engine + engines: [sglang, vllm, atom] + status: carry-upstream-pr-open + alive_because: our Mooncake#2725 is open and unreviewed + drop_signal: pin-advance-only + ours_upstream_pr: kvcache-ai/Mooncake#2725 + summary: >- + Upstream #2682 installs the HIP (hipIpc) transport unconditionally and prefers it over + RDMA, so cross-node PD dies with hipIpcOpenMemHandle 201. Gates it behind an env var. The + risk here is INVERTED — a build that silently skips the rebuild ships stock #2682 — which + is why every image asserts the gate is present in the .so it will load. + - patch: deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.diff + record: deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.upstream.status.yaml + component: transfer-engine-rdma + engines: [sglang, vllm, atom] + status: drop-candidate + alive_because: our Mooncake#2644 MERGED 2026-07-28; the pin is 2026-06-26 + drop_signal: pin-advance-only + ours_upstream_pr: kvcache-ai/Mooncake#2644 + summary: >- + Buffers larger than the device max_mr_size were silently truncated while BufferDesc kept + advertising the full length, so remote operations past the boundary hit + IBV_WC_REM_ACCESS_ERR. Chunks at registration time and makes the truncation site a hard + error. Ours, merged; only the pin holds it here. + - patch: deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.diff + record: deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.upstream.status.yaml + component: transfer-engine-rdma + engines: [sglang, vllm, atom] + status: drop-candidate + alive_because: Mooncake#2543 MERGED 2026-07-24; the pin is 2026-06-26 + drop_signal: pin-advance-only + ours_upstream_pr: null + summary: >- + Adds dma-buf GPUDirect registration (ibv_reg_dmabuf_mr), which is the ONLY way to pin VRAM + on a fabric without ib_peer_mem — required on crusoe amd-spur. Opt-in via + MOONCAKE_HIP_DMABUF=1, default off. + + atom: + repo: internal + note: >- + ATOM is an internal engine with no public upstream, so "file it upstream" means landing it in + ATOM itself. These three cannot be retired by any external event. + patches: + - patch: deploy/docker/patches/atom/patch_gdn_pd_state_transfer.py + record: deploy/docker/patches/atom/patch_gdn_pd_state_transfer.upstream.status.yaml + component: attention-metadata + engines: [atom] + status: internal-engine + alive_because: internal engine — no public upstream to file against + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + SILENT CORRECTNESS BUG. Hybrid GDN models never transfer the GatedDeltaNet recurrent state + in PD, so decode starts from a zero mamba state and cannot recall anything from the + prompt. Masked because weight-stored world knowledge still answers correctly: 5/5 recall + single-node mixed versus 0/5 in PD. + - patch: deploy/docker/patches/atom/patch_mooncake_consumer_slot.py + record: deploy/docker/patches/atom/patch_mooncake_consumer_slot.upstream.status.yaml + component: mooncake-connector + engines: [atom] + status: internal-engine + alive_because: internal engine — no public upstream to file against + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + consumer_staging_pool_idx is bound only inside the slot_regions branch but read + unconditionally, so models with cache-group slot state and no registered slot_regions + (Qwen3.5 GDN) crash the decode worker with UnboundLocalError on the first PD request. + - patch: deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.py + record: deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.upstream.status.yaml + component: model-definition + engines: [atom] + status: internal-engine + alive_because: internal engine — a base image whose model file and rope API disagree + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + The stock minimax_m2.py does not load — it passes dtype= to a get_rope that does not + accept it — and the TP fused QK-norm kernel needs a batch guard. Makes MiniMax-M2 START; + correct OUTPUT additionally needs ATOM_USE_UNIFIED_ATTN=1 and --block-size 64 at launch. + + hipfile: + repo: ROCm/rocm-systems + note: >- + Built by Dockerfile.vllm from an UNPINNED clone of the default branch, so the target moves on + every build. + patches: + - patch: deploy/docker/patches/hipfile_async/hipfile_async.patch + record: deploy/docker/patches/hipfile_async/hipfile_async.upstream.status.yaml + component: python-bindings + engines: [vllm] + status: drop-candidate + alive_because: >- + ours, MERGED upstream 2026-07-16, and the target is unpinned — so the fix arrives on its + own and this is "not yet deleted" rather than alive + drop_signal: self-guard-marker-skips + ours_upstream_pr: ROCm/rocm-systems#7386 + summary: >- + Adds async stream I/O to the ROCm hipFile Python binding, keeping the driver's + out-parameters on the heap — Cython stack locals passed as &local die before the driver + writes to them, giving bytes_done == 0 and intermittent HipFileException 5022. Upstream + #7386 is the same feature on the same four files by the same author, with a better + mechanism (a Cython AsyncIOHandle owning the slots). Delete ours. + +archived_record: deploy/docker/patches/archived/patch.archived.yaml + +totals: + active_patches: 25 + records: 25 + archived_patches: 6 + by_status: + carry-no-upstream-fix: 11 + carry-upstream-pr-open: 5 + drop-candidate: 6 + internal-engine: 3 + +not_patches: + - path: deploy/docker/patches/mooncake_cpp/apply_mooncake_cpp_patches.sh + role: >- + Driver that applies the three Mooncake C++ diffs, with a `git apply --reverse --check` + so an already-fixed pin is an idempotent skip. + - path: deploy/docker/patches/sglang_dsa/README.md + role: Directory notes for the sglang DSA patch set. + - path: deploy/docker/patches/sglang_disagg/README.md + role: Directory notes for the sglang disaggregation patch. + - path: deploy/docker/patches/archived/vllm-dsv4/README.md + role: Original notes for the archived legacy DSv4 patches, kept as provenance. + +maintenance: + drop_rule: >- + A patch is ready to delete when its upstream fix is merged AND present in the pinned base. + Merged-but-not-in-base still needs the local patch — sglang#30265, Mooncake#2644/#2543 and + rocm-systems#7386 are all that case. Two rows do not fit the rule's shape at all: the staged + write-back drops on sglang#30350 because its MERGED sibling #28534 is what INTRODUCED the + defect, and the Mooncake diffs drop on the pin advancing rather than on any merge. + add_rule: >- + A new patch needs a .upstream.status.yaml beside it and an entry here, or + scripts/validate-patch-status.py fails the lint gate. If a file under patches/ carries no fix + of its own, list it under not_patches instead. + warnings: + - >- + Six patches cannot announce their own obsolescence. Anything with drop_signal + base-version-only or pin-advance-only keeps applying cleanly after upstream has fixed the + problem, so only the pinned version tells you. Review those on every base bump. + - >- + A self-guard marker that keys on text UPSTREAM added — not on text we add — prints + "already patched" on a fixed base. That is safe but indistinguishable from success in the + build log. patch_vllm_mooncake_blocksize.py and patch_moriio_write.py are both this shape. + - >- + Dockerfile.vllm's patch loop swallows sys.exit(1) into "[vllm-patch] skipped", so a genuine + failure and a benign no-op look the same. Read the per-layer build log when bumping the base. + - >- + Four patches carry silent correctness bugs — wrong output, no crash, no log line: + patch_mooncake_early_send_wait_event.py, patch_moriio_pagelen.py, + patch_gdn_pd_state_transfer.py, and patch_vllm_mooncake_blocksize.py. Never assume a smoke + test covers these; three of them pass one. + - >- + HIPFILE_GIT_REF is unpinned, so images from the same infera commit can contain different + hipfile code depending on the build date. That is a reproducibility problem in its own right. diff --git a/deploy/docker/patches/_schema/patch.archived.schema.json b/deploy/docker/patches/_schema/patch.archived.schema.json new file mode 100644 index 00000000..f7a142bd --- /dev/null +++ b/deploy/docker/patches/_schema/patch.archived.schema.json @@ -0,0 +1,199 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/AMD-AGI/infera/deploy/docker/patches/_schema/patch.archived.schema.json", + "title": "Infera retired patches", + "description": "deploy/docker/patches/archived/patch.archived.yaml — one list entry per patch we no longer need. Retired patches get no per-patch record and are not applied by any surface; this file is the whole memory of them, so each entry has to stand alone.", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "status_updated", "patches"], + "properties": { + "schema_version": { "const": 1 }, + "status_updated": { "$ref": "#/$defs/date" }, + "note": { "type": "string", "minLength": 1 }, + + "patches": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "source", + "library", + "engines_affected", + "retired_on", + "retired_reason", + "problem", + "resolution" + ], + "properties": { + "name": { + "type": "string", + "minLength": 1, + "description": "Original patch file name, without directory." + }, + "source": { + "type": "object", + "additionalProperties": false, + "required": ["original_path", "current_path"], + "properties": { + "original_path": { + "$ref": "#/$defs/repo_path", + "description": "Where it lived while it was applied." + }, + "current_path": { + "anyOf": [ + { "$ref": "#/$defs/repo_path" }, + { "const": "deleted" } + ], + "description": "Where the file sits now, or 'deleted' when it was removed before this record existed — in which case last_commit_with_file must be set so it can still be recovered." + }, + "extra_files": { + "type": "array", + "items": { "$ref": "#/$defs/repo_path" } + }, + "last_commit_with_file": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "type": "null" }], + "description": "Required in spirit when current_path is 'deleted': the commit to recover the file from." + } + } + }, + "library": { "$ref": "#/$defs/library" }, + "component": { "type": ["string", "null"] }, + "engines_affected": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { "$ref": "#/$defs/engine" }, + "description": "Which engines were failing before this patch existed." + }, + "born": { + "type": ["object", "null"], + "additionalProperties": false, + "required": ["date", "commit"], + "properties": { + "date": { "$ref": "#/$defs/date" }, + "commit": { "$ref": "#/$defs/commit" }, + "subject": { "type": "string", "minLength": 1 } + } + }, + "retired_on": { + "type": "object", + "additionalProperties": false, + "required": ["date", "infera_commit"], + "properties": { + "date": { "$ref": "#/$defs/date" }, + "infera_commit": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "const": "pending" }], + "description": "The infera commit that stopped applying it. 'pending' while the retiring change is still in review." + }, + "subject": { "type": "string", "minLength": 1 } + } + }, + "retired_reason": { + "enum": [ + "upstream-fixed-and-in-base", + "upstream-fixed-differently", + "base-refactor-made-anchor-obsolete", + "superseded-by-local-patch", + "never-reproduced-again", + "wrong-diagnosis" + ] + }, + "upstream_fix": { + "type": ["object", "null"], + "additionalProperties": false, + "required": ["ref", "url", "state"], + "properties": { + "ref": { "$ref": "#/$defs/gh_ref" }, + "url": { "$ref": "#/$defs/url" }, + "title": { "type": "string", "minLength": 1 }, + "state": { "enum": ["MERGED", "CLOSED", "OPEN"] }, + "merged_at": { + "anyOf": [{ "$ref": "#/$defs/date" }, { "type": "null" }] + }, + "commit": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "type": "null" }], + "description": "Upstream commit that carries the fix, when known." + }, + "author": { "type": ["string", "null"] }, + "same_approach": { "type": ["boolean", "null"] } + }, + "description": "The upstream change that made this patch unnecessary. null when it was retired for a local reason — a base refactor, or a diagnosis that turned out wrong." + }, + "integrated_by": { + "type": ["object", "null"], + "additionalProperties": false, + "required": ["infera_commit", "how"], + "properties": { + "infera_commit": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "const": "pending" }] + }, + "how": { + "type": "string", + "minLength": 1, + "description": "How the upstream fix arrived here — base image bump, pin advance, engine version bump." + }, + "base_before": { "type": ["string", "null"] }, + "base_after": { "type": ["string", "null"] } + }, + "description": "How and when infera picked up the upstream fix." + }, + "problem": { + "type": "object", + "additionalProperties": false, + "required": ["what", "why", "how", "before_fix", "after_fix", "context"], + "properties": { + "what": { "type": "string", "minLength": 1 }, + "why": { "type": "string", "minLength": 1 }, + "how": { "type": "string", "minLength": 1 }, + "before_fix": { "type": "string", "minLength": 1 }, + "after_fix": { "type": "string", "minLength": 1 }, + "context": { + "type": "string", + "minLength": 1, + "description": "The original failing scenario: model, arch, topology, engine and flags." + }, + "call_chain": { + "type": "array", + "items": { "type": "string", "minLength": 1 } + }, + "symptom_signature": { "type": ["string", "null"] } + } + }, + "resolution": { + "type": "string", + "minLength": 1, + "description": "The one-paragraph account of how this ended, written so it makes sense without the surrounding table." + }, + "reusable_on": { + "type": ["string", "null"], + "description": "Bases where this patch would still apply, for anyone reproducing on an older stack." + }, + "lesson": { + "type": ["string", "null"], + "description": "What this cost us and what would have caught it sooner. Optional, but this is where the value of an archive actually is." + } + } + } + } + }, + + "$defs": { + "date": { + "type": "string", + "format": "date", + "pattern": "^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])$" + }, + "commit": { "type": "string", "pattern": "^[0-9a-f]{7,40}$" }, + "repo_path": { "type": "string", "pattern": "^[A-Za-z0-9._/-]+$" }, + "url": { "type": "string", "format": "uri", "pattern": "^https://" }, + "gh_ref": { + "type": "string", + "pattern": "^([A-Za-z0-9._-]+/[A-Za-z0-9._-]+#[0-9]+|internal#[0-9]+)$" + }, + "engine": { "enum": ["sglang", "vllm", "atom"] }, + "library": { "enum": ["sglang", "vllm", "atom", "mooncake", "mori", "aiter", "hipfile"] } + } +} diff --git a/deploy/docker/patches/_schema/patch.upstream.index.schema.json b/deploy/docker/patches/_schema/patch.upstream.index.schema.json new file mode 100644 index 00000000..157d1b3c --- /dev/null +++ b/deploy/docker/patches/_schema/patch.upstream.index.schema.json @@ -0,0 +1,205 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/AMD-AGI/infera/deploy/docker/patches/_schema/patch.upstream.index.schema.json", + "title": "Infera patch index", + "description": "deploy/docker/patch.upstream.status.yaml — the one-screen view of every patch this repo carries, grouped by the library it patches. Each entry references the per-patch record that holds the argument; this file carries only what you need to decide whether to open it.", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "status_updated", "libraries", "archived_record", "totals"], + "properties": { + "schema_version": { "const": 1 }, + + "status_updated": { + "$ref": "#/$defs/date", + "description": "Date of the last full sweep. Individual entries carry their own date when they were checked separately." + }, + + "verification_note": { + "type": "string", + "minLength": 1, + "description": "What 'verified' means for this sweep, and what it does not cover." + }, + + "pinned_bases": { + "type": "array", + "description": "What the images actually build against. A merged upstream fix that is not in one of these still needs its local patch.", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["surface", "library", "ref", "commit", "pinned_ref_on_main"], + "properties": { + "surface": { "type": "string", "minLength": 1 }, + "library": { "$ref": "#/$defs/library" }, + "ref": { "type": "string", "minLength": 1 }, + "commit": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "type": "null" }] + }, + "pinned_ref_on_main": { "type": ["boolean", "null"] }, + "image": { + "type": ["object", "null"], + "additionalProperties": false, + "required": ["tag", "digest", "digest_source"], + "properties": { + "tag": { "type": "string", "minLength": 1 }, + "digest": { + "anyOf": [{ "$ref": "#/$defs/sha256" }, { "type": "null" }] + }, + "digest_source": { "enum": ["dockerfile-pin", "unresolved"] } + } + }, + "note": { "type": ["string", "null"] } + } + } + }, + + "libraries": { + "type": "object", + "minProperties": 1, + "additionalProperties": false, + "description": "Keyed by the library being patched. Only libraries this repo actually patches appear; add a key when a new dependency needs one.", + "propertyNames": { "$ref": "#/$defs/library" }, + "patternProperties": { + "^(sglang|vllm|atom|mooncake|mori|aiter|hipfile)$": { + "type": "object", + "additionalProperties": false, + "required": ["repo", "patches"], + "properties": { + "repo": { + "anyOf": [{ "$ref": "#/$defs/gh_repo" }, { "const": "internal" }] + }, + "note": { "type": ["string", "null"] }, + "patches": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "patch", + "record", + "component", + "engines", + "status", + "alive_because", + "drop_signal", + "summary" + ], + "properties": { + "patch": { "$ref": "#/$defs/repo_path" }, + "record": { + "$ref": "#/$defs/repo_path", + "description": "The per-patch .upstream.status.yaml. Must exist and must validate." + }, + "component": { "type": ["string", "null"] }, + "engines": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { "$ref": "#/$defs/engine" } + }, + "status": { + "enum": [ + "carry-no-upstream-fix", + "carry-upstream-pr-open", + "drop-candidate", + "internal-engine", + "archival" + ], + "description": "drop-candidate means an upstream fix exists and merged; it is waiting only on a base or pin bump here." + }, + "alive_because": { + "type": "string", + "minLength": 1, + "description": "One clause. The record carries the argument." + }, + "drop_signal": { + "enum": [ + "anchor-drift-fails-loudly", + "self-guard-marker-skips", + "precondition-check-refuses", + "base-version-only", + "pin-advance-only", + "manual-review" + ] + }, + "ours_upstream_pr": { + "anyOf": [{ "$ref": "#/$defs/gh_ref" }, { "type": "null" }], + "description": "Our own upstream PR for this defect, when one exists. null here on a crash-class fix is a gap worth acting on." + }, + "summary": { "type": "string", "minLength": 1 }, + "status_updated": { "$ref": "#/$defs/date" } + } + } + } + } + } + } + }, + + "archived_record": { + "$ref": "#/$defs/repo_path", + "description": "The single file recording retired patches." + }, + + "totals": { + "type": "object", + "additionalProperties": false, + "required": ["active_patches", "records", "archived_patches"], + "properties": { + "active_patches": { "type": "integer", "minimum": 0 }, + "records": { "type": "integer", "minimum": 0 }, + "archived_patches": { "type": "integer", "minimum": 0 }, + "by_status": { + "type": "object", + "additionalProperties": { "type": "integer", "minimum": 0 } + } + }, + "description": "Cross-checked by the validator against what is on disk, so a patch added without a record fails CI." + }, + + "not_patches": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["path", "role"], + "properties": { + "path": { "$ref": "#/$defs/repo_path" }, + "role": { "type": "string", "minLength": 1 } + } + }, + "description": "Files under patches/ that carry no fix of their own — drivers, READMEs, schemas. The validator uses this list to decide what is allowed to have no record." + }, + + "maintenance": { + "type": "object", + "additionalProperties": false, + "properties": { + "drop_rule": { "type": "string", "minLength": 1 }, + "add_rule": { "type": "string", "minLength": 1 }, + "warnings": { + "type": "array", + "items": { "type": "string", "minLength": 1 } + } + } + } + }, + + "$defs": { + "date": { + "type": "string", + "format": "date", + "pattern": "^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])$" + }, + "commit": { "type": "string", "pattern": "^[0-9a-f]{7,40}$" }, + "sha256": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" }, + "repo_path": { "type": "string", "pattern": "^[A-Za-z0-9._/-]+$" }, + "gh_repo": { "type": "string", "pattern": "^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$" }, + "gh_ref": { + "type": "string", + "pattern": "^([A-Za-z0-9._-]+/[A-Za-z0-9._-]+#[0-9]+|internal#[0-9]+)$" + }, + "engine": { "enum": ["sglang", "vllm", "atom"] }, + "library": { "enum": ["sglang", "vllm", "atom", "mooncake", "mori", "aiter", "hipfile"] } + } +} diff --git a/deploy/docker/patches/_schema/patch.upstream.status.schema.json b/deploy/docker/patches/_schema/patch.upstream.status.schema.json new file mode 100644 index 00000000..4bd2563f --- /dev/null +++ b/deploy/docker/patches/_schema/patch.upstream.status.schema.json @@ -0,0 +1,585 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/AMD-AGI/infera/deploy/docker/patches/_schema/patch.upstream.status.schema.json", + "title": "Infera per-patch upstream status", + "description": "One record per patch under deploy/docker/patches/. Lives beside the patch it describes, named .upstream.status.yaml. The patch's own header carries the full argument; this file is the machine-checkable index of where that patch stands against the project it patches.", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "status_updated", + "verified_by", + "patch", + "target", + "upstream_main_affected", + "applies_to", + "alive_because", + "history", + "upstream_issues", + "upstream_prs", + "related_refs", + "problem" + ], + "properties": { + "schema_version": { + "description": "Bumped when this schema makes a breaking change.", + "const": 1 + }, + + "status_updated": { + "$ref": "#/$defs/date", + "description": "Date every upstream claim in this file was last checked. Upstream moves and this file does not, so a stale date is the signal to re-check rather than to trust." + }, + + "verified_by": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "enum": [ + "gh-pr-issue-state", + "gh-search", + "upstream-source-read", + "git-compare", + "local-repro", + "web-ui", + "patch-header-only" + ] + }, + "description": "How the upstream claims below were established. 'gh-search' proves only that a title/body search missed — it does not prove absence. 'upstream-source-read' is stronger: the defect was confirmed present or absent by reading the upstream file." + }, + + "patch": { + "type": "object", + "additionalProperties": false, + "required": ["path", "kind", "applied_by"], + "properties": { + "path": { + "$ref": "#/$defs/repo_path", + "description": "Primary patch file, relative to the git root." + }, + "extra_files": { + "type": "array", + "items": { "$ref": "#/$defs/repo_path" }, + "description": "Further files that are part of the same patch (a fragment appended by a script, a companion .diff)." + }, + "kind": { + "enum": ["python-anchor-script", "shell-script", "context-diff", "git-diff"], + "description": "python-anchor-script and shell-script locate their target at run time and no-op on a miss; context-diff and git-diff fail loudly on ref drift." + }, + "idempotent": { + "type": "boolean", + "description": "Re-running is a no-op. Absent means unknown, which for a patch that runs on every container start is a defect." + }, + "applied_by": { + "type": "array", + "minItems": 1, + "items": { + "enum": [ + "Dockerfile.sglang", + "Dockerfile.sglang.gfx942", + "Dockerfile.vllm", + "Dockerfile.atom", + "deploy/overlay/Dockerfile.payload", + "deploy/overlay/infera-exec", + "deploy/docker/scripts/apply_sglang_dsa_patches.sh", + "deploy/docker/scripts/build_mooncake_rocm.sh", + "deploy/docker/scripts/build_mooncake_sglang.sh", + "deploy/docker/patches/mooncake_cpp/apply_mooncake_cpp_patches.sh", + "none" + ] + }, + "description": "Every surface that applies this patch. 'none' means carried but not wired up, which needs a reason in alive_because.detail." + }, + "opt_in_flag": { + "type": ["string", "null"], + "description": "Build ARG or env var that gates application, when it is not applied unconditionally." + } + } + }, + + "target": { + "type": "object", + "additionalProperties": false, + "required": ["library", "repo", "files", "versions"], + "properties": { + "library": { + "$ref": "#/$defs/library", + "description": "The project whose source this patch edits — not the engine that runs it. A patch to vLLM's bundled MoRIIO connector has library 'vllm', component 'moriio'." + }, + "component": { + "type": ["string", "null"], + "description": "Subsystem inside that library: mooncake, moriio, hicache, dsa, disaggregation, hipfile, flydsl-moe, scheduler." + }, + "repo": { + "anyOf": [ + { "$ref": "#/$defs/gh_repo" }, + { "const": "internal" } + ], + "description": "owner/name on GitHub, or 'internal' for a closed-source engine." + }, + "files": { + "type": "array", + "minItems": 1, + "items": { "type": "string", "minLength": 1 }, + "description": "Paths edited inside the target project, as the patch addresses them." + }, + "versions": { + "type": "array", + "minItems": 1, + "description": "Every pinned version of the target this patch is carried against. Two entries when a patch spans two engine bases — they can disagree about whether the defect is even present.", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["ref", "commit", "pinned_ref_on_main"], + "properties": { + "surface": { + "type": ["string", "null"], + "description": "Which build this version belongs to, when the patch spans more than one." + }, + "ref": { + "type": "string", + "minLength": 1, + "description": "Tag, branch, version string, or 'unpinned-default-branch'." + }, + "commit": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "type": "null" }], + "description": "Commit the ref resolves to. null only when the ref is genuinely unpinned, in which case unpinned_risk must say what that costs." + }, + "pinned_ref_on_main": { + "type": ["boolean", "null"], + "description": "Is the pinned commit an ancestor of the upstream default branch? false means a release branch, so upstream main fixes do not arrive by bumping a patch level. null for internal or unpinned targets." + }, + "release_branch": { + "type": ["string", "null"], + "description": "Upstream branch the pinned ref sits on, when it is not main." + }, + "unpinned_risk": { + "type": ["string", "null"], + "description": "Required in spirit when commit is null: what an unpinned target means for this patch." + }, + "images": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["tag", "digest", "digest_source"], + "properties": { + "tag": { "type": "string", "minLength": 1 }, + "digest": { + "anyOf": [{ "$ref": "#/$defs/sha256" }, { "type": "null" }] + }, + "digest_source": { + "enum": ["dockerfile-pin", "unresolved"], + "description": "'unresolved' means the Dockerfile pins the tag only; the digest is whatever the registry serves at build time." + }, + "note": { "type": ["string", "null"] } + } + } + } + } + } + } + } + }, + + "upstream_main_affected": { + "type": "object", + "additionalProperties": false, + "required": ["value", "evidence"], + "properties": { + "value": { + "type": ["boolean", "null"], + "description": "Does the upstream default branch still carry the defect this patch fixes? null when there is no public upstream to check." + }, + "evidence": { + "type": "string", + "minLength": 1, + "description": "What settles it. Naming the file and line read from main is the standard here; 'no search hit' is not." + } + } + }, + + "applies_to": { + "type": "array", + "minItems": 1, + "description": "Field 4a — what this patch actually does in each engine image it reaches.", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["engine", "surface", "effect", "evidence"], + "properties": { + "engine": { "$ref": "#/$defs/engine" }, + "surface": { + "type": "string", + "minLength": 1, + "description": "Dockerfile or runtime path this row is about." + }, + "effect": { + "enum": ["op", "no-op", "not-applied", "unverified"], + "description": "op = changes bytes and is load-bearing. no-op = reached but changed nothing (target absent, already fixed, or anchor gone). not-applied = never traversed on this surface. unverified = never checked on this surface, which is a gap, not a state." + }, + "required_to_run": { + "type": "boolean", + "description": "true when the engine cannot serve the target workload without this patch." + }, + "evidence": { "type": "string", "minLength": 1 } + } + } + }, + + "alive_because": { + "type": "object", + "additionalProperties": false, + "required": ["reason", "detail", "consumers", "drop_when", "drop_signal"], + "description": "Field 4b — why we still carry this.", + "properties": { + "reason": { + "enum": [ + "no-upstream-pr", + "upstream-pr-open", + "upstream-pr-closed-unfixed", + "merged-not-in-pinned-base", + "merged-not-released", + "released-but-infera-not-bumped", + "upstream-fixed-differently", + "internal-engine-no-upstream", + "local-only-by-design", + "archival" + ] + }, + "detail": { "type": "string", "minLength": 1 }, + "consumers": { + "type": "array", + "minItems": 1, + "items": { "type": "string", "minLength": 1 }, + "description": "Which infera engines, images or workloads still need it. 'none' as the sole entry means the patch is a drop candidate on every surface." + }, + "drop_when": { + "type": "string", + "minLength": 1, + "description": "The concrete condition that retires this patch." + }, + "drop_signal": { + "enum": [ + "anchor-drift-fails-loudly", + "self-guard-marker-skips", + "precondition-check-refuses", + "base-version-only", + "pin-advance-only", + "manual-review" + ], + "description": "How we find out. 'base-version-only' and 'pin-advance-only' mean the patch keeps applying cleanly on a fixed base, so nothing fails and a human has to decide." + }, + "silent_misapply_risk": { + "type": ["string", "null"], + "description": "What goes wrong if this patch outlives its reason and nobody notices. Non-null whenever drop_signal cannot self-announce." + } + } + }, + + "history": { + "type": "object", + "additionalProperties": false, + "required": ["born", "last_modified"], + "properties": { + "born": { "$ref": "#/$defs/commit_event" }, + "last_modified": { + "allOf": [ + { "$ref": "#/$defs/commit_event" }, + { + "type": "object", + "required": ["reason"], + "properties": { + "reason": { + "type": "string", + "minLength": 1, + "description": "Why it changed — not a restatement of the commit subject." + } + } + } + ] + }, + "supersedes": { + "type": "array", + "items": { "type": "string", "minLength": 1 }, + "description": "Earlier patch files this one replaced." + } + } + }, + + "upstream_issues": { + "description": "Field 7 — upstream issues for this defect. null is a legitimate value and must be written explicitly.", + "anyOf": [ + { "type": "null" }, + { + "type": "array", + "minItems": 1, + "items": { "$ref": "#/$defs/upstream_ref" } + } + ] + }, + + "upstream_prs": { + "description": "Field 8 — PRs that fix, or claim to fix, this defect.", + "anyOf": [ + { "type": "null" }, + { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["ref", "url", "title", "state", "author", "ours", "same_approach", "approach_note"], + "properties": { + "ref": { "$ref": "#/$defs/gh_ref" }, + "url": { "$ref": "#/$defs/url" }, + "title": { "type": "string", "minLength": 1 }, + "state": { "$ref": "#/$defs/pr_state" }, + "review_decision": { + "anyOf": [ + { "enum": ["APPROVED", "CHANGES_REQUESTED", "REVIEW_REQUIRED"] }, + { "type": "null" } + ] + }, + "merged_at": { + "anyOf": [{ "$ref": "#/$defs/date" }, { "type": "null" }] + }, + "author": { + "type": ["string", "null"], + "description": "GitHub handle. null means not recorded — better than a placeholder, and note that `ours` can still be answered without it." + }, + "ours": { + "type": "boolean", + "description": "Opened by a contributor of this repo." + }, + "same_approach": { + "type": ["boolean", "null"], + "description": "Does it fix the defect the way our patch does? null when it does not fix this defect at all and is listed for collision or context." + }, + "approach_note": { + "type": "string", + "minLength": 1, + "description": "Why ours differs, or why it does not. 'we needed a local narrow fix and are not carrying the general case' is a valid and useful answer." + }, + "requested_action": { + "type": ["string", "null"], + "description": "What an upstream maintainer explicitly asked for, and who asked." + }, + "in_pinned_base": { + "type": ["boolean", "null"], + "description": "Is this PR present in the version we actually build against? A merged PR that is not in the pinned base still needs the local patch." + } + } + } + } + ] + }, + + "related_refs": { + "description": "Field 9 — PRs and issues that do not decide whether we keep this patch but inform how it should be fixed: the same defect on another platform, a refactor that will move our anchor, a test that proves no CI covers this.", + "anyOf": [ + { "type": "null" }, + { + "type": "array", + "minItems": 1, + "items": { + "allOf": [ + { "$ref": "#/$defs/upstream_ref" }, + { + "type": "object", + "required": ["relevance"], + "properties": { + "relevance": { + "enum": [ + "same-defect-other-platform", + "anchor-collision", + "alternative-repair", + "introduced-the-defect", + "adjacent-site", + "missing-test-coverage", + "shape-we-copied", + "background" + ] + } + } + } + ] + } + } + ] + }, + + "problem": { + "type": "object", + "additionalProperties": false, + "required": ["what", "why", "how", "before_fix", "after_fix", "context"], + "properties": { + "what": { + "type": "string", + "minLength": 1, + "description": "The defect, in one or two sentences." + }, + "why": { + "type": "string", + "minLength": 1, + "description": "Root cause — why the upstream code is wrong, not just what it does." + }, + "how": { + "type": "string", + "minLength": 1, + "description": "What the patch changes, and why that is the minimal correct change." + }, + "before_fix": { + "type": "string", + "minLength": 1, + "description": "Observed failure. Quote the error or the wrong output." + }, + "after_fix": { + "type": "string", + "minLength": 1, + "description": "Observed behaviour with the patch, measured where possible." + }, + "context": { + "type": "string", + "minLength": 1, + "description": "Which model, topology, arch and flags this reproduces under, and where it does not." + }, + "call_chain": { + "type": "array", + "items": { "type": "string", "minLength": 1 }, + "description": "Ordered path from entry point to failure site." + }, + "symptom_signature": { + "type": ["string", "null"], + "description": "The grep-able fingerprint — the exact assert, error string or fault address shape." + }, + "silent": { + "type": "boolean", + "description": "true when the defect produces wrong results rather than a crash. These are the expensive ones." + } + } + }, + + "verification": { + "type": "object", + "additionalProperties": false, + "description": "How the fix was established locally.", + "properties": { + "date": { "$ref": "#/$defs/date" }, + "hardware": { "type": "string", "minLength": 1 }, + "software": { "type": "string", "minLength": 1 }, + "workload": { "type": "string", "minLength": 1 }, + "result": { "type": "string", "minLength": 1 }, + "notes": { "type": ["string", "null"] } + } + }, + + "scope_limits": { + "type": ["string", "null"], + "description": "What this patch deliberately does not cover, so the next person does not assume it does." + }, + + "open_actions": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["action", "owner"], + "properties": { + "action": { "type": "string", "minLength": 1 }, + "owner": { "type": "string", "minLength": 1 }, + "blocked_on": { "type": ["string", "null"] } + } + }, + "description": "What still has to happen upstream or here. An empty list is fine; a missing PR of ours is not." + } + }, + + "$defs": { + "date": { + "type": "string", + "format": "date", + "pattern": "^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])$" + }, + "commit": { + "type": "string", + "pattern": "^[0-9a-f]{7,40}$", + "description": "Lowercase hex, 7 to 40 characters." + }, + "sha256": { + "type": "string", + "pattern": "^sha256:[0-9a-f]{64}$" + }, + "url": { + "type": "string", + "format": "uri", + "pattern": "^https://" + }, + "repo_path": { + "type": "string", + "pattern": "^[A-Za-z0-9._/-]+$", + "description": "Path relative to the git root, no leading slash and no '..'." + }, + "gh_repo": { + "type": "string", + "pattern": "^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$" + }, + "gh_ref": { + "type": "string", + "pattern": "^([A-Za-z0-9._-]+/[A-Za-z0-9._-]+#[0-9]+|internal#[0-9]+)$", + "description": "owner/name#123, or internal#123 for the private tracker." + }, + "engine": { + "enum": ["sglang", "vllm", "atom"] + }, + "library": { + "enum": ["sglang", "vllm", "atom", "mooncake", "mori", "aiter", "hipfile"] + }, + "pr_state": { + "enum": ["OPEN", "MERGED", "CLOSED", "DRAFT", "not-submitted", "source-only"] + }, + "issue_state": { + "enum": ["OPEN", "CLOSED"] + }, + "commit_event": { + "type": "object", + "required": ["date", "commit", "subject"], + "properties": { + "date": { "$ref": "#/$defs/date" }, + "commit": { "$ref": "#/$defs/commit" }, + "subject": { "type": "string", "minLength": 1 }, + "reason": { "type": "string", "minLength": 1 } + }, + "additionalProperties": false + }, + "upstream_ref": { + "type": "object", + "required": ["ref", "url", "kind", "state", "note"], + "properties": { + "ref": { "$ref": "#/$defs/gh_ref" }, + "url": { "$ref": "#/$defs/url" }, + "kind": { "enum": ["pr", "issue"] }, + "title": { "type": "string", "minLength": 1 }, + "state": { + "anyOf": [ + { "$ref": "#/$defs/pr_state" }, + { "$ref": "#/$defs/issue_state" } + ] + }, + "state_reason": { + "type": ["string", "null"], + "description": "Why it closed — 'inactive', 'superseded by #X', 'fixed'." + }, + "author": { "type": ["string", "null"] }, + "relevance": { "type": "string" }, + "note": { + "type": "string", + "minLength": 1, + "description": "What this reference tells us that the title does not." + } + }, + "additionalProperties": false + } + } +} diff --git a/deploy/docker/patches/archived/patch.archived.yaml b/deploy/docker/patches/archived/patch.archived.yaml new file mode 100644 index 00000000..f9f1a687 --- /dev/null +++ b/deploy/docker/patches/archived/patch.archived.yaml @@ -0,0 +1,360 @@ +# yaml-language-server: $schema=../_schema/patch.archived.schema.json +# +# Retired patches. These are applied by nothing and have no per-patch record — this +# file is their entire memory, so each entry is written to stand on its own. +# +# Two kinds of entry: +# * source still present, moved under archived/ — recoverable by reading the file +# * source deleted before this file existed — recoverable from the named commit +# +# Nothing here should be deleted to tidy up. The value of an archive is answering +# "did we already try this, and what happened" two years later. +schema_version: 1 +status_updated: 2026-08-05 +note: >- + Added in the change that introduced the per-patch record system. The four legacy / + already-deleted patches below were previously described in prose in + patch.upstream.status.md, or in the case of the three MoRIIO patches, only in a + Dockerfile comment. Source files were MOVED rather than deleted so this can be + reviewed in a PR without losing anything. + +patches: + - name: patch_dsv4_aiter_moe.py + source: + original_path: deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_aiter_moe.py + current_path: deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_aiter_moe.py + last_commit_with_file: null + library: aiter + component: moe + engines_affected: + - vllm + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + retired_on: + date: 2026-08-05 + infera_commit: pending + subject: "move the legacy DSv4 patches under archived/ and record them" + retired_reason: upstream-fixed-and-in-base + upstream_fix: + ref: ROCm/aiter#3123 + url: https://github.com/ROCm/aiter/pull/3123 + title: "[MoE] Align Swiglu MXFP4 fused quant paths" + state: MERGED + merged_at: 2026-05-12 + commit: null + author: XiaobingSuper + same_approach: true + integrated_by: + infera_commit: pending + how: >- + Arrived by rebuilding from the open-source vllm/vllm-openai-rocm base instead of the + private pre-baked image early bring-up used. No deliberate bump — the fix was already + in the public base. + base_before: private pre-baked DSv4 image (older vLLM / aiter) + base_after: vLLM 0.23.1rc1.dev748+g2dfaae752 with amd-aiter 0.1.16.post2 + problem: + what: >- + DSv4 MXFP4 MoE needed gate-mode plumbing (SWIGLUOAI / activation_interleave) that aiter + did not yet carry, so the fused quant path could not be selected correctly. + why: >- + aiter's rocm_aiter_moe.py had no notion of the interleaved-activation SwiGLU variant DSv4 + uses, so there was no way to ask for it from vLLM. + how: >- + Monkey-patched rocm_aiter_moe.py to add the gate-mode plumbing, with a self-detect that + skips if upstream already has it. + before_fix: DSv4 MXFP4 MoE could not select the correct fused quant path. + after_fix: >- + Upstream aiter carries SWIGLUOAI / activation_interleave and defaults gate_mode="" for + Silu, so the patch self-detects and skips. + context: >- + DeepSeek-V4 MXFP4 MoE on vLLM ROCm during early DSv4 bring-up, debugged against a private + pre-baked image with an older aiter. + call_chain: + - "vLLM MoE layer -> aiter rocm_aiter_moe.py fused quant path selection" + symptom_signature: null + resolution: >- + Obsolete by the time it was first published. Early DSv4 bring-up was debugged against a + private pre-baked image with an older aiter; once the image was rebuilt from the + open-source vllm/vllm-openai-rocm base and markers were checked on real hardware, aiter + already carried the feature via #3123 (merged 2026-05-12). The patch self-detects upstream + and skips, so it had been a no-op on every published image. + reusable_on: >- + An aiter older than #3123 — e.g. the earlier vLLM dev301/dev424 bases. Reference the file + under archived/ to reproduce on such a stack. + lesson: >- + Patches written against a private pre-baked image should be re-verified against the public + base before being carried, not after. Three of the four entries in this file are the same + story. + + - name: patch_dsv4_mhc_aiter.py + source: + original_path: deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_mhc_aiter.py + current_path: deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.py + extra_files: + - deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.diff + last_commit_with_file: null + library: aiter + component: mhc + engines_affected: + - vllm + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + retired_on: + date: 2026-08-05 + infera_commit: pending + subject: "move the legacy DSv4 patches under archived/ and record them" + retired_reason: upstream-fixed-differently + upstream_fix: + ref: ROCm/aiter#3033 + url: https://github.com/ROCm/aiter/pull/3033 + title: Fix sqrsum store race condition + state: MERGED + merged_at: 2026-05-06 + commit: null + author: kkHuang-amd + same_approach: false + integrated_by: + infera_commit: pending + how: >- + Same as the sibling entry — arrived with the move to the open-source base, not by a + deliberate bump. + base_before: private pre-baked DSv4 image (older vLLM / aiter) + base_after: vLLM 0.23.1rc1.dev748+g2dfaae752 with amd-aiter 0.1.16.post2 + problem: + what: >- + A store race in aiter's mhc_pre_gemm_sqrsum_kernel killed the vLLM EngineCore. + why: >- + The kernel's sqrsum store was not correctly synchronized, so concurrent stores raced. + how: >- + Locally, an env-gated backend selection (VLLM_MHC_BACKEND) that routed around the racing + path, hooked on a _mhc_backend() anchor. + before_fix: EngineCore dies during DSv4 MHC execution. + after_fix: >- + Upstream fixed the race itself and made aiter the MHC default — mhc.py now reads + `if HAS_AITER_MHC and hidden % 256 == 0: mhc_pre_aiter` with no env gate at all — so the + local anchor _mhc_backend() no longer exists and the patch auto-skips. + context: >- + DeepSeek-V4 MHC on vLLM ROCm during early bring-up against an older aiter. + call_chain: + - "vLLM DSv4 MHC path -> aiter mhc.py backend selection -> mhc_pre_gemm_sqrsum_kernel" + symptom_signature: EngineCore death during DSv4 MHC execution + resolution: >- + Functionally upstream in a DIFFERENT form, which is why it retired quietly rather than by + the anchor drifting into a conflict. We routed around the race with an env-gated backend + choice; upstream fixed the race (#3033, merged 2026-05-06) and then made aiter the MHC + default outright, so the env gate our anchor keyed on does not exist. The .diff is the + unified-diff companion, never executed, kept as reference. + reusable_on: "An aiter older than #3033, e.g. the earlier vLLM dev301/dev424 bases." + lesson: >- + "Functionally upstream, different form" is the failure mode worth naming: the local patch + does not conflict, it simply stops matching, so nothing announces that it has become dead + weight. This is the same class as the still-active drop candidates whose drop_signal is + base-version-only. + + - name: patch_moriio_dsa_write.py + source: + original_path: deploy/docker/patches/vllm/patch_moriio_dsa_write.py + current_path: deleted + last_commit_with_file: 89c86fb + library: vllm + component: moriio + engines_affected: + - vllm + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + retired_on: + date: 2026-07-23 + infera_commit: "3375773" + subject: Remove three no-op MoRIIO patches dead on the v0.25.1 base + retired_reason: base-refactor-made-anchor-obsolete + upstream_fix: null + integrated_by: + infera_commit: "3375773" + how: >- + vLLM base bump to v0.25.1, which refactored the MoRIIO connector into moriio_layout.py. + base_before: vLLM 0.23.x ROCm + base_after: vllm/vllm-openai-rocm:v0.25.1 + problem: + what: >- + GLM-5.1 over MoRIIO PD needed a full-layer wait_for_save so the KV write completed before + the transfer was considered done. + why: >- + The pre-v0.25.1 MoRIIO connector did not wait for the whole layer to be saved on the DSA + write path, so a transfer could observe a partially written layer. + how: Monkey-patched the connector to wait for the full layer on save. + before_fix: GLM-5.1 MoRIIO PD could transfer a partially-written layer. + after_fix: >- + v0.25.1's moriio_layout.py handles full-layer wait_for_save natively, so the patch's anchor + no longer matches and it contributed nothing. + context: GLM-5.1 over MoRIIO PD on vLLM ROCm 0.23.x. + call_chain: + - "moriio connector save path -> wait_for_save on the DSA write" + symptom_signature: null + resolution: >- + Retired as part of a group of three. v0.25.1 refactored the MoRIIO connector into + moriio_layout.py, which natively covers all three concerns these patches addressed, so their + anchors stopped matching. They were deleted rather than archived at the time — specifically + to stop the build log printing misleading "skipped" lines — which is why this entry has no + file and cites commit 89c86fb for recovery instead. + reusable_on: vLLM 0.23.x and earlier ROCm bases, before the moriio_layout.py refactor. + lesson: >- + Deleting them was right, but the reasoning lived only in a Dockerfile comment. That is the + gap this file closes. + + - name: patch_moriio_hetero.py + source: + original_path: deploy/docker/patches/vllm/patch_moriio_hetero.py + current_path: deleted + last_commit_with_file: 89c86fb + library: vllm + component: moriio + engines_affected: + - vllm + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + retired_on: + date: 2026-07-23 + infera_commit: "3375773" + subject: Remove three no-op MoRIIO patches dead on the v0.25.1 base + retired_reason: base-refactor-made-anchor-obsolete + upstream_fix: null + integrated_by: + infera_commit: "3375773" + how: vLLM base bump to v0.25.1 and its moriio_layout.py refactor. + base_before: vLLM 0.23.x ROCm + base_after: vllm/vllm-openai-rocm:v0.25.1 + problem: + what: >- + Per-layer transfer geometry was not handled for heterogeneous layer layouts, so GLM-5.1 + MoRIIO PD computed transfer offsets from a single global geometry. + why: >- + The pre-v0.25.1 connector assumed a uniform layer geometry across the model. + how: Monkey-patched the connector to compute transfer geometry per layer. + before_fix: Wrong transfer geometry on models with heterogeneous layer layouts. + after_fix: >- + v0.25.1's moriio_layout.py derives per-layer transfer geometry natively, so the anchor + no longer matched. + context: GLM-5.1 over MoRIIO PD on vLLM ROCm 0.23.x. + call_chain: + - "moriio connector -> per-layer transfer geometry computation" + symptom_signature: null + resolution: >- + One of the three retired together on the v0.25.1 move. Note what did NOT retire with them: + patch_moriio_pagelen.py addresses the same file and the same subject — MLA per-block transfer + geometry — and is still load-bearing, because the refactor changed WHERE the geometry is + computed without fixing that it is derived from tensor shape instead of spec.page_size_bytes. + reusable_on: vLLM 0.23.x and earlier ROCm bases. + lesson: >- + A refactor that makes three patches obsolete does not necessarily make the fourth one + obsolete. Each anchor has to be checked individually against the new base. + + - name: patch_vllm_moriio_blocksize.py + source: + original_path: deploy/docker/patches/vllm/patch_vllm_moriio_blocksize.py + current_path: deleted + last_commit_with_file: 89c86fb + library: vllm + component: moriio + engines_affected: + - vllm + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + retired_on: + date: 2026-07-23 + infera_commit: "3375773" + subject: Remove three no-op MoRIIO patches dead on the v0.25.1 base + retired_reason: base-refactor-made-anchor-obsolete + upstream_fix: null + integrated_by: + infera_commit: "3375773" + how: vLLM base bump to v0.25.1 and its moriio_layout.py refactor. + base_before: vLLM 0.23.x ROCm + base_after: vllm/vllm-openai-rocm:v0.25.1 + problem: + what: >- + The MoRIIO connector ignored the kernel/logical block-size ratio, so it indexed logical + pages at kernel granularity. + why: >- + Same defect class as the Mooncake connector's: backends that force a kernel block size of + 1 make cache.shape[0] a multiple of the logical page count, and the connector registered + at the wrong granularity. + how: Monkey-patched the connector to account for the ratio. + before_fix: >- + Transfers addressed the wrong offsets for models whose kernel block size differs from the + scheduler's. + after_fix: >- + v0.25.1's moriio_layout.py handles the kernel/logical ratio natively for the K/V branches + (via kernel_blocks_per_block), so the anchor no longer matched. + context: GLM-5.1 over MoRIIO PD on vLLM ROCm 0.23.x. + call_chain: + - "moriio connector registration -> block_len / num_blocks at kernel rather than logical granularity" + symptom_signature: null + resolution: >- + The third of the group retired on the v0.25.1 move. Worth pairing with its Mooncake + counterpart: patch_vllm_mooncake_blocksize.py is the same defect in the Mooncake connector, + and that one was fixed by UPSTREAM (vllm#46807, merged 2026-06-30) rather than by a refactor. + Same bug, two connectors, two entirely different retirement routes. + reusable_on: vLLM 0.23.x and earlier ROCm bases. + lesson: >- + The kernel-versus-logical block-size confusion has now appeared in three connectors + (Mooncake, MoRIIO, and MoRIIO MLA geometry). It is a design-level trap in the connector + interface, not three unrelated bugs. + + - name: dsa_indexer_hip_dp_padded_rows.diff + source: + original_path: deploy/docker/patches/sglang_dsa/dsa_indexer_hip_dp_padded_rows.diff + current_path: deleted + last_commit_with_file: c91db76 + library: sglang + component: dsa + engines_affected: + - sglang + born: + date: 2026-08-01 + commit: c91db76 + subject: "sglang DSA: enable PD + DP-attention + EAGLE MTP for GLM-5.2 on gfx950" + retired_on: + date: 2026-08-03 + infera_commit: "1380228" + subject: "fix(image): carry the DSA indexer row fix onto the gfx942 v0.5.16 base" + retired_reason: superseded-by-local-patch + upstream_fix: null + integrated_by: null + problem: + what: >- + The DSA indexer's aiter (HIP) paged-MQA branch sizes its logits from the padded row count + while `lengths` is sized from the real one, so fast_topk_v2 asserts under DP-attention. + why: >- + Same root cause as its replacement — see + sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.upstream.status.yaml for the full account. + how: >- + A `patch -p1 --fuzz=0` context diff cut against sglang v0.5.15.post1. + before_fix: >- + "RuntimeError: Expected lengths.size(0) == B to be true, but got false" on gfx950. + after_fix: Fixed on gfx950 — but only on the base the diff was cut against. + context: >- + gfx950 / MI355X, sglang v0.5.15.post1, GLM-5.2 DSA with DP-attention and EAGLE MTP. + call_chain: + - "dsa_indexer.py :: forward (aiter/HIP branch) -> fast_topk_v2 asserts lengths.size(0) == B" + symptom_signature: "Expected lengths.size(0) == B to be true, but got false" + resolution: >- + Not obsolete — REPLACED, by a Python anchor script doing the same repair. A --fuzz=0 diff + only applies to the exact base it was cut against, and the fix was needed on the gfx942 + v0.5.16 base too. The script form also made room for the second direction of the bug (the + DP-idle rank under MTP draft-extend, where there are FEWER query rows than lengths entries), + which was added the same day as GLM52_P1V3 and which a context diff would have made awkward. + reusable_on: sglang v0.5.15.post1 only, by construction. + lesson: >- + Prefer an anchor script to a context diff for anything that has to span two engine bases. + The three sglang_dsa patches now split exactly along that line: the one that became a script + runs on both images, and the two that stayed diffs run on one. diff --git a/deploy/docker/patches/vllm-dsv4/legacy/README.md b/deploy/docker/patches/archived/vllm-dsv4/README.md similarity index 79% rename from deploy/docker/patches/vllm-dsv4/legacy/README.md rename to deploy/docker/patches/archived/vllm-dsv4/README.md index e2ceb32e..c0ea06bf 100644 --- a/deploy/docker/patches/vllm-dsv4/legacy/README.md +++ b/deploy/docker/patches/archived/vllm-dsv4/README.md @@ -1,4 +1,8 @@ -# vllm-dsv4/legacy — deprecated / upstreamed DSv4 patches (not baked into any image) +# archived/vllm-dsv4 — deprecated / upstreamed DSv4 patches (not baked into any image) + +> Moved here from `patches/vllm-dsv4/legacy/` on 2026-08-05. The authoritative record for +> these is now [`../patch.archived.yaml`](../patch.archived.yaml); this file is kept as the +> original provenance note. Nothing under `archived/` is copied into any image. ## Purpose @@ -30,4 +34,5 @@ archival and provenance. - `patch_dsv4_mhc_aiter`'s functionality is upstream via a **different implementation** (aiter as the MHC default rather than an env gate) — "functionally upstream, different form" — so the monkey-patch anchor mismatches on dev748 and auto-skips. -- Active DSv4 patches live in the parent directory `../` (the moriio_dsv4 trio). +- Active DSv4 patches live in `deploy/docker/patches/vllm-dsv4/` (the moriio_dsv4 trio), + each with its own `.upstream.status.yaml` record. diff --git a/deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_aiter_moe.py b/deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_aiter_moe.py similarity index 100% rename from deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_aiter_moe.py rename to deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_aiter_moe.py diff --git a/deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_mhc_aiter.diff b/deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.diff similarity index 100% rename from deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_mhc_aiter.diff rename to deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.diff diff --git a/deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_mhc_aiter.py b/deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.py similarity index 100% rename from deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_mhc_aiter.py rename to deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.py diff --git a/deploy/docker/patches/atom/patch_gdn_pd_state_transfer.upstream.status.yaml b/deploy/docker/patches/atom/patch_gdn_pd_state_transfer.upstream.status.yaml new file mode 100644 index 00000000..84f5cf5e --- /dev/null +++ b/deploy/docker/patches/atom/patch_gdn_pd_state_transfer.upstream.status.yaml @@ -0,0 +1,146 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - local-repro + - patch-header-only + +patch: + path: deploy/docker/patches/atom/patch_gdn_pd_state_transfer.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.atom + opt_in_flag: null + +target: + library: atom + component: attention-metadata + repo: internal + files: + - atom/attention/backends/gdn_attn.py + versions: + - surface: Dockerfile.atom + ref: atom0.1.4 + commit: null + pinned_ref_on_main: null + release_branch: null + unpinned_risk: >- + ATOM is an internal engine consumed only as a base image, so there is no public commit + to record and no way to express "on main". The image tag is the entire pin. + images: + - tag: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612 + digest: null + digest_source: unresolved + note: >- + ROCm 7.2.4, Ubuntu 24.04, Python 3.12, PyTorch 2.10.0, ATOM 0.1.4, built 2026-06-12. + Dockerfile.atom pins the tag only. + +upstream_main_affected: + value: null + evidence: >- + No public upstream exists. ATOM is internal, so there is no repository to check and this + field is null by construction rather than by omission. + +applies_to: + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: true + evidence: >- + Without it, hybrid GDN models cannot recall prompt context in PD at all: memory probes + measured single-node MIXED 5/5 recall against 2-node PD 0/5. + +alive_because: + reason: internal-engine-no-upstream + detail: >- + ATOM has no public upstream to file against, so this cannot be upstreamed in the sense the + other records use. The equivalent action is landing it in ATOM itself; until then the image + patch is the only delivery mechanism. + consumers: + - Dockerfile.atom, for hybrid GDN models (e.g. Qwen3.5-397B-A17B) in PD + drop_when: >- + ATOM's GDNAttentionMetadataBuilder publishes the mamba per-request state itself, at which + point the anchor is gone and the hunk is skipped with a warning. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + Hybrid GDN models produce garbage in PD disaggregation — they cannot recall anything from + the prompt — because the GatedDeltaNet recurrent state is never transferred from the + prefill worker to the decode worker. + why: >- + GDNAttentionMetadataBuilder inherits get_kv_transfer_tensors from + AiterAttentionMetadataBuilder, which registers only the full-attention paged KV cache as + block_regions and returns slot_regions=[]. The linear-attention layers' recurrent state — + conv_state plus temporal/ssm state, held per request in mamba_k_cache / mamba_v_cache — is + therefore not in the transfer set at all, so the decode worker decodes from a ZERO mamba + state. + how: >- + Override get_kv_transfer_tensors on GDNAttentionMetadataBuilder to also publish the mamba + per-request state as direct slot_regions, one region per GDN layer for K and V. The Mooncake + connector already supports exactly this path — register_kv_caches sets + _has_slot_regions=True and _execute_block_slot_transfer Phase 2a does a direct per-slot RDMA + write (src_base + src_slot*unit_bytes -> dst_base + dst_slot*unit_bytes) with no + staging, gather or scatter, because one request's state is contiguous within each layer + buffer of shape [n_gdn, num_slots, *state_shape]. unit_bytes is sized per CACHE GROUP + (= slots_per_req contiguous tensor slots) because the local_slot_index carried in + kv_transfer_params is the per-request cache-group index; with speculative decoding off, + slots_per_req == 1 and a group is a single slot. + before_fix: >- + Anything depending on prompt context comes out as token-salad or off-topic. Memory probes: + single-node MIXED 5/5 recall, 2-node PD 0/5. + after_fix: PD recall matches single-node mixed serving. + context: >- + Hybrid GDN models such as Qwen3.5-397B-A17B on ATOM in 2-node PD over Mooncake. What makes + this dangerous is the masking: world-knowledge facts stored in the WEIGHTS (Paris, Tokyo, + 2+2) still look correct, so a smoke test passes while in-context recall is entirely broken. + call_chain: + - "GDNAttentionMetadataBuilder inherits get_kv_transfer_tensors from AiterAttentionMetadataBuilder" + - "returns block_regions = full-attention paged KV only, slot_regions = []" + - "mamba_k_cache / mamba_v_cache recurrent state is never transferred" + - "decode worker decodes from a zero mamba state" + symptom_signature: >- + in-context recall fails while weight-stored world knowledge answers correctly — 5/5 mixed + versus 0/5 PD + silent: true + +verification: + date: 2026-07-20 + hardware: 2-node AMD ROCm PD + software: ATOM 0.1.4 base image + workload: Qwen3.5-397B-A17B hybrid GDN, memory-probe recall test + result: 0/5 PD recall becomes parity with the 5/5 single-node mixed baseline. + notes: >- + The probe design is the transferable part: testing only world knowledge would have passed. + Any PD correctness check for a hybrid model needs an in-context recall probe. + +scope_limits: >- + Assumes slots_per_req == 1 in the common case (speculative decoding off). With speculative + decoding on, a cache group spans several slots and the unit_bytes sizing is what makes that + correct — worth re-reading before enabling both together. + +open_actions: + - action: >- + Land the equivalent inside ATOM so the image patch can be dropped. There is no public + upstream, so this is the only path to retiring it. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.upstream.status.yaml b/deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.upstream.status.yaml new file mode 100644 index 00000000..b11ba5e4 --- /dev/null +++ b/deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.upstream.status.yaml @@ -0,0 +1,140 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - local-repro + - patch-header-only + +patch: + path: deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.atom + opt_in_flag: null + +target: + library: atom + component: model-definition + repo: internal + files: + - atom/models/minimax_m2.py + versions: + - surface: Dockerfile.atom + ref: atom0.1.4 + commit: null + pinned_ref_on_main: null + release_branch: null + unpinned_risk: >- + Internal engine consumed as a base image; the tag is the entire pin. + images: + - tag: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612 + digest: null + digest_source: unresolved + note: ATOM 0.1.4, built 2026-06-12. Tag-pinned only. + +upstream_main_affected: + value: null + evidence: No public upstream exists — ATOM is internal. + +applies_to: + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: true + evidence: >- + The stock model file does not load at all on this base — get_rope() raises a TypeError on + import. Required to START MiniMax-M2; see scope_limits for what it does not give you. + +alive_because: + reason: internal-engine-no-upstream + detail: >- + No public upstream. This is a base-image inconsistency rather than a design defect: the + shipped minimax_m2.py calls a get_rope API the shipped rope module does not have, so the + two halves of the same image disagree. + consumers: + - Dockerfile.atom, for MiniMax-M2 + drop_when: >- + An ATOM base image ships a minimax_m2.py consistent with its own rope API, and a QK-norm + kernel that does not need the batch guard. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: >- + Hunk B caps the fused QK-norm kernel at qkv.shape[0] <= 256 and falls back to a manual fp32 + path above that. If a future ATOM raises the kernel's real limit, the cap silently keeps the + slower fallback for large batches — correct output, quietly worse throughput, and nothing + fails. The 256 threshold is the number to re-check on a base bump. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + The stock atom/models/minimax_m2.py in the base image fails to load, and its TP fused + QK-norm all-reduce kernel is unsafe for large batches. + why: >- + Two independent problems. (A) The model file passes dtype= to get_rope(), but the installed + rope API does not accept it, so import raises a TypeError — the model file and the rope + module in the same image are out of sync. (B) The TP fused QK-norm all-reduce kernel does not + hold for large batches, so it needs a size guard rather than being taken unconditionally. + how: >- + Two minimal, independently guarded hunks: (A) drop the dtype= kwarg from the get_rope call; + (B) guard the fused kernel with qkv.shape[0] <= 256 so larger batches fall back to the manual + fp32 QK-norm path. Each hunk is applied only if its exact, unique pre-image is present, and + a missing or non-unique pre-image skips that hunk with a loud warning instead of editing code + the patch does not recognise. + before_fix: TypeError from get_rope() at model load — MiniMax-M2 cannot start. + after_fix: >- + MiniMax-M2 loads and runs. Correct OUTPUT additionally requires the triton/unified attention + backend, which is selected at launch (ATOM_USE_UNIFIED_ATTN=1 plus --block-size 64 in + models/minimax_m27.conf) and is not this patch's job. + context: >- + MiniMax-M2 on the rocm/atom 0.1.4 base image. Nothing model-agnostic here — both hunks are + specific to minimax_m2.py. + call_chain: + - "atom/models/minimax_m2.py imported at model load" + - "get_rope(..., dtype=getattr(quant_config, 'torch_dtype', None)) -> TypeError" + - "(separately) TP fused QK-norm all-reduce kernel taken unconditionally for any batch size" + symptom_signature: "TypeError from get_rope() during MiniMax-M2 model load" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm + software: ATOM 0.1.4 base image + workload: MiniMax-M2 model load and serve + result: >- + Model loads and runs. Coherent output needs the unified attention backend selected at launch, + which is configured separately. + notes: >- + Worth keeping the two concerns distinct: this patch makes the model START; the launch config + makes it CORRECT. Conflating them has previously led to blaming the patch for output quality. + +scope_limits: >- + Makes MiniMax-M2 start. It does not make its output correct — that needs + ATOM_USE_UNIFIED_ATTN=1 and --block-size 64 from models/minimax_m27.conf. + +open_actions: + - action: >- + Get a base image whose minimax_m2.py matches its own rope API. Hunk A is fixing an + inconsistency inside the vendor image, which is the vendor's to resolve. + owner: unassigned + blocked_on: null + - action: >- + Establish the fused QK-norm kernel's real batch limit so the 256 guard is a measured + threshold rather than a safe-looking round number. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/atom/patch_mooncake_consumer_slot.upstream.status.yaml b/deploy/docker/patches/atom/patch_mooncake_consumer_slot.upstream.status.yaml new file mode 100644 index 00000000..ac445fb4 --- /dev/null +++ b/deploy/docker/patches/atom/patch_mooncake_consumer_slot.upstream.status.yaml @@ -0,0 +1,123 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - local-repro + - patch-header-only + +patch: + path: deploy/docker/patches/atom/patch_mooncake_consumer_slot.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.atom + opt_in_flag: null + +target: + library: atom + component: mooncake-connector + repo: internal + files: + - atom/kv_transfer/disaggregation/mooncake/mooncake_connector.py + versions: + - surface: Dockerfile.atom + ref: atom0.1.4 + commit: null + pinned_ref_on_main: null + release_branch: null + unpinned_risk: >- + Internal engine consumed as a base image; the tag is the entire pin. + images: + - tag: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612 + digest: null + digest_source: unresolved + note: ATOM 0.1.4, built 2026-06-12. Tag-pinned only. + +upstream_main_affected: + value: null + evidence: No public upstream exists — ATOM is internal. + +applies_to: + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: true + evidence: >- + Without it the decode worker crashes on the FIRST PD request for any model with + cache-group slot state but no registered slot_regions. + +alive_because: + reason: internal-engine-no-upstream + detail: >- + No public upstream. Note the coupling worth knowing: this defect is reached by exactly the + model class that the GDN state-transfer patch also serves — Qwen3.5's GDN layout, where + local_slot_index >= 0 while _has_slot_regions is False. + consumers: + - Dockerfile.atom, for hybrid models with slot state but no slot_regions (e.g. Qwen3.5 GDN) + drop_when: ATOM initialises consumer_staging_pool_idx unconditionally. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + In the Mooncake PD KV-transfer consumer, start_load_kv reads consumer_staging_pool_idx on a + path where it was never bound, so the decode worker dies with an UnboundLocalError on the + first PD request. + why: >- + The variable is bound only inside the `if self._has_slot_regions:` branch but referenced + unconditionally when recording a pending recv slot. Hybrid models that use cache-group slot + state but do NOT register slot_regions — Qwen3.5's GDN linear-attention layout, where + local_slot_index >= 0 while _has_slot_regions is False — take the else path and hit the + unbound read. + how: >- + Initialise consumer_staging_pool_idx = -1 at the top of the per-request loop. The WRITE_DONE + release path is already guarded by `if pool_idx >= 0:`, so -1 is safe and simply skips the + staging scatter and release for the block-only path. + before_fix: >- + "UnboundLocalError: cannot access local variable 'consumer_staging_pool_idx'" and the decode + worker crashes on the first PD request. + after_fix: The decode worker survives and the block-only path skips staging cleanly. + context: >- + ATOM Mooncake PD with a hybrid model that has cache-group slot state but no registered + slot_regions. Models that do register slot_regions never take the failing path. + call_chain: + - "atom/kv_transfer/disaggregation/mooncake/mooncake_connector.py :: start_load_kv" + - "consumer_staging_pool_idx bound only inside `if self._has_slot_regions:`" + - "else path records a pending recv slot and reads it anyway" + - "UnboundLocalError kills the decode worker on the first PD request" + symptom_signature: "UnboundLocalError: cannot access local variable 'consumer_staging_pool_idx'" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm PD + software: ATOM 0.1.4 base image + workload: Qwen3.5 GDN PD, first request + result: Crash cleared. + notes: >- + The -1 sentinel is safe only because the release path already tests pool_idx >= 0 — that + pre-existing guard is what makes a one-line fix correct here. + +scope_limits: null + +open_actions: + - action: Land the equivalent inside ATOM so the image patch can be dropped. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/hipfile_async/hipfile_async.upstream.status.yaml b/deploy/docker/patches/hipfile_async/hipfile_async.upstream.status.yaml new file mode 100644 index 00000000..42b0fbc4 --- /dev/null +++ b/deploy/docker/patches/hipfile_async/hipfile_async.upstream.status.yaml @@ -0,0 +1,180 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/hipfile_async/hipfile_async.patch + extra_files: + - deploy/docker/patches/hipfile_async/file_async_fragment.py + - deploy/docker/patches/hipfile_async/patch_hipfile_async.sh + kind: git-diff + idempotent: true + applied_by: + - Dockerfile.vllm + opt_in_flag: BUILD_HIPFILE=1 (build ARG; default 1) + +target: + library: hipfile + component: python-bindings + repo: ROCm/rocm-systems + files: + - projects/hipfile/python/hipfile/__init__.py + - projects/hipfile/python/hipfile/_chipfile.pxd + - projects/hipfile/python/hipfile/_hipfile.pyx + - projects/hipfile/python/hipfile/file.py + versions: + - surface: Dockerfile.vllm (build_hipfile.sh) + ref: unpinned-default-branch + commit: null + pinned_ref_on_main: true + release_branch: null + unpinned_risk: >- + This is the important fact about this patch. build_hipfile.sh defaults + HIPFILE_GIT_REF to empty, i.e. HEAD of the ROCm/rocm-systems default branch, so the + target moves on every build with no record of what was built. Since #7386 merged on + 2026-07-16, any clone taken after that date already carries the async API — which + means this patch is expected to be a no-op on any fresh build today, and the image + content depends on the build DATE rather than on anything in this repo. + images: [] + +upstream_main_affected: + value: false + evidence: >- + ROCm/rocm-systems#7386 is MERGED (2026-07-16) and lands the same feature on the same + four files this patch edits, with the same root-cause account of the pointer-lifetime + bug. The default branch therefore carries the API. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (BUILD_HIPFILE=1, step 4) + effect: unverified + required_to_run: false + evidence: >- + Effect depends on the build date, not on the repo: the script greps the cloned tree + for the hipFileStreamRegister marker and skips if present. Any build after 2026-07-16 + clones a tree that has #7386, so this is very likely a no-op now — but nothing in the + repo records which it was for a given image, so it cannot be asserted either way. + +alive_because: + reason: released-but-infera-not-bumped + detail: >- + Ours, merged upstream on 2026-07-16, and the target is unpinned — so upstream's fix + arrives on its own the next time anyone builds. This is not really "alive" so much as + "not yet deleted". Note the two implementations differ in HOW they keep the driver's + out-parameters alive, so this is merged-but-not-equivalent: upstream owns the slots in a + Cython AsyncIOHandle cdef class, while this patch sidesteps Cython and calls + libhipfile.so through ctypes with heap-allocated c_size_t / c_int64 / c_ssize_t slots. + Upstream's is the better shape. + consumers: + - >- + none, most likely — only a build pinned to a rocm-systems tree older than #7386 would + still need it + drop_when: >- + Now, in practice: the base image's /opt/rocm-systems-src/projects/hipfile carries #7386 + on any fresh clone. Confirm with one grep for hipFileStreamRegister and delete. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + The marker grep makes the skip safe, so there is no misapply risk. The real risk is the + opposite and worth stating plainly: because HIPFILE_GIT_REF is unpinned, two images built + from the same infera commit on either side of 2026-07-16 contain DIFFERENT hipfile code — + one with our ctypes implementation, one with upstream's Cython one. That is an + unreproducible build, independent of whether the patch is kept. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged. The upstream PR was ours and merged four days before this repo's v0.1.0 + commit landed the local copy, and nobody revisited it afterwards. + +upstream_issues: null + +upstream_prs: + - ref: ROCm/rocm-systems#7386 + url: https://github.com/ROCm/rocm-systems/pull/7386 + title: "hipfile: feat(hipfile/python): add async stream I/O bindings" + state: MERGED + review_decision: null + merged_at: 2026-07-16 + author: jiejingzhangamd + ours: true + same_approach: false + approach_note: >- + Same feature, same four files, same root-cause account — but a different mechanism for + keeping the driver's out-parameters alive. Upstream owns the slots in a Cython + AsyncIOHandle cdef class; the local patch avoids Cython entirely and calls libhipfile.so + through ctypes with heap-allocated slots. Ours was the fast local implementation; the + upstream one is the general fix and is better. Do not carry the local one forward. + requested_action: null + in_pinned_base: null + +related_refs: null + +problem: + what: >- + The ROCm hipFile Python binding exposes no async API, and the obvious Cython + implementation of one is unsafe: out-parameters passed as `&local` die before the driver + dereferences them. + why: >- + Cython stack locals passed by address to an asynchronous driver call go out of scope when + the Python frame returns, while the driver writes to them later. The write then lands on + reclaimed stack, so bytes_done reads 0 and HipFileException 5022 appears + intermittently — a lifetime bug, so it is timing-dependent and looks flaky. + how: >- + Add write_async / read_async, Stream and supports_async() to the binding, and keep the + driver's out-parameter slots on the HEAP: the wrapper calls libhipfile.so via ctypes with + heap-allocated c_size_t / c_int64 / c_ssize_t slots that outlive the call. + before_fix: >- + No async API at all; a naive Cython binding gives bytes_done == 0 and intermittent + HipFileException 5022. + after_fix: >- + Async stream I/O works and bytes_done is correct, which is what kvd's gpu-direct L3 load + path needs. + context: >- + ROCm hipFile Python bindings, used by kvd's gpu-direct L3. Built into Dockerfile.vllm + from a clone of ROCm/rocm-systems, so the effective target is whatever that default + branch holds at build time. + call_chain: + - "hipfile/file.py :: write_async / read_async" + - "_hipfile.pyx passes &local out-parameters to the async driver call" + - "the Python frame returns; the driver writes to reclaimed stack" + - "bytes_done reads 0 / HipFileException 5022" + symptom_signature: "HipFileException 5022 with bytes_done == 0, intermittently" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm with hipFile / AIS + software: ROCm hipfile python bindings from ROCm/rocm-systems + workload: kvd gpu-direct L3 async read/write + result: Async I/O completes with correct bytes_done; the intermittent 5022 is gone. + notes: >- + Also note the neighbouring trap in the same build step: a missing ais-check probe + SILENTLY downgrades the kvd load path to CPU-bounce, which is why Dockerfile.vllm asserts + `test -x /opt/rocm/bin/ais-check`. + +scope_limits: >- + A local implementation, superseded upstream by a better one. It should not be carried + forward on its own merits. + +open_actions: + - action: >- + Delete this patch. It is ours, merged upstream on 2026-07-16, the target is unpinned so + the fix arrives by itself, and upstream's implementation is the better of the two. + owner: unassigned + blocked_on: null + - action: >- + Separately, pin HIPFILE_GIT_REF. An unpinned default-branch clone makes image contents a + function of the build date; that is a reproducibility problem regardless of this patch. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.upstream.status.yaml b/deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.upstream.status.yaml new file mode 100644 index 00000000..cccbd185 --- /dev/null +++ b/deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.upstream.status.yaml @@ -0,0 +1,177 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - git-compare + - local-repro + +patch: + path: deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.diff + kind: git-diff + idempotent: true + applied_by: + - deploy/docker/patches/mooncake_cpp/apply_mooncake_cpp_patches.sh + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + - Dockerfile.vllm + - Dockerfile.atom + - deploy/overlay/Dockerfile.payload + opt_in_flag: null + +target: + library: mooncake + component: transfer-engine-rdma + repo: kvcache-ai/Mooncake + files: + - mooncake-transfer-engine/include/transport/rdma_transport/rdma_transport.h + - mooncake-transfer-engine/src/transport/rdma_transport/rdma_context.cpp + - mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp + versions: + - surface: all images (MOONCAKE_GIT_REF) + ref: main @ 747003c + commit: 747003c058015c4077a266e7ccd7549bbc9baede + pinned_ref_on_main: true + release_branch: null + images: [] + +upstream_main_affected: + value: false + evidence: >- + Mooncake#2644 is MERGED (2026-07-28) and the pin is an ancestor of main, so main + carries the fix. Established with `git compare` between #2644's merge commit and + 747003c: the pin is BEHIND the merge, so main has it and we do not. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang / Dockerfile.sglang.gfx942 (Mooncake rebuilt in place) + effect: op + required_to_run: false + evidence: >- + Load-bearing only above the device max_mr_size — on ionic that is ~2 GiB, which a + real KV pool exceeds. `git apply --reverse --check` in the apply script makes the + already-applied case an idempotent skip. + - engine: vllm + surface: Dockerfile.vllm (build_mooncake_rocm.sh, MOONCAKE_DMABUF=1) + effect: op + required_to_run: false + evidence: Same diff, same pin, same reasoning. + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: false + evidence: Same diff, same pin. + +alive_because: + reason: merged-not-in-pinned-base + detail: >- + Ours, and merged upstream on 2026-07-28 — but MOONCAKE_GIT_REF is still 747003c + (2026-06-26), which is behind that merge. So the local diff is still doing the work. + consumers: + - all four engine images plus the overlay payload, on any fabric whose max_mr_size is + smaller than the KV pool + drop_when: "MOONCAKE_GIT_REF advances past #2644's merge commit." + drop_signal: pin-advance-only + silent_misapply_risk: >- + Low but not zero: `git apply --reverse --check` detects the already-applied case and + skips, so a pin advance past #2644 produces a clean skip rather than a failure. That + is the desired behaviour, but it means nothing announces that the diff has become + dead weight — only the pin tells you. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged since it was written — the upstream review of #2644 did not require a + change to the local diff. + +upstream_issues: + - ref: kvcache-ai/Mooncake#2017 + url: https://github.com/kvcache-ai/Mooncake/issues/2017 + kind: issue + title: Buffers larger than device max_mr_size are silently truncated + state: CLOSED + state_reason: "fixed by #2644" + author: null + note: >- + Our own report, and the number the diff and its chunk_map_ comments cite + throughout. + +upstream_prs: + - ref: kvcache-ai/Mooncake#2644 + url: https://github.com/kvcache-ai/Mooncake/pull/2644 + title: "fix(rdma): auto-chunk MRs larger than device max_mr_size (#2017)" + state: MERGED + review_decision: null + merged_at: 2026-07-28 + author: jiejingzhangamd + ours: true + same_approach: true + approach_note: >- + Same change, upstreamed by us and merged. The local diff is the same content + carried against the older pin. + requested_action: null + in_pinned_base: false + +related_refs: null + +problem: + what: >- + Buffers larger than the device max_mr_size are silently truncated by ibv_reg_mr while + BufferDesc.length keeps advertising the full size, so RDMA operations whose target + lands past the boundary fail. + why: >- + registerMemoryRegionInternal saw `length > max_mr_size`, logged a warning and simply + shortened `length`. The registration then succeeded for a shorter region than the + descriptor advertises, so a remote peer computes a valid-looking address inside a + region that was never registered. + how: >- + Chunk at registration time: registerLocalMemory splits buffers into <= max_mr_size + MRs, one BufferDesc per chunk. Because unregisterLocalMemory() only receives the base + address, the transport remembers each base buffer's chunk start addresses in a + chunk_map_ guarded by chunk_map_mutex_ so cleanup can find them. The truncation site + itself becomes a hard error rather than a warning — after chunking, no larger buffer + should ever reach it, so failing loudly is correct. + before_fix: >- + IBV_WC_REM_ACCESS_ERR on RDMA operations whose target lands past the truncation + boundary, with only a PLOG(WARNING) at registration to hint at why. + after_fix: >- + Large KV pools register as multiple MRs and cross-node transfers complete. A buffer + that somehow still exceeds max_mr_size now returns ERR_INVALID_ARGUMENT instead of + registering short. + context: >- + AMD MI355X + ionic RoCE, where max_mr_size is about 2 GiB — comfortably smaller than + a production KV pool. Any fabric with a max_mr_size below the pool size hits this; + it is not ROCm-specific, which is why upstream took it. + call_chain: + - "RdmaTransport::registerLocalMemory(addr, length, ...)" + - "RdmaContext::registerMemoryRegionInternal — compared length > globalConfig().max_mr_size and shrank length" + - "ibv_reg_mr registers a shorter region than BufferDesc.length advertises" + - "remote RDMA op targeting past the boundary -> IBV_WC_REM_ACCESS_ERR" + symptom_signature: IBV_WC_REM_ACCESS_ERR past the max_mr_size boundary + silent: false + +verification: + date: 2026-07-20 + hardware: AMD MI355X, ionic RoCE (max_mr_size ~2 GiB) + software: Mooncake main @ 747003c + workload: cross-node PD KV transfer with a KV pool larger than max_mr_size + result: "REM_ACCESS_ERR cleared; upstream accepted the same change as #2644." + notes: null + +scope_limits: null + +open_actions: + - action: >- + Advance MOONCAKE_GIT_REF past #2644 and drop this diff. It is the cleanest + retirement available — our own merged fix, only the pin holding it. + owner: unassigned + blocked_on: >- + A pin advance also pulls #2682/#2725, which changes the hip-transport picture; do + the two together, not separately. diff --git a/deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.upstream.status.yaml b/deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.upstream.status.yaml new file mode 100644 index 00000000..9707aae4 --- /dev/null +++ b/deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.upstream.status.yaml @@ -0,0 +1,159 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.diff + kind: git-diff + idempotent: true + applied_by: + - deploy/docker/patches/mooncake_cpp/apply_mooncake_cpp_patches.sh + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + - Dockerfile.vllm + - Dockerfile.atom + - deploy/overlay/Dockerfile.payload + opt_in_flag: MOONCAKE_HIP_DMABUF=1 (build ARG; default 0) + +target: + library: mooncake + component: transfer-engine-rdma + repo: kvcache-ai/Mooncake + files: + - mooncake-transfer-engine/CMakeLists.txt + - mooncake-transfer-engine/src/transport/rdma_transport/rdma_context.cpp + versions: + - surface: all images (MOONCAKE_GIT_REF) + ref: main @ 747003c + commit: 747003c058015c4077a266e7ccd7549bbc9baede + pinned_ref_on_main: true + release_branch: null + images: [] + +upstream_main_affected: + value: false + evidence: >- + Mooncake#2543 is MERGED (2026-07-24) and 747003c predates it, so main carries + dma-buf GPUDirect registration and our pin does not. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang / Dockerfile.sglang.gfx942 + effect: op + required_to_run: false + evidence: >- + Only when MOONCAKE_HIP_DMABUF=1. Default is 0, in which case GPU memory is + registered with bare ibv_reg_mr via host-libionic injection and this diff is not + needed. It IS required on a dma-buf-only RoCE fabric with no ib_peer_mem, e.g. the + crusoe amd-spur cluster, where bare ibv_reg_mr cannot pin VRAM at all. + - engine: vllm + surface: Dockerfile.vllm (MOONCAKE_HIP_DMABUF ARG, default 0) + effect: op + required_to_run: false + evidence: Same ARG, same condition. + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: false + evidence: Same ARG, same condition. + +alive_because: + reason: merged-not-in-pinned-base + detail: >- + Upstream merged the equivalent as #2543 on 2026-07-24; MOONCAKE_GIT_REF is still + 747003c (2026-06-26), which is behind it. Purely a pin lag. + consumers: + - any image built with MOONCAKE_HIP_DMABUF=1 (dma-buf-only fabrics such as crusoe amd-spur) + drop_when: "MOONCAKE_GIT_REF advances past #2543's merge commit." + drop_signal: pin-advance-only + silent_misapply_risk: >- + The apply script's `git apply --reverse --check` makes an already-fixed pin a clean + skip, so advancing the pin will not fail the build — nothing will announce that this + diff has become redundant. Track it by the pin, not by the build log. + +history: + born: + date: 2026-07-29 + commit: a322e14 + subject: add pd utest on crusoe cluster + last_modified: + date: 2026-07-29 + commit: a322e14 + subject: add pd utest on crusoe cluster + reason: >- + Added for the crusoe amd-spur bring-up, where the fabric offers no ib_peer_mem and + dma-buf is the only way to register VRAM. Unchanged since. + +upstream_issues: null + +upstream_prs: + - ref: kvcache-ai/Mooncake#2543 + url: https://github.com/kvcache-ai/Mooncake/pull/2543 + title: "feat(rdma): support dma-buf GPUDirect memory registration" + state: MERGED + review_decision: null + merged_at: 2026-07-24 + author: null + ours: false + same_approach: true + approach_note: >- + Same mechanism — detect ibv_reg_dmabuf_mr at configure time and use it for GPU + buffers. Ours is the same content carried against the older pin; nothing here is + local-only. + requested_action: null + in_pinned_base: false + +related_refs: null + +problem: + what: >- + Mooncake at our pin has no dma-buf GPUDirect registration path, so on a fabric that + exposes only dma-buf it cannot register VRAM for RDMA at all. + why: >- + Registration goes through bare ibv_reg_mr, which needs the legacy ib_peer_mem kernel + module to pin device memory. On a dma-buf-only RoCE fabric that module is absent, so + there is no mechanism by which the VRAM can be pinned. + how: >- + Detect ibv_reg_dmabuf_mr availability in CMakeLists.txt and, when + MOONCAKE_HIP_DMABUF=1, register GPU buffers through it (dma-buf GPUDirect) instead of + bare ibv_reg_mr. + before_fix: >- + On crusoe amd-spur, GPU memory registration fails outright — RDMA cannot be set up, + so cross-node PD never starts. + after_fix: >- + GPU buffers register via ibv_reg_dmabuf_mr and cross-node PD works on the dma-buf-only + fabric. PD unit tests pass there. + context: >- + Fabric-dependent, not model-dependent. Required on crusoe amd-spur (dma-buf only, no + ib_peer_mem). On ionic the default path (bare ibv_reg_mr + host-libionic injection) + works, but must not be used at high utilisation — see HIP-209. + call_chain: + - "mooncake-transfer-engine/CMakeLists.txt — probes for ibv_reg_dmabuf_mr" + - "RdmaContext::registerMemoryRegionInternal — bare ibv_reg_mr for GPU buffers" + - "no ib_peer_mem -> VRAM cannot be pinned -> registration fails" + symptom_signature: GPU memory registration failure on a fabric without ib_peer_mem + silent: false + +verification: + date: 2026-07-29 + hardware: crusoe amd-spur cluster (dma-buf-only RoCE, no ib_peer_mem) + software: Mooncake main @ 747003c with MOONCAKE_HIP_DMABUF=1 + workload: cross-node PD unit tests + result: Registration succeeds and the PD tests pass on that fabric. + notes: null + +scope_limits: >- + Opt-in only. With the default MOONCAKE_HIP_DMABUF=0 this changes nothing, so it is not + a fix for the ionic default path. + +open_actions: + - action: >- + Advance MOONCAKE_GIT_REF past #2543 and drop this diff — same pin bump that retires + rdma_auto_chunk_mr_2017.diff. + owner: unassigned + blocked_on: >- + The same pin advance also pulls #2682/#2725; sequence it with the hip-transport gate. diff --git a/deploy/docker/patches/mooncake_cpp/transfer_engine_impl.upstream.status.yaml b/deploy/docker/patches/mooncake_cpp/transfer_engine_impl.upstream.status.yaml new file mode 100644 index 00000000..bad5aa4c --- /dev/null +++ b/deploy/docker/patches/mooncake_cpp/transfer_engine_impl.upstream.status.yaml @@ -0,0 +1,183 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/mooncake_cpp/transfer_engine_impl.diff + kind: git-diff + idempotent: true + applied_by: + - deploy/docker/patches/mooncake_cpp/apply_mooncake_cpp_patches.sh + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + - Dockerfile.vllm + - Dockerfile.atom + - deploy/overlay/Dockerfile.payload + opt_in_flag: MC_DISABLE_HIP_TRANSPORT / MC_ENABLE_HIP_TRANSPORT (runtime env; the gate itself is always compiled in) + +target: + library: mooncake + component: transfer-engine + repo: kvcache-ai/Mooncake + files: + - mooncake-transfer-engine/src/transfer_engine_impl.cpp + versions: + - surface: all images (MOONCAKE_GIT_REF) + ref: main @ 747003c + commit: 747003c058015c4077a266e7ccd7549bbc9baede + pinned_ref_on_main: true + release_branch: null + images: [] + +upstream_main_affected: + value: true + evidence: >- + Mooncake#2725, which would add the env gate upstream, is still OPEN with no review + activity, so main still installs the HIP transport unconditionally. #2682, the PR + that introduced the unconditional install, is merged and in our pin. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang / Dockerfile.sglang.gfx942 + effect: op + required_to_run: true + evidence: >- + Cross-node PD does not work without it. Both sglang Dockerfiles additionally + ASSERT the gate is present in the .so they will load (REQUIRE_MOONCAKE_HIP_GATE=1 + greps the binary for MC_ENABLE_HIP_TRANSPORT / MC_DISABLE_HIP_TRANSPORT), because + a build that silently skipped the rebuild ships stock #2682 and dies at run time. + - engine: vllm + surface: Dockerfile.vllm + effect: op + required_to_run: true + evidence: Same assertion layer (step 3b of Dockerfile.vllm), same reasoning. + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: true + evidence: Same diff and pin. + +alive_because: + reason: upstream-pr-open + detail: >- + Ours as #2725, open and unreviewed. Until it merges AND the pin advances past it, + every ROCm image needs the local gate — and because the failure is a hard runtime + error on cross-node PD, the images assert the gate is compiled in rather than trust + the build. + consumers: + - every ROCm engine image doing cross-node PD over RDMA + drop_when: >- + #2725 merges and MOONCAKE_GIT_REF advances past it. + drop_signal: pin-advance-only + silent_misapply_risk: >- + Inverted here, and worth being clear about: the danger is not the patch outliving its + reason but the patch silently NOT being applied. That is what the + REQUIRE_MOONCAKE_HIP_GATE assertion exists for, plus + deploy/docker/scripts/verify_image_mooncake.sh for images built FROM an already + published image where the build stage never re-runs. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-08-01 + commit: b7df12a + subject: "fix(mooncake): disable the HIP (hipIpc) transport on ROCm" + reason: >- + Turned into a real env-var gate. Before this the diff addressed the same defect + differently; b7df12a made it MC_DISABLE_HIP_TRANSPORT / MC_ENABLE_HIP_TRANSPORT so + the behaviour can be selected at run time and asserted in the shipped .so. + +upstream_issues: + - ref: kvcache-ai/Mooncake#2724 + url: https://github.com/kvcache-ai/Mooncake/issues/2724 + kind: issue + title: HIP transport is installed unconditionally and preferred over RDMA on ROCm + state: OPEN + state_reason: null + author: jiejingzhangamd + note: "Our report; #2725 is the fix for it." + +upstream_prs: + - ref: kvcache-ai/Mooncake#2725 + url: https://github.com/kvcache-ai/Mooncake/pull/2725 + title: "fix(hip): gate the HIP transport behind an env var (#2724)" + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: jiejingzhangamd + ours: true + same_approach: true + approach_note: Same gate, submitted upstream. No review activity yet. + requested_action: null + in_pinned_base: false + +related_refs: + - ref: kvcache-ai/Mooncake#2682 + url: https://github.com/kvcache-ai/Mooncake/pull/2682 + kind: pr + title: Add HIP (hipIpc) transport + state: MERGED + state_reason: fixed + author: null + relevance: introduced-the-defect + note: >- + This is what installs the HIP transport unconditionally and prefers it over RDMA. + It is IN our pin, which is why every image has to rebuild Mooncake rather than use + the one the base ships. + +problem: + what: >- + Mooncake installs the HIP (hipIpc) transport unconditionally and prefers it over + RDMA, so on a multi-node ROCm deployment the engine picks a transport that only works + within a node. + why: >- + hipIpc memory handles are node-local by construction. Upstream #2682 added the + transport and registered it without any topology or reachability test, and the + selection order puts it ahead of RDMA, so a cross-node target is attempted over + hipIpc and fails at handle-open time. + how: >- + Gate installation behind an explicit env var — MC_DISABLE_HIP_TRANSPORT to force it + off, MC_ENABLE_HIP_TRANSPORT to opt in — so ROCm multi-node deployments fall through + to RDMA. Compiling the gate in unconditionally is what lets the images assert on it. + before_fix: >- + Cross-node PD dies with hipIpcOpenMemHandle error 201. + after_fix: >- + Cross-node PD runs over RDMA. The strings MC_ENABLE_HIP_TRANSPORT and + MC_DISABLE_HIP_TRANSPORT are present in the shipped engine .so, which is what the + build-time assertion checks. + context: >- + Any ROCm multi-node deployment on a Mooncake carrying #2682. Not model-specific and + not arch-specific beyond being ROCm. + call_chain: + - "TransferEngineImpl::init -> installTransport(hip) with no reachability test" + - transport selection prefers hip over rdma + - "cross-node target -> hipIpcOpenMemHandle -> error 201" + symptom_signature: "hipIpcOpenMemHandle ... 201" + silent: false + +verification: + date: 2026-08-01 + hardware: multi-node AMD ROCm with RDMA + software: "Mooncake main @ 747003c (i.e. including #2682)" + workload: cross-node PD KV transfer + result: hipIpcOpenMemHandle 201 cleared; RDMA is selected. + notes: >- + The build-time gate assertion is a separate Dockerfile layer on purpose so it runs in + both BUILD_MOONCAKE branches and shows up in `docker history`. + +scope_limits: >- + The gate does not decide anything by topology — it is an explicit switch. A single-node + ROCm deployment that would genuinely benefit from hipIpc has to opt in with + MC_ENABLE_HIP_TRANSPORT. + +open_actions: + - action: "Get #2725 reviewed; it has had no activity since it was opened." + owner: jiejingzhangamd + blocked_on: upstream review diff --git a/deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.upstream.status.yaml b/deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.upstream.status.yaml new file mode 100644 index 00000000..0afdbfb4 --- /dev/null +++ b/deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.upstream.status.yaml @@ -0,0 +1,169 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.sglang + opt_in_flag: null + +target: + library: sglang + component: quantization + repo: sgl-project/sglang + files: + - python/sglang/srt/models/deepseek_nextn.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: >- + This release line was cut without #30265, which is the whole reason the + backport exists. + +upstream_main_affected: + value: false + evidence: >- + #30265 is MERGED and gives GLM-5.2 its own GlmMoeDsaForCausalLMNextN in + glm4_moe.py, so the GLM path on main is fixed. Read from main on 2026-08-05. + Note that the anchor STRING is still on main — see silent_misapply_risk. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + Without it the GLM-5.2 draft weight-load dies and MTP cannot start. + apply_sglang_dsa_patches.sh asserts this patch as a PREREQUISITE of the DSA set + by grepping deepseek_nextn.py for 'num_hidden_layers}.eh_proj'. + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: not-applied + required_to_run: false + evidence: >- + Deliberately not applied — v0.5.16 carries #30265, so the fix is already in that + base. Dockerfile.sglang.gfx942's header says so explicitly. + +alive_because: + reason: merged-not-in-pinned-base + detail: >- + #30265 merged upstream on 2026-07-08, but our mi35x base is the v0.5.15.post1 + release tag and that release line was cut without it. Bumping a patch level does + not help: v0.5.15.post1 sits on release/v0.5.15, not main. + consumers: + - Dockerfile.sglang (GLM-5.2 MTP on the v0.5.15.post1 mi35x base) + drop_when: >- + Dockerfile.sglang moves to a base sglang that carries #30265 — v0.5.16 or later + already does, which is why the gfx942 image does not apply this. + drop_signal: base-version-only + silent_misapply_risk: >- + CORRECTED 2026-08-05. The patch's own header, and the previous status page, both + claimed the anchor would disappear on a fixed base and the script would no-op. It + does not. #30265 touched deepseek_nextn.py (+13/-11) only to lift the quark check + out of __init__ into _resolve_nextn_quant_config(); the literal + 'ckpt_prefix = f"model.layers.{config.num_hidden_layers}"' is still on main at line + 347, and this patch matches that string without leading indentation. On a + post-#30265 base it therefore still applies — and now edits the DeepSeek MTP path, + which GLM-5.2 no longer takes, narrowing that family's quark-exclude probe to + eh_proj alone while logging success. Decide the drop from the base version. + +history: + born: + date: 2026-07-28 + commit: 0d8d0ff + subject: "fix(image): patch sglang for GLM-5.2 MTP nextn quark-exclude (backport #30265)" + last_modified: + date: 2026-07-28 + commit: 0d8d0ff + subject: "fix(image): patch sglang for GLM-5.2 MTP nextn quark-exclude (backport #30265)" + reason: >- + Written as a deliberately narrow backport in one go — a one-line anchor swap + rather than a port of #30265's new model class. + +upstream_issues: null + +upstream_prs: + - ref: sgl-project/sglang#30265 + url: https://github.com/sgl-project/sglang/pull/30265 + title: "[AMD] Fix GLM-5.2 MTP Quark excludes" + state: MERGED + review_decision: APPROVED + merged_at: 2026-07-08 + author: wangjiaxin99 + ours: false + same_approach: false + approach_note: >- + #30265 is a superset: it adds a dedicated GlmMoeDsaForCausalLMNextN class in + glm4_moe.py plus model_config.py plumbing. Ours is a one-line probe change on the + shared DeepSeek nextn path. We took the narrow form on purpose — this is a + backport onto a frozen release base, not a general fix, and porting a new model + class into a running container is not something to do blind. + requested_action: null + in_pinned_base: false + +related_refs: null + +problem: + what: >- + GLM-5.2's MTP (nextn) layer is entirely bf16/unquantized, but sglang's quark + exclude check does not recognise it as excluded, so eh_proj is built as an MXFP4 + (uint8) parameter and the draft weight-load fails. + why: >- + The quark quantization_config lists exclude entries at SUBMODULE level + ('model.layers..eh_proj'), while DeepseekV3ForCausalLMNextN probes the BARE layer + prefix ('model.layers.'). should_ignore_layer() therefore returns False, the + layer is treated as quantized, and nextn_quant_config is not cleared. + how: >- + Probe the eh_proj submodule — an exact exclude entry — instead of the bare layer, so + the match succeeds, nextn_quant_config becomes None, and the whole bf16 MTP layer is + built bf16. + before_fix: >- + "AssertionError: param.shape=[6144,6144] uint8 vs loaded_weight.shape=[6144,12288] + bf16" during draft weight-load — reported on the status page previously as + '3072 vs 6144'. + after_fix: Coherent GLM-5.2 MTP output on the v0.5.15.post1 base. + context: >- + gfx950 / MI355X, sglang v0.5.15.post1, GLM-5.2 with quark MXFP4 weights and EAGLE + MTP. Not reachable on v0.5.16+, which carries #30265. + call_chain: + - "sglang/srt/models/deepseek_nextn.py :: DeepseekV3ForCausalLMNextN.__init__" + - "should_ignore_layer(mapped_prefix, quant_config.exclude_layers) — probes the bare layer prefix" + - "eh_proj built as an MXFP4 uint8 param" + - "draft weight-load shape assertion fails" + symptom_signature: "param.shape=[6144,6144] uint8 vs loaded_weight.shape=[6144,12288] bf16" + silent: false + +verification: + date: 2026-07-28 + hardware: 8x MI355X (gfx950) + software: sglang v0.5.15.post1, ROCm 7.2.0 + workload: GLM-5.2 quark MXFP4 with EAGLE MTP + result: Draft weight-load succeeds and MTP output is coherent. + notes: null + +scope_limits: >- + Narrow by design: it fixes the probe, not the model class. It does not give GLM-5.2 + the dedicated nextn class #30265 added, and it must not be carried onto a base that + has one. + +open_actions: + - action: >- + Fix this patch's own header, which still claims the anchor disappears once + upstream carries #30265. It does not, and the claim is what makes a silent + misapply possible. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml b/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml new file mode 100644 index 00000000..fd0c06d5 --- /dev/null +++ b/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml @@ -0,0 +1,188 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + opt_in_flag: null + +target: + library: sglang + component: disaggregation + repo: sgl-project/sglang + files: + - python/sglang/srt/disaggregation/common/utils.py + - python/sglang/srt/disaggregation/mooncake/conn.py + - python/sglang/srt/disaggregation/prefill.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: null + - surface: Dockerfile.sglang.gfx942 + ref: v0.5.16 + commit: fdebc938f7f4 + pinned_ref_on_main: false + release_branch: release/v0.5.16 + images: + - tag: lmsysorg/sglang:v0.5.16-rocm720-mi30x + digest: null + digest_source: unresolved + note: The anchors are present in both v0.5.15.post1 and v0.5.16. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: disaggregation/mooncake/conn.py contains ZERO + occurrences of `wait_event`, while disaggregation/mori/conn.py has five (the + TransferKVChunk field, the pickup off the sender, the forward, and the + transfer_worker wait). Main is affected, and this is a source read rather than a + search miss. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + Required for correctness, not startup: without it any prompt longer than one + prefill chunk comes back partially wrong over mooncake PD. + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: op + required_to_run: true + evidence: >- + Verified on this base directly — 2x 8x MI325X, GLM-5.2-FP8 1P1D, needle retrieval + 5/9 -> 9/9. + +alive_because: + reason: no-upstream-pr + detail: >- + Not submitted upstream. The closest existing report, #25583, describes the identical + corruption shape but on an AGGREGATED server with no PD and no mooncake, so a shared + root cause is unestablished; it was auto-closed inactive with no follow-up. The + aggregated-vs-PD A/B this patch rests on is exactly what that issue was missing. + consumers: + - Dockerfile.sglang (any PD deployment with chunked prefill over mooncake) + - Dockerfile.sglang.gfx942 (same) + drop_when: >- + A base sglang synchronizes on the completion event prefill.py already records — the + script then reports "already present" and no-ops. + drop_signal: self-guard-marker-skips + silent_misapply_risk: null + +history: + born: + date: 2026-07-31 + commit: ea07d53 + subject: "fix(image): apply the mooncake PD wait-event patch at image build time" + last_modified: + date: 2026-07-31 + commit: ea07d53 + subject: "fix(image): apply the mooncake PD wait-event patch at image build time" + reason: >- + Moved into the image build in the same change that introduced it, so a PD + deployment cannot silently run without it. Preceded by 786f238, which carried the + patch itself. + +upstream_issues: + - ref: sgl-project/sglang#25583 + url: https://github.com/sgl-project/sglang/issues/25583 + kind: issue + title: "[Bug] Long text response error on GLM-5-FP8 with NSA backend" + state: CLOSED + state_reason: inactive — auto-closed 2026-07-18 with no follow-up + author: dingfen + note: >- + Same corruption shape on GLM-5, but aggregated: no PD and no mooncake. So it may + or may not share this root cause. Treat as suggestive, not confirming. + +upstream_prs: null + +related_refs: null + +problem: + what: >- + With chunked prefill over the mooncake KV transport, every prefill chunk except the + last can be RDMA-read while the forward that writes those pages is still running, so + the decode leg receives half-written KV. + why: >- + prefill.py's early-send path already records a completion event as the barrier + (req.disagg_kv_sender._early_send_wait_event), and the comment there says exactly + what it is for — but only mori/conn.py ever reads it. mooncake/conn.py has no + wait_event or synchronize() anywhere, so on mooncake the barrier has never taken + effect. Worse, the overlap-scheduling path that actually moves non-final chunks + (process_batch_result_disagg_prefill) does not even record an event. The final chunk + is always correct because it goes through the sampling path, which has a real + copy_done.synchronize(). + how: >- + Three edits, all mirroring what mori already does. TransferKVChunk gains a + wait_event field so the barrier travels with the work item; mooncake's send() picks + the event up off the sender, add_transfer_request() forwards it, and transfer_worker + synchronizes on it BEFORE reading device memory; and prefill.py's overlap non-final + chunk send records an event on forward_stream. + before_fix: >- + No crash and nothing in any log — output is PARTIALLY wrong for prompts longer than + one prefill chunk. On GLM-5.2 DSA needle-in-a-haystack returns the first digits of + the needle then degenerates: want=2183762 got='21832183218'. + The corruption boundary lands exactly on the chunk boundary. + after_fix: >- + Needle retrieval 5/9 -> 9/9 and a 29k depth sweep 4/9 -> 9/9, with logs confirming + the failing prompt is still really split into 4 chunks afterwards (so the fix is a + barrier, not an accidental disabling of chunking). + context: >- + 2x 8x MI325X (gfx942), ROCm 7.2.0, sglang v0.5.16, GLM-5.2-FP8 1P1D over mooncake + RDMA, overlap scheduling ON, --chunked-prefill-size 131072, --enable-dp-attention. + NOT DSA-specific: any PD deployment running chunked prefill over mooncake with + overlap scheduling is affected. The same model, same backends and same chunk size in + an aggregated single-node server passes 9/9, which is the A/B that localises it to + the transport. + call_chain: + - "sglang/srt/disaggregation/prefill.py :: process_batch_result_disagg_prefill (overlap path)" + - "send_kv_chunk(..., last_chunk=False) — no event recorded on this path" + - "sglang/srt/disaggregation/mooncake/conn.py :: send() -> add_transfer_request()" + - "transfer_worker reads device memory with no synchronize() — races the forward still writing those pages" + symptom_signature: >- + partially-correct long-prompt output with the error boundary on the prefill chunk + boundary; no log line, no crash + silent: true + +verification: + date: 2026-07-31 + hardware: 2x 8x MI325X (gfx942) + software: sglang v0.5.16, ROCm 7.2.0 + workload: GLM-5.2-FP8 1P1D over mooncake RDMA, overlap scheduling on, chunked prefill 131072 + result: needle 5/9 -> 9/9; 29k depth sweep 4/9 -> 9/9; chunk count unchanged at 4 + notes: >- + DSA's sparse retrieval only makes this conspicuous ("retrieved half the digits"); + on a dense model the same defect is a quiet quality drop. + +scope_limits: null + +open_actions: + - action: >- + Submit upstream. The aggregated-vs-PD A/B is the evidence #25583 lacked, and no + upstream PR exists for a correctness bug that is silent by construction. + owner: unassigned + blocked_on: null + - action: >- + Measure the added synchronize()'s cost on prefill throughput before upstreaming — + it sits on the hot chunk path. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/sglang_dsa/README.md b/deploy/docker/patches/sglang_dsa/README.md index b6c6c24a..8a2b333a 100644 --- a/deploy/docker/patches/sglang_dsa/README.md +++ b/deploy/docker/patches/sglang_dsa/README.md @@ -36,8 +36,11 @@ established, the upstream issue / third-party PR / our own PR, how it differs from our own upstream PR, and whether the IndexShare workaround substitutes for it. Read the `.diff` before changing it. -Upstream linkage for these and every other patch in the repo is indexed in -[`deploy/docker/patch.upstream.status.md`](../../patch.upstream.status.md). +Each patch also has a machine-checked record beside it — +`.upstream.status.yaml` — carrying its upstream state, per-engine +effect and drop condition. Upstream linkage for these and every other patch in +the repo is indexed in +[`deploy/docker/patch.upstream.status.yaml`](../../patch.upstream.status.yaml). ## Applying diff --git a/deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.upstream.status.yaml b/deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.upstream.status.yaml new file mode 100644 index 00000000..a4d9e857 --- /dev/null +++ b/deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.upstream.status.yaml @@ -0,0 +1,227 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.diff + kind: context-diff + idempotent: false + applied_by: + - Dockerfile.sglang + - deploy/docker/scripts/apply_sglang_dsa_patches.sh + opt_in_flag: APPLY_SGLANG_DSA_PATCHES=1 with DSA_PATCH_SET=full (Dockerfile.sglang only) + +target: + library: sglang + component: speculative-decoding + repo: sgl-project/sglang + files: + - python/sglang/srt/speculative/eagle_worker_v2.py + - python/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py + - python/sglang/srt/layers/dp_attention.py + - python/sglang/srt/model_executor/forward_batch_info.py + - python/sglang/srt/managers/schedule_batch.py + - python/sglang/srt/disaggregation/decode.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: >- + --fuzz=0 against this tag. The gfx942 v0.5.16 image substitutes a runtime + flag instead of taking this diff. + +upstream_main_affected: + value: true + evidence: >- + #32209, the PR carrying this fix, is still OPEN, and the guard it repairs was + introduced by #30839/#31083 which are merged. So main has the guard and not the fix. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + apply_sglang_dsa_patches.sh asserts five bytecode markers for this diff + (dp_attn.py:can_draft_cuda_graph, eagle_worker_v2.py:requires_dp_attention_eager_forward, + eagle_draft_cuda_graph_runner.py:can_run_dp_draft_cuda_graph, + forward_batch_info.py:can_run_dp_draft_cuda_graph, + schedule_batch.py + decode.py:force_disable_draft_cuda_graph). + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: not-applied + required_to_run: false + evidence: >- + Substituted at runtime with + --json-model-override-args '{"index_share_for_mtp_iteration":false}', which makes + the rank-dependent term-4 seed unreachable rather than making the choice a group + decision. Cheaper, narrower, and it costs the IndexShare optimisation. + +alive_because: + reason: upstream-pr-open + detail: >- + #32209 carries this same fix with the same strategy and this diff adopts its + placement verbatim. We deliberately did NOT open a competing PR — the right move is + to converge on #32209, which is better placed. It has been open and unreviewed + since before our fix existed. + consumers: + - Dockerfile.sglang (GLM-5.2 PD decode leg with DP-attention + EAGLE MTP on gfx950) + drop_when: >- + #32209 merges and lands in a base sglang we build against, or the guard introduced + by #30839 is reworked so the choice is no longer per-rank. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-08-01 + commit: c91db76 + subject: "sglang DSA: enable PD + DP-attention + EAGLE MTP for GLM-5.2 on gfx950" + last_modified: + date: 2026-08-01 + commit: c91db76 + subject: "sglang DSA: enable PD + DP-attention + EAGLE MTP for GLM-5.2 on gfx950" + reason: >- + Cut once against v0.5.15.post1 in the change that enabled the topology; no + revision needed since. + +upstream_issues: + - ref: sgl-project/sglang#32527 + url: https://github.com/sgl-project/sglang/issues/32527 + kind: issue + title: "[BUG] EAGLE + DP Attention + PD Disaggregation: Deadlock when index_share_for_mtp_iteration is enabled for GLM-5.2" + state: OPEN + state_reason: null + author: Xavier1994 + note: >- + The same defect, reported independently on 8x Blackwell / GLM-5.2-FP8 on + 2026-07-27 — two days before we fixed it — with the same analysis. It proposes a + THIRD strategy: a dummy all-zeros seed so term 4 is always false. No activity + since it was filed. + +upstream_prs: + - ref: sgl-project/sglang#32209 + url: https://github.com/sgl-project/sglang/pull/32209 + title: Fix PD decode hang with DP attention and GLM-5.2 MTP + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: HZY-Wade + ours: false + same_approach: true + approach_note: >- + Same defect, same strategy — a group-wide decision instead of a per-rank one. This + diff adopts its placement verbatim so the two do not diverge. We opened no PR of + our own on purpose: a competing PR would be noise and #32209 is better placed. + requested_action: null + in_pinned_base: false + +related_refs: + - ref: sgl-project/sglang#30839 + url: https://github.com/sgl-project/sglang/pull/30839 + kind: pr + title: "[bug-fix] Stabilize GLM-5.2 MTP IndexShare across PD and CUDA graph replay" + state: MERGED + state_reason: fixed + author: zRzRzRzRzRzRzR + relevance: introduced-the-defect + note: >- + This is where the per-rank guard came from, cherry-picked to release/v0.5.15 as + #31083. So the deadlock is a regression in our baseline, not a legacy wart. + - ref: sgl-project/sglang#31083 + url: https://github.com/sgl-project/sglang/pull/31083 + kind: pr + title: "[Cherry-pick to release/v0.5.15] Stabilize GLM-5.2 MTP IndexShare across PD and CUDA graph replay (#30839)" + state: MERGED + state_reason: fixed + author: kpham-sgl + relevance: introduced-the-defect + note: The cherry-pick that put the guard into the exact release line we pin. + - ref: sgl-project/sglang#32196 + url: https://github.com/sgl-project/sglang/pull/32196 + kind: pr + title: "[PD] Keep EAGLE DP graph and token metadata consistent" + state: OPEN + state_reason: null + author: weireweire + relevance: adjacent-site + note: Same area, different site. Not a fix for this defect. + - ref: sgl-project/sglang#32722 + url: https://github.com/sgl-project/sglang/pull/32722 + kind: pr + title: "[RED regression] Test GLM-5.2 PD + DP attention + MTP" + state: OPEN + state_reason: null + author: IvanShan177 + relevance: missing-test-coverage + note: >- + A test for exactly this topology. Its existence proves no CI covers it today, + which is why this family of defects reaches a released base. + +problem: + what: >- + EagleDraftWorker.draft() decides PER RANK whether to replay the draft CUDA graph or + run the multi-step draft eagerly. The two paths do not issue the same host-side + collective sequence, so when ranks disagree the whole DP group deadlocks. + why: >- + Two of the guard's four terms are rank-dependent by construction. `not + forward_batch.forward_mode.is_idle()` differs per rank every step. And + `draft_input.dsa_topk_indices is None` is, on the PD DECODE leg, seeded from + RDMA-shipped per-request payloads (eagle_disaggregation.py), so it is a function of + which requests a rank happens to hold. Single-node MIX never hangs because with no + disaggregation the top-k seed is produced locally by identical code on every rank, + so term 4 never flips asymmetrically. + how: >- + Make the choice a GROUP decision using one more int64 slot in the MLP-sync + all-gather the scheduler already performs, min()-reduced so any rank needing eager + takes the whole group eager. Inactive ranks contribute 1 (permissive) and so can + never drag the group into eager on their own. No new collective is introduced. + before_fix: >- + The first routed request hard-deadlocks the PD decode leg. No error, no traceback — + the group simply stops. + after_fix: >- + GLM-5.2 PD with DP-attention and EAGLE MTP serves normally on gfx950, with + IndexShare left enabled. + context: >- + gfx950 / MI355X, sglang v0.5.15.post1, GLM-5.2-FP8, PD disaggregation, DP-attention + with EAGLE MTP. Reproduces only under disaggregation; single-node mixed serving is + unaffected. Independently reported upstream on 8x Blackwell, so it is not + ROCm-specific. + call_chain: + - "sglang/srt/speculative/eagle_worker_v2.py :: EagleDraftWorker.draft()" + - "guard term 2: forward_batch.forward_mode.is_idle() — per-rank occupancy" + - "guard term 4: draft_input.dsa_topk_indices is None — seeded from RDMA payloads on the PD decode leg" + - "divergent branch -> mismatched host-side collective sequence -> group deadlock" + symptom_signature: silent hang on the first routed PD decode request, no log output + silent: false + +verification: + date: 2026-08-01 + hardware: 8x MI355X (gfx950) + software: sglang v0.5.15.post1, ROCm 7.2.0 + workload: GLM-5.2-FP8 PD, DP-attention + EAGLE MTP, IndexShare enabled + result: First-request deadlock cleared; the group takes the same path on every rank. + notes: >- + The all-gather slot rides the sync the scheduler already does, so the fix costs no + extra collective. + +scope_limits: >- + Cut --fuzz=0 against v0.5.15.post1. Not re-cut for v0.5.16 — the gfx942 image works + around the same defect with a runtime flag that disables IndexShare instead. + +open_actions: + - action: >- + Support #32209 upstream rather than opening a competing PR; add the ROCm datapoint + to that thread, and to #32527 which has had no follow-up. + owner: unassigned + blocked_on: "upstream review of #32209" diff --git a/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml b/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml new file mode 100644 index 00000000..67abcd7d --- /dev/null +++ b/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml @@ -0,0 +1,196 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff + kind: context-diff + idempotent: false + applied_by: + - Dockerfile.sglang + - deploy/docker/scripts/apply_sglang_dsa_patches.sh + opt_in_flag: APPLY_SGLANG_DSA_PATCHES=1 with DSA_PATCH_SET=full (Dockerfile.sglang only) + +target: + library: sglang + component: dsa + repo: sgl-project/sglang + files: + - python/sglang/srt/layers/attention/dsa_backend.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: >- + Applied with `patch -p1 --fuzz=0`, cut against this tag. It cannot apply to + the gfx942 image's v0.5.16 base and is deliberately not attempted there. + +upstream_main_affected: + value: true + evidence: >- + No upstream PR exists for 2a at all, and #32209 (the only PR touching 2b) is still + OPEN, so neither half is on main. Established by search plus the absence of any + merged candidate, not by reading main — weaker than this repo's usual standard. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + apply_sglang_dsa_patches.sh asserts the dsa_backend.py:_glm52_match_page_table_rows + bytecode marker and patch 2a's source marker under DSA_PATCH_SET=full. + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: not-applied + required_to_run: false + evidence: >- + DSA_PATCH_SET=indexer skips it: the diff is --fuzz=0 against v0.5.15.post1 and + will not apply to v0.5.16. That image substitutes 2b at runtime with + --json-model-override-args '{"index_share_for_mtp_iteration":false}' and does not + address 2a at all — 2a's deadlock has not been observed on that base. + +alive_because: + reason: no-upstream-pr + detail: >- + 2a has no upstream issue and no upstream PR. 2b has one third-party PR (#32209) + that solves the same row mismatch by trimming q/top-k instead of expanding the page + table; porting that half here fails at concurrency 32 and is unresolved, so we + cannot simply adopt it. + consumers: + - Dockerfile.sglang (GLM-5.2 DSA PD decode leg with DP-attention + MTP) + drop_when: >- + A base sglang removes the host sync from the DP-divergent branch (2a) and makes the + decode page table token-major under MTP (2b) — or #32209's trimming approach is + made to work at concurrency 32. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-08-01 + commit: c91db76 + subject: "sglang DSA: enable PD + DP-attention + EAGLE MTP for GLM-5.2 on gfx950" + last_modified: + date: 2026-08-01 + commit: c91db76 + subject: "sglang DSA: enable PD + DP-attention + EAGLE MTP for GLM-5.2 on gfx950" + reason: >- + Born and last touched in the same change — the diff was cut once against + v0.5.15.post1 and has not needed a revision. + +upstream_issues: null + +upstream_prs: + - ref: sgl-project/sglang#32209 + url: https://github.com/sgl-project/sglang/pull/32209 + title: Fix PD decode hang with DP attention and GLM-5.2 MTP + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: HZY-Wade + ours: false + same_approach: false + approach_note: >- + Addresses 2b's row mismatch by TRIMMING q and top-k down to the request count, + where this diff EXPANDS the page table to the token count with repeat_interleave. + Porting the trimming half here fails at concurrency 32 and the cause is + unresolved, so we kept the expansion. Nothing upstream covers 2a. + requested_action: null + in_pinned_base: false + +related_refs: + - ref: sgl-project/sglang#32722 + url: https://github.com/sgl-project/sglang/pull/32722 + kind: pr + title: "[RED regression] Test GLM-5.2 PD + DP attention + MTP" + state: OPEN + state_reason: null + author: IvanShan177 + relevance: missing-test-coverage + note: >- + Its existence proves no upstream CI covers this topology today, which is why this + whole family of defects survives in a released base. + +problem: + what: >- + Two independent defects in dsa_backend.py, both on the PD decode leg with MTP. + (2a) A blocking device-to-host sync sits on a branch only SOME DP ranks take, so + the DP collectives desynchronize and the group deadlocks. (2b) The page table has + one row per REQUEST while top-k has one row per TOKEN, so an assert kills every + rank. + why: >- + 2a — `max_seqlen_k = int(forward_batch.seq_lens.max().item())` is a host sync, and + idle DP peers keep their host mirror and take the cheap arm, so the ranks do not + issue the same collective sequence. Two further unconditional `.cpu()` syncs on the + same branch have to go too; with the first fix alone the hang persists. + 2b — `metadata.page_table_1` is per-request, while `topk_indices` has just been + padded to `q.shape[0]`, i.e. per-token. Equal for plain decode, unequal under MTP. + how: >- + 2a — replace the sync with the sync-free `self.req_to_token.shape[1]`. Over- + allocating page-table columns is safe: the table is only indexed THROUGH top-k, + which masks per row by cache_seqlens, so extra columns get -inf and are never + selected. The idiom is already established in-tree three times (triton_backend.py, + trtllm_mha_backend.py, and DSA's own graph path). The removed `.cpu()` mirrors are + dead for DRAFT_EXTEND_V2, which is not is_extend(), so every consumer is + unreachable. 2b — expand the page table with repeat_interleave to reproduce the + token-major layout, at BOTH decode call sites; the rows `_pad_topk_indices` added + hold all -1 and the triton kernel already masks them, so only the row COUNT has to + match. + before_fix: >- + 2a — the whole DP group hangs with no error. 2b — `assert page_table.shape[0] == + topk_indices.shape[0]` in transform_index_page_table_decode_fast, and every rank + dies. + after_fix: >- + GLM-5.2 DSA serves PD with DP-attention and EAGLE MTP on gfx950. Both bytecode + markers assert clean in the build. + context: >- + gfx950 / MI355X, sglang v0.5.15.post1, GLM-5.2 DSA, PD disaggregation, DP-attention + with EAGLE MTP. The traceback for 2b named only the first decode call site; the + second pairs the same unpadded page table with a padded topk_indices, so both are + patched. The prefill sibling solves 2b with an explicit output_num_tokens; the + decode entry point has no equivalent. + call_chain: + - "dsa_backend.py :: init_forward_metadata (decode, DRAFT_EXTEND_V2)" + - "2a: seq_lens.max().item() — host sync on a DP-divergent branch" + - "2b: transform_index_page_table_decode_fast — asserts page_table rows == topk rows" + symptom_signature: "assert page_table.shape[0] == topk_indices.shape[0]" + silent: false + +verification: + date: 2026-08-01 + hardware: 8x MI355X (gfx950) + software: sglang v0.5.15.post1, ROCm 7.2.0 + workload: GLM-5.2-FP8 PD, DP-attention + EAGLE MTP + result: Deadlock and assert both cleared; every consumer of the removed mirrors walked by hand. + notes: >- + 2a2's removals were established dead by reading each consumer: extend_prefix_lens_cpu + is read only inside the is_extend() arm, seq_lens_sum at the is_extend()-gated + capacity check, indexer_seq_lens_cpu is already None here, and both + _cal_indexer_k_start_end and dsa_indexer's k-only path open with + is_extend_without_speculative(). + +scope_limits: >- + Cut --fuzz=0 against v0.5.15.post1 and not re-cut for v0.5.16, so the gfx942 image + runs without it. 2a's deadlock has not been observed on that base, and 2b is worked + around there with a runtime flag rather than fixed. + +open_actions: + - action: >- + Establish why #32209's trimming approach fails at concurrency 32 here, so 2b can + converge upstream instead of diverging. + owner: unassigned + blocked_on: null + - action: File an upstream issue for 2a — the DP host-sync deadlock has no upstream record at all. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.upstream.status.yaml b/deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.upstream.status.yaml new file mode 100644 index 00000000..dac00fba --- /dev/null +++ b/deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.upstream.status.yaml @@ -0,0 +1,229 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + - deploy/docker/scripts/apply_sglang_dsa_patches.sh + opt_in_flag: APPLY_SGLANG_DSA_PATCHES=1 (default on; DSA_PATCH_SET selects the arm) + +target: + library: sglang + component: dsa + repo: sgl-project/sglang + files: + - python/sglang/srt/layers/attention/dsa_indexer.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: Dockerfile.sglang pins the tag, not a digest. + - surface: Dockerfile.sglang.gfx942 + ref: v0.5.16 + commit: fdebc938f7f4 + pinned_ref_on_main: false + release_branch: release/v0.5.16 + images: + - tag: lmsysorg/sglang:v0.5.16-rocm720-mi30x + digest: null + digest_source: unresolved + note: >- + This is the only one of the three sglang_dsa patches the gfx942 base can + take, because it anchors on source text rather than being a --fuzz=0 diff. + +upstream_main_affected: + value: true + evidence: >- + Our own PR sglang#33059 is still OPEN and unreviewed against main, i.e. the aiter + branch on main still sizes its logits from the padded row count. Not re-read from + main source on this sweep; the open PR is the evidence. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + Without it GLM-5.2 DSA with DP-attention + EAGLE MTP crashes on the very first + batch on gfx950. Verified by the 8-marker bytecode assertion in + apply_sglang_dsa_patches.sh (DSA_PATCH_SET=full). + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: op + required_to_run: true + evidence: >- + DSA_PATCH_SET=indexer asserts the dsa_indexer.py:_p1v2_trim bytecode marker. + Without it DP-attention crashes at concurrency > 1 on this base. + +alive_because: + reason: upstream-pr-open + detail: >- + Our fix is upstream as sglang#33059 but has had no review since it was opened, and + the two engine bases are release-branch pins that would not pick up a main merge + anyway. Three other open PRs touch the same two aiter call sites, so the shape of + the eventual upstream fix is still unsettled. + consumers: + - Dockerfile.sglang (GLM-5.2 DSA PD + DP-attention + MTP on gfx950) + - Dockerfile.sglang.gfx942 (GLM-5.2 DSA on gfx942, any concurrency > 1) + drop_when: >- + A base sglang reconciles the aiter paged-MQA row count with lengths — whether via + #33059, #32762 or the #31480 backend extraction. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-08-03 + commit: "1380228" + subject: "fix(image): carry the DSA indexer row fix onto the gfx942 v0.5.16 base" + last_modified: + date: 2026-08-03 + commit: 9380f07 + subject: "fix(glm5.2): agentic-serving fixes for DSA indexer, ROCm hicache and PD/kvd wiring" + reason: >- + Added the GLM52_P1V3 arm. The original revision guarded only real < padded, + assuming DP padding always makes q_fp8 longer; a DP-idle rank under MTP + draft-extend inverts that and there is nothing to trim, so the lengths side is + clipped through topk_transform's existing ke_offset instead. + supersedes: + - deploy/docker/patches/sglang_dsa/dsa_indexer_hip_dp_padded_rows.diff + +upstream_issues: null + +upstream_prs: + - ref: sgl-project/sglang#33059 + url: https://github.com/sgl-project/sglang/pull/33059 + title: Fix DSA indexer aiter (HIP) padding mismatch under DP-attention + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: dorado269 + ours: true + same_approach: true + approach_note: >- + This is the same fix, submitted upstream. It is written in the shape of the NPU + PR #32762 deliberately, so the two can converge rather than compete. + requested_action: null + in_pinned_base: false + +related_refs: + - ref: sgl-project/sglang#32762 + url: https://github.com/sgl-project/sglang/pull/32762 + kind: pr + title: "[NPU] Fix DSA eager padding mismatch in PD MTP warm-up" + state: OPEN + state_reason: null + author: stellaxcpeng + relevance: same-defect-other-platform + note: >- + Same bug class on NPU. Our diff is written in its shape; the real < padded arm is + exactly its case. + - ref: sgl-project/sglang#32738 + url: https://github.com/sgl-project/sglang/pull/32738 + kind: pr + title: "[Fix] DSA Indexer: pad heads for DeepGEMM paged MQA logits on decode/target-verify" + state: OPEN + state_reason: null + author: aqni + relevance: anchor-collision + note: >- + Pads heads at the same two aiter call sites we edit. Not a fix for our defect — + expect a conflict, not a resolution. + - ref: sgl-project/sglang#31480 + url: https://github.com/sgl-project/sglang/pull/31480 + kind: pr + title: "[DSA] Add an arch-independent torch paged-MQA-logits backend with a fused Triton fast path" + state: OPEN + state_reason: null + author: vroomfondel + relevance: anchor-collision + note: >- + Extracts the paged-MQA backend and restructures the is_aiter() dispatch our + anchors sit in. If this lands first the patch has to be rewritten, not rebased. + - ref: sgl-project/sglang#30378 + url: https://github.com/sgl-project/sglang/pull/30378 + kind: pr + title: "[DSA] Re-enable fused top-k v2 for MTP: clamp padded-row seq_lens to >= 0" + state: MERGED + state_reason: fixed + author: DarkSharpness + relevance: background + note: >- + Already in the base. Clamps padded-row seq_lens VALUES; this patch fixes the + HIP-side row COUNT. Different defect, adjacent code. + +problem: + what: >- + The aiter (HIP) paged-MQA branch of the DSA lightning indexer sizes its logits + output from one row count while `lengths` is sized from the other, so fast_topk_v2 + is handed a score tensor and a lengths tensor that disagree. + why: >- + Under DP-attention the two counts diverge in BOTH directions. When q_fp8 carries DP + padding the real count is smaller, which is the case every CUDA backend already + handles by taking a q_offset= and slicing internally; ROCm's aiter entry point has + no such parameter, so nothing slices. On a DP-idle rank under MTP draft-extend the + inequality inverts: q_offset (= sum of dsa_extend_len_cpu) is 2 while only 1 row is + materialized in q_fp8, so there are FEWER query rows than lengths entries and a + trim cannot help. + how: >- + Reconcile both counts to min(real, padded). When q_fp8 is the longer side, trim + q/weights to the real count and restore the padding after topk_transform — the + contract the CUDA backends already honour. When q_fp8 is the shorter side, clip the + lengths instead, through topk_transform's existing ke_offset parameter. `_p1v2_clip` + and `_p1v2_rows` are bound BEFORE the is_aiter() test because the former is read + unconditionally at the topk_transform call. + before_fix: >- + RuntimeError "Expected lengths.size(0) == B to be true, but got false" — on gfx950 + at the very first batch, on gfx942 at concurrency > 1. In the inverted (P1V3) + direction the scheduler rank dies and the router drops to `active_workers: 1`. + after_fix: >- + GLM-5.2 DSA runs with DP-attention and EAGLE MTP on both bases. The debug hook + SGLANG_DEBUG_DSA_ROWS=1 shows the reconciled counts, including the idle-rank case + mode=IDLE q_fp8=(1,32,128) q_offset=2 -> mqa_q=(1,32,128) with lengths clipped. + context: >- + ROCm-specific: gfx950 (MI355X) and gfx942 (MI325X), GLM-5.2 DSA, DP-attention with + EAGLE MTP. CUDA was never affected because those backends take q_offset= and slice + internally. The GLM52_P1V2 edit sites are byte-identical on v0.5.15.post1 and + v0.5.16; the later GLM52_P1V3 anchor has only been re-read against v0.5.15.post1. + call_chain: + - "sglang/srt/layers/attention/dsa_indexer.py :: forward (aiter/HIP branch)" + - "aiter paged-MQA logits kernel — sizes its output from q_fp8.shape[0]" + - "fast_topk_v2 / topk_transform — asserts lengths.size(0) == B" + symptom_signature: "Expected lengths.size(0) == B to be true, but got false" + silent: false + +verification: + date: 2026-08-03 + hardware: 8x MI355X (gfx950) and 2x 8x MI325X (gfx942) + software: sglang v0.5.15.post1 and v0.5.16, ROCm 7.2.0 + workload: GLM-5.2-FP8, DSA attention, --enable-dp-attention with EAGLE MTP + result: >- + Crash at first batch (gfx950) / conc>1 (gfx942) becomes a clean run. Captured the + inverted direction live with the patch's own SGLANG_DEBUG_DSA_ROWS=1 hook. + notes: >- + Do NOT reintroduce a `0 < q_offset` lower bound on the trim arm — an earlier + revision had one and it made fast_topk_v2 assert on exactly the idle ranks. + +scope_limits: >- + The GLM52_P1V3 anchor (the bare topk_transform call) has been re-read against + v0.5.15.post1 only, not v0.5.16. + +open_actions: + - action: "Get sglang#33059 reviewed, or converge it onto #32762 if that lands first." + owner: dorado269 + blocked_on: upstream review diff --git a/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml new file mode 100644 index 00000000..03303897 --- /dev/null +++ b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml @@ -0,0 +1,203 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + opt_in_flag: null + +target: + library: sglang + component: hicache + repo: sgl-project/sglang + files: + - python/sglang/srt/mem_cache/pool_host/common.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: gfx950 — where the fault was actually observed. + - surface: Dockerfile.sglang.gfx942 + ref: v0.5.16 + commit: fdebc938f7f4 + pinned_ref_on_main: false + release_branch: release/v0.5.16 + images: + - tag: lmsysorg/sglang:v0.5.16-rocm720-mi30x + digest: null + digest_source: unresolved + note: >- + Carried preventively here. MI300X measures the host and device addresses + EQUAL, so this image has no fault to fix. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05 (contents API, pool_host/common.py): + ALLOC_MEMORY_FUNCS still overrides only "npu" and "musa", with no HIP entry, over a + defaultdict whose default is alloc_with_host_register. Main is affected. This is a + source read, not a search miss. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + Without it the first kvd write-back aborts the process with a GPU memory access + fault. Reproduced standalone with the exact write-back kernel. + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: op + required_to_run: false + evidence: >- + It changes the allocator here too, but preventively: MI300X (amdgpu 6.14.14, ROCm + 7.2.0) measures hipHostRegister's device pointer EQUAL to the host VA, so the + fault does not occur on this arch. Do not read this row as evidence the fault was + seen on gfx942. + +alive_because: + reason: no-upstream-pr + detail: >- + No upstream issue and no upstream PR — and none of ours either, which is the gap + worth closing here. Upstream main is affected, established by reading the file, so + this is not a case of "probably already fixed". + consumers: + - Dockerfile.sglang (kvd / hierarchical cache on gfx950 — required) + - Dockerfile.sglang.gfx942 (preventive) + drop_when: >- + A base sglang routes HIP to alloc_with_pin_memory in ALLOC_MEMORY_FUNCS. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-08-03 + commit: 9380f07 + subject: "fix(glm5.2): agentic-serving fixes for DSA indexer, ROCm hicache and PD/kvd wiring" + last_modified: + date: 2026-08-04 + commit: 71fe42c + subject: "fix(sglang-rocm): make hicache survive its first write-back on gfx942" + reason: >- + Carried onto the gfx942 image alongside the staged write-back patch, and its + header corrected: the fault is gfx950-only so far, so this one is preventive on + MI300X rather than a fix for an observed crash. + +upstream_issues: null + +upstream_prs: null + +related_refs: + - ref: sgl-project/sglang#23361 + url: https://github.com/sgl-project/sglang/pull/23361 + kind: pr + title: "[MUSA][19/N] Support HiCache with pin_memory allocator" + state: MERGED + state_reason: fixed + author: yafengio + relevance: shape-we-copied + note: >- + The same one-line dispatch override for the same reason, for MUSA. This patch is + written in its shape, which is also the argument that upstream would accept a HIP + entry. + - ref: sgl-project/sglang#32503 + url: https://github.com/sgl-project/sglang/pull/32503 + kind: pr + title: "[XPU] Enable HiCache support on Intel XPU" + state: OPEN + state_reason: null + author: kpjeeja + relevance: anchor-collision + note: Touches the same ALLOC_MEMORY_FUNCS dict. Expect a conflict, not a fix. + - ref: sgl-project/sglang#32792 + url: https://github.com/sgl-project/sglang/pull/32792 + kind: pr + title: "[XPU]Enable HiSparse hierarchical sparse KV cache on Intel XPU" + state: OPEN + state_reason: null + author: Amrutha-M05 + relevance: anchor-collision + note: Same dict again. + +problem: + what: >- + sglang allocates every hierarchical-cache host pool with anonymous mmap + + hipHostRegister, then stores the resulting HOST virtual addresses in a device-side + pointer table that a GPU kernel dereferences. On ROCm those host VAs are not + dereferenceable from the device. + why: >- + On CUDA, cudaHostRegister maps the pages at the SAME address in the device address + space, so a host VA is directly usable from a kernel. On ROCm, hipHostRegister maps + them at a DIFFERENT device address which you must obtain via + hipHostGetDevicePointer. Measured on gfx950 / ROCm 7.2.0 with an 8 MiB buffer: + pin_memory gives host==devPtr, while mmap+hipHostRegister gives + host=0x7d3bee790000 devPtr=0x7d3bede00000 — and every register-flag variant + (Mapped, Portable|Mapped, MAP_PRIVATE) also differs. gfx950 reports xnack-, so + there is no page-migration path to paper over it. + how: >- + Register ROCm in ALLOC_MEMORY_FUNCS to use alloc_with_pin_memory — torch + pin_memory=True, hipHostMalloc underneath — which returns memory whose device + pointer IS the host pointer. Exactly what the existing "npu" and "musa" entries do, + for the same reason. This is the smallest change that makes the existing + pointer-table design correct on ROCm; translating every host pointer through + hipHostGetDevicePointer is the right upstream shape but touches roughly ten call + sites and is not something to apply blind to a running container. + before_fix: >- + "Memory access fault by GPU node-2 (Agent handle: ...) on address . Reason: + Unknown." on the first kvd write-back. The fault address equals the host pointer + exactly, which is the fingerprint. + after_fix: >- + Standalone repro of the exact write-back kernel — 78 layers, 7.33 GB host indexer + buffer, page ranges head / tail / last — goes from a fault on every mmap variant to + ALL OK with pin_memory. + context: >- + gfx950 / MI355X, ROCm 7.2.0, sglang hierarchical cache with a DSA model + (DSAIndexerPoolHost). MI300X measures the two addresses equal, so the fault is + gfx950-only so far. Cost of the fix: the hugepage path in alloc_mmap + (SGLANG_HUGEPAGE_SIZE) is bypassed on ROCm; that env var is unset here. + call_chain: + - "sglang/srt/mem_cache/pool_host/common.py :: ALLOC_MEMORY_FUNCS[device] -> alloc_with_host_register" + - "alloc_mmap(dims, dtype) then cudaHostRegister(buffer.data_ptr(), n, 0)" + - "memory_pool_host.py :: DSAIndexerPoolHost.init_kv_buffer builds index_k_data_ptrs from host data_ptr()s on the GPU" + - "transfer_kv_all_layer_mla(dst_layers=self.index_k_data_ptrs, ...) — kernel dereferences a host VA" + symptom_signature: "Memory access fault by GPU node-N ... on address " + silent: false + +verification: + date: 2026-08-03 + hardware: MI355X (gfx950, xnack-) and MI300X (amdgpu 6.14.14) for the negative control + software: ROCm 7.2.0, sglang v0.5.15.post1 + workload: standalone repro of the hicache write-back kernel, 78 layers, 7.33 GB host indexer buffer + result: mmap+hipHostRegister faults on every variant; pin_memory is clean on all page ranges + notes: >- + The address-equality measurement is the cheap way to tell whether a given arch is + affected — run it before assuming a new GPU needs this. + +scope_limits: >- + Fixes the allocator, not the design. The pointer tables still hold raw host pointers; + any future pool that builds a device-side table the same way is correct only because + pin_memory happens to return matching addresses. + +open_actions: + - action: >- + File the upstream PR. This is the only crash-class fix in the tree with no upstream + PR at all, upstream main is affected, and #23361 is the merged precedent to copy. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.upstream.status.yaml b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.upstream.status.yaml new file mode 100644 index 00000000..c433c436 --- /dev/null +++ b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.upstream.status.yaml @@ -0,0 +1,238 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + opt_in_flag: null + +target: + library: sglang + component: hicache + repo: sgl-project/sglang + files: + - python/sglang/srt/mem_cache/pool_host/mla.py + versions: + - surface: Dockerfile.sglang.gfx942 + ref: v0.5.16 + commit: fdebc938f7f4 + pinned_ref_on_main: false + release_branch: release/v0.5.16 + images: + - tag: lmsysorg/sglang:v0.5.16-rocm720-mi30x + digest: null + digest_source: unresolved + note: >- + The base that actually needs this. #28534 landed the HIP enablement after + v0.5.15.post1 was cut, so this defect exists only from v0.5.16 onward. + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: >- + No pool_host/mla.py exists at this tag — MLA and MHA had not yet moved out of + memory_pool_host.py — so the patch tolerates the absent file and no-ops. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: pool_host/mla.py still says + `self.can_use_write_back_jit = (_is_cuda or _is_hip) and can_use_write_back_jit_kernel(...)` + while DSAIndexerPoolHost and both DeepSeekV4 host pools still gate on _is_cuda alone. + The two gates still disagree on ROCm, so main is affected. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: op + required_to_run: true + evidence: >- + Without it the v0.5.16 gfx942 base kills the prefill scheduler on the first reused + prefix, so the image cannot run kvd at all. Measured on MI300X with GLM-5.2-FP8, + DSA, dp8 and the default hicache flags. + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: no-op + required_to_run: false + evidence: >- + Nothing to fix on this base: every can_use_write_back_jit gate at v0.5.15.post1 is + still _is_cuda (MHA, MLA, both V4 pools and DSAIndexerPoolHost, all in + memory_pool_host.py), and _is_hip appears only in the kernel import guard, so the + group's AND and its anchor agree on False. pool_host/mla.py does not exist and the + script exits 0 on that. + +alive_because: + reason: upstream-pr-open + detail: >- + The upstream repair is #30350, which is OPEN with CHANGES_REQUESTED. Note the + inversion: the MERGED PR here (#28534) is what INTRODUCED the disagreement, so this + patch does not drop on a merge — it drops on #30350. #28534's own reasoning argues + for the opposite repair, teaching the remaining pools the JIT rather than gating MLA + down, so upstream will likely close this differently than we did. + consumers: + - Dockerfile.sglang.gfx942 (kvd / hierarchical cache — required to serve at all) + drop_when: >- + A base sglang stops gating DSAIndexerPoolHost on _is_cuda alone — i.e. #30350 or an + equivalent lands. check_group_still_poisoned() tests that precondition and refuses. + drop_signal: precondition-check-refuses + silent_misapply_risk: >- + Anchor drift cannot be the signal here. #30350 never touches pool_host/mla.py, so + our anchor would keep matching and the patch would keep applying on top of the fix. + That is not a crash — both gates read False again — but it silently forfeits the + staged kernel #30350 enables. Hence the precondition check rather than an anchor + check: once DSAIndexerPoolHost stops gating on _is_cuda alone, the script exits 1 and + tells the operator to drop it. + +history: + born: + date: 2026-08-04 + commit: 71fe42c + subject: "fix(sglang-rocm): make hicache survive its first write-back on gfx942" + last_modified: + date: 2026-08-04 + commit: 71fe42c + subject: "fix(sglang-rocm): make hicache survive its first write-back on gfx942" + reason: >- + Written with check_group_still_poisoned() from the start, precisely because the + upstream fix in flight does not touch our anchor and a plain anchor check would + let the patch outlive its reason silently. + +upstream_issues: null + +upstream_prs: + - ref: sgl-project/sglang#30350 + url: https://github.com/sgl-project/sglang/pull/30350 + title: Add HiCache JIT test and benchmark for ROCm/HIP CI support + state: OPEN + review_decision: CHANGES_REQUESTED + merged_at: null + author: Emmanuel0612 + ours: false + same_approach: false + approach_note: >- + Opposite direction, and better. It adds _is_cuda_alike = _is_cuda or _is_hip and + flips the three CUDA-only gates (DSAIndexerPoolHost, DeepSeekV4PagedHostPool, + DeepSeekV4StateHostPool) so the group AND stops reading False on ROCm — including + the V4 stack our patch does not cover. It also teaches staged_write_back.cuh to + accept kDLROCM/kDLROCMHost, which is the TensorMatcher check that emits our crash, + and adds an AMD CI lane. We gate MLA down instead because that is the change that + is safe to apply to a running container; #30350 is the fix that should land. + requested_action: >- + HaiShaw approved on 2026-07-08 then requested changes on 2026-07-13. The conflict + that accompanied the request was cleared the same day and the branch last moved + 2026-07-16, but no re-review was requested, so the block still stands. Our MI300X + datapoint was posted to the thread by llying-001 on 2026-08-04 — the thread's only + non-MI355X evidence, and what this row previously listed as our open action. + in_pinned_base: false + - ref: sgl-project/sglang#28534 + url: https://github.com/sgl-project/sglang/pull/28534 + title: "[AMD] Enable JIT staged HiCache write-back and fix CPU-index crash" + state: MERGED + review_decision: REVIEW_REQUIRED + merged_at: 2026-07-09 + author: AMD-yanfeiwang + ours: false + same_approach: false + approach_note: >- + This is the PR that CREATED the defect, not one that fixes it. It added the HIP + enablement to pool_host/mla.py and pool_host/mha.py and aligned + cache_controller.py and memory_pool_host.py, but never touched the three CUDA-only + pools that share a HostPoolGroup with them. A merged PR that a local patch exists + to undo. + requested_action: null + in_pinned_base: true + +related_refs: null + +problem: + what: >- + Two gates decide one thing and disagree on ROCm, so the hicache controller puts the + destination indices on the GPU while the MLA pool launches the JIT kernel that + requires them on the host. The first write-back kills the scheduler. + why: >- + pool_host/mla.py opts HIP into the staged write-back JIT + (`(_is_cuda or _is_hip) and can_use_write_back_jit_kernel(...)`), and pool_host/mha.py + carries the identical enablement — #28534 did both. But the pools that can share a + HostPoolGroup with them do not: DSAIndexerPoolHost, which any DSA model such as + GLM-5.2 always instantiates alongside the MLA pool, plus DeepSeekV4PagedHostPool and + DeepSeekV4StateHostPool, still gate on _is_cuda alone and read False on ROCm. + HostPoolGroup ANDs the flag over its entries, so one CUDA-only member makes the + group flag False while its ANCHOR entry — the MLA pool — still says True. The two + then feed different decisions: hybrid_cache_controller.start_writing() reads the + GROUP flag, concludes the JIT will not run, and calls move_hybrid_indices() so the + destination indices end up on the GPU; HostPoolGroup.backup_from_device_all_layer() + delegates to the ANCHOR pool, whose own flag is True, so the JIT launches anyway. + how: >- + Make the anchor agree with the group: gate the staged JIT on CUDA only, exactly as + the three CUDA-only pools already spell it. The MLA pool then falls through to + transfer_kv_all_layer_mla_lf_pf — the non-JIT kernel in the same branch, which + asserts its destination indices ARE on the GPU, which is where the controller just + put them. Both gates read False on ROCm and the two decisions match. + before_fix: >- + "tvm.error.InternalError: Tensor match failed for Tensor<1152>[strides=<1>, + dtype=int64, device=rocm:0] at jit_kernel/csrc/kvcacheio/staged_write_back.cuh:248 - + Root cause: Device value [rocm:0] not in the allowed options: [cpu, rocm_host]" and + "Subprocess scheduler_0 crashed with exit code -3", on the first write-back after the + first reusable prefix. + after_fix: >- + The gfx942 image serves kvd. On any model that reaches this crash the fix costs + nothing that worked before: DSAIndexerPoolHost is CUDA-gated, so the group flag was + already False on ROCm for every DSA model and the staged JIT was unreachable + regardless. + context: >- + MI300X (gfx942), ROCm 7.2.0, sglang v0.5.16, GLM-5.2-FP8, DSA attention, dp8, with + --hicache-mem-layout page_first --hicache-io-backend kernel (both defaults). Not + specific to kvd: any hierarchical cache on this path dies the moment it writes a + page back, so the engine survives startup and then crashes on the first real request. + call_chain: + - "hybrid_cache/hybrid_cache_controller.py :: start_writing() reads the GROUP flag (False on ROCm) -> move_hybrid_indices() puts dst indices on the GPU" + - "HostPoolGroup.backup_from_device_all_layer() delegates to the ANCHOR pool" + - "mem_cache/pool_host/mla.py :: backup_from_device_all_layer reads its OWN flag (True) -> launches the JIT" + - "jit_transfer_hicache_all_layer_mla_staged_lf_pf -> staged_write_back.cuh:248 TensorMatcher rejects device=rocm:0" + symptom_signature: "Device value [rocm:0] not in the allowed options: [cpu, rocm_host]" + silent: false + +verification: + date: 2026-08-04 + hardware: MI300X (gfx942) + software: sglang v0.5.16 and current upstream main copies, ROCm 7.2.0 + workload: GLM-5.2-FP8, DSA, dp8, hicache page_first + kernel io-backend + result: >- + Exercised against the v0.5.16 AND current-main copies of both files on throwaway + trees. Stock: applies, exit 0. Re-run: "already applied", exit 0. #30350 simulated by + flipping the three CUDA-only gates: refuses, exit 1, with mla.py byte-identical to + pristine. pool_host/mla.py removed (the v0.5.15.post1 shape): tolerated, exit 0. + Anchor or pool renamed: exit 1. + notes: >- + The five-case exercise is the point — a patch whose drop signal is a precondition + check is only trustworthy if the refusal path is tested. + +scope_limits: >- + This patch gates pool_host/mla.py ONLY. DSAIndexerPoolHost is not the only CUDA-only + member that can poison the group's AND: build_deepseek_v4_hicache_stack puts + DeepSeekV4PagedHostPool in a group anchored by LogicalHostPool, whose flag is + unconditionally True, so expect the same crash on a V4 hicache stack on gfx942. No V4 + stack runs on this branch, so that gate would be untested; the script's SCOPE section + records it for whoever gets there. #30350 does cover the V4 pools. + +open_actions: + - action: >- + Nudge #30350 for re-review — it is CHANGES_REQUESTED with the conflict long cleared + and our MI300X datapoint already on the thread. Landing it retires this patch and + also covers the V4 pools we do not. + owner: llying-001 + blocked_on: HaiShaw re-review diff --git a/deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.upstream.status.yaml b/deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.upstream.status.yaml new file mode 100644 index 00000000..749642d1 --- /dev/null +++ b/deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.upstream.status.yaml @@ -0,0 +1,129 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + opt_in_flag: BUILD_AITER=1 (build ARG; default 1 — the patch targets the aiter this step builds) + +target: + library: aiter + component: flydsl-moe + repo: ROCm/aiter + files: + - aiter/ops/flydsl/kernels/moe_gemm_2stage.py + versions: + - surface: Dockerfile.vllm (build_aiter_rocm.sh) + ref: v0.1.16.post1 + commit: null + pinned_ref_on_main: null + release_branch: null + unpinned_risk: >- + AITER_GIT_REF pins a tag, not a commit, and aiter is built in-container so no digest + exists. The tag is the pin. Note the patch's own header records verification against + aiter 0.1.16.post2 / flydsl 0.2.0, one patch level ABOVE the tag the Dockerfile now + builds — the two have drifted and nobody re-verified. + images: [] + +upstream_main_affected: + value: null + evidence: >- + Not established. aiter main was not read, and no search hit was found on either the + symptom or the symbol across vLLM and ROCm/aiter as of 2026-08-05. Recorded as null + rather than guessed: this is a fast-moving kernel library where a version-specific + fx.ptrtoint behaviour may simply have changed without a traceable fix. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (vllm-dsv4-patches loop) + effect: unverified + required_to_run: false + evidence: >- + Load-bearing for Kimi-K2.6 int4 W4A16 MoE when the anchors are present, and a + documented no-op for DSv4 MXFP4. Marked unverified because the patch was verified + against aiter 0.1.16.post2 while the Dockerfile now builds v0.1.16.post1, and nobody + has confirmed the anchors still match what that tag produces. + +alive_because: + reason: no-upstream-pr + detail: >- + No upstream issue and no upstream PR, ours included. Searched again on 2026-08-05 across + vLLM and ROCm/aiter, on the symptom and on the symbol. + consumers: + - Dockerfile.vllm, for Kimi-K2.6 int4 W4A16 MoE + drop_when: >- + aiter's flydsl accepts memref-typed pointer arguments in fx.ptrtoint, or the built aiter + version no longer contains the buffer-resource helper this reverts. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + The patch REVERTS gemm1/gemm2 to an older buffer_ops formulation. If aiter later changes + those kernels for reasons unrelated to the memref problem, the anchors could still match + while the revert now discards a genuine improvement. The marker guard prevents double + application but cannot detect that case. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged, while AITER_GIT_REF moved underneath it. That drift is the open action below. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + vLLM passes tensors into aiter's flydsl 2-stage MoE GEMM, which arrive as flydsl memrefs, + and aiter 0.1.16's fx.ptrtoint rejects memrefs — the kernel build dies with an MLIRError. + why: >- + The gemm1/gemm2 buffer-resource construction was changed to go through fx.ptrtoint, which + assumes a raw pointer. A memref is not one, so the conversion raises during MLIR + construction rather than at a type boundary where it would be obvious. + how: >- + Revert the gemm1/gemm2 buffer-resource builds to the memref-friendly buffer_ops calls + (create_buffer_resource / extract_base_index). gemm3 is deliberately untouched. + before_fix: MLIRError crash in the 2-stage MoE GEMM on Kimi-K2.6 int4 W4A16. + after_fix: The MoE GEMM builds and runs. No-op for DSv4 MXFP4, which does not take this path. + context: >- + Kimi-K2.6 int4 W4A16 MoE on ROCm. Verified against aiter 0.1.16.post2 with flydsl 0.2.0 on + a vLLM dev748 base — note that BOTH of those differ from what Dockerfile.vllm builds today. + call_chain: + - "vLLM MoE layer passes tensors to the aiter flydsl 2-stage GEMM" + - "aiter/ops/flydsl/kernels/moe_gemm_2stage.py :: _ptr_buffer_resource -> fx.ptrtoint(ptr)" + - "ptr is a flydsl memref -> MLIRError during MLIR construction" + symptom_signature: MLIRError from fx.ptrtoint in moe_gemm_2stage + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm + software: aiter 0.1.16.post2, flydsl 0.2.0, vLLM dev748 + workload: Kimi-K2.6 int4 W4A16 MoE + result: MLIRError cleared. + notes: >- + This verification is now stale relative to the Dockerfile — recorded as-is rather than + restated against v0.1.16.post1, which would be a claim nobody has checked. + +scope_limits: gemm1 and gemm2 only; gemm3 is left alone. + +open_actions: + - action: >- + Re-verify against the aiter version Dockerfile.vllm actually builds (v0.1.16.post1). The + patch was verified on 0.1.16.post2 and the two have drifted apart. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.upstream.status.yaml b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.upstream.status.yaml new file mode 100644 index 00000000..5fdfb735 --- /dev/null +++ b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.upstream.status.yaml @@ -0,0 +1,134 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + opt_in_flag: null + +target: + library: vllm + component: moriio + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. The + patch's own header records verification against vllm 0.23.x, two minor versions back. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: null + evidence: >- + Not established — main was not read for this one and no search hit exists. MoRIIO attracts + little third-party traffic, so absence of a report says little either way. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (vllm-dsv4-patches loop) + effect: unverified + required_to_run: false + evidence: >- + Verified load-bearing on vllm 0.23.x for DSv4-Pro MoRIIO 1P1D — without it the prefill + worker dies at KV registration. Not re-verified on the v0.25.1 base, which refactored + the MoRIIO connector enough to retire three sibling patches, so op-vs-no-op here is + genuinely open. + +alive_because: + reason: no-upstream-pr + detail: >- + Nothing upstream and nothing of ours. All four DSv4 MoRIIO patches were searched again on + 2026-08-05 across vLLM and ROCm/aiter, on the symptom and on the symbol, with no hits. + consumers: + - Dockerfile.vllm, for DeepSeek-V4 MoRIIO PD + drop_when: >- + A base vLLM drops the global block_size equality check, or the v0.25.1+ refactor is + confirmed to have removed the anchors (in which case the patch is already a no-op). + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + This patch REMOVES a check and DEMOTES a raise to a debug log. If upstream later adds a + genuinely load-bearing block-size validation at the same site, the patch would silence it. + The narrow anchors make that unlikely rather than impossible. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged across two base bumps. Its three GLM-5.1 siblings were deleted on the v0.25.1 + move for having become no-ops; this one was kept without being re-verified. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + A global block_size equality check in the MoRIIO connector kills the prefill worker at KV + registration for DeepSeek-V4, which legitimately registers per-layer caches with different + block sizes. + why: >- + The connector asserts `first_geometry.block_size == self.block_size` and raises on any + per-layer mismatch, but DSv4's KV geometry is hybrid by design — c128a sparse layers use + block size 2 while SWA and state layers use 256. The check is spurious because the actual + offset arithmetic already consults the per-layer self.block_lens dict, so nothing + downstream depends on the global equality. + how: >- + Drop the global assert and demote the per-layer mismatch raise to a debug log, leaving the + per-layer offset path untouched. + before_fix: The prefill worker dies at KV registration, so DSv4 MoRIIO PD never starts. + after_fix: DSv4-Pro serves over MoRIIO 1P1D. + context: >- + DeepSeek-V4-Pro with MoRIIOConnector 1P1D on ROCm; verified on vllm 0.23.x. Only hybrid + per-layer block-size models reach the check. + call_chain: + - "moriio_connector.py :: register_kv_caches" + - "assert first_geometry.block_size == self.block_size (global check)" + - "per-layer mismatch raise" + - "prefill worker dies before serving" + symptom_signature: block_size assertion at MoRIIO KV registration on a hybrid-KV model + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm 1P1D + software: vllm 0.23.x ROCm, MoRIIOConnector + workload: DeepSeek-V4-Pro MoRIIO PD + result: Prefill worker survives registration and PD serves. + notes: Verification predates the v0.25.1 base by two minor versions. + +scope_limits: >- + Relaxes validation only; it does not make any transfer path block-size aware, because the + per-layer path already was. + +open_actions: + - action: >- + Re-verify all four DSv4 MoRIIO patches on the v0.25.1 base. The refactor that landed + moriio_layout.py already turned three sibling patches into no-ops, and these four were + carried across on the assumption they still apply. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.upstream.status.yaml b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.upstream.status.yaml new file mode 100644 index 00000000..a6366227 --- /dev/null +++ b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.upstream.status.yaml @@ -0,0 +1,122 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + opt_in_flag: null + +target: + library: vllm + component: moriio + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_engine.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref. The patch's own header + records verification against vllm 0.23.x. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: null + evidence: Not established — main was not read, and no search hit exists as of 2026-08-05. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (vllm-dsv4-patches loop) + effect: unverified + required_to_run: false + evidence: >- + Verified load-bearing on vllm 0.23.x for DSv4-Pro 1P1D. Not re-verified on v0.25.1. + +alive_because: + reason: no-upstream-pr + detail: >- + Nothing upstream and nothing of ours. Worth noting the fix is not novel — sglang's mori + connector already registers this way, so the shape is established; only vLLM's MoRIIO + lacks it. + consumers: + - Dockerfile.vllm, for DeepSeek-V4 MoRIIO PD + drop_when: >- + A base vLLM's MoRIIO engine registers non-contiguous tensors by storage span, as sglang's + mori connector already does. + drop_signal: self-guard-marker-skips + silent_misapply_risk: null + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged across two base bumps. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + MoRI's register_torch_tensor rejects DeepSeek-V4's KV cache because it is a non-contiguous + view, and the obvious workaround — calling .contiguous() — would be silently wrong. + why: >- + DSv4's fp8_ds_mla KV cache is a 576-byte-aligned non-contiguous view over a larger buffer. + register_torch_tensor forces contiguity and raises. .contiguous() would copy to a NEW + address, so the registered memory would no longer be the buffer the model forward writes + into — registration would succeed and transfers would move stale data. + how: >- + For a non-contiguous tensor, register the storage span through the low-level + register_memory(data_ptr, storage_nbytes - storage_offset_bytes, dev, loc), using the same + base that _compute_block_transfer_offsets already uses so the offset arithmetic stays + consistent. Contiguous tensors keep the original path. This mirrors sglang's mori connector. + before_fix: >- + register_torch_tensor raises on the non-contiguous DSv4 KV view, so registration fails. + after_fix: DSv4-Pro registers its KV cache and MoRIIO 1P1D serves. + context: >- + DeepSeek-V4-Pro (fp8_ds_mla, 576B-aligned non-contiguous KV view) with MoRIIO on ROCm, + verified on vllm 0.23.x. Contiguous fp16/bf16 caches are unaffected. + call_chain: + - "moriio_engine.py :: register_torch_tensor(kv_view)" + - "non-contiguous 576B-aligned fp8_ds_mla view -> raises" + - "(.contiguous() would copy to a new address, detaching from the forward's buffer)" + symptom_signature: register_torch_tensor rejecting a non-contiguous tensor at MoRIIO registration + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm 1P1D + software: vllm 0.23.x ROCm, MoRIIOConnector + workload: DeepSeek-V4-Pro MoRIIO PD + result: Registration succeeds against the live forward buffer. + notes: >- + The .contiguous() trap is the transferable lesson: it would have "worked" and moved stale + KV, which is far worse than the exception. + +scope_limits: Non-contiguous registration only; contiguous tensors keep the stock path. + +open_actions: + - action: Re-verify on the v0.25.1 base together with the other three DSv4 MoRIIO patches. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.upstream.status.yaml b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.upstream.status.yaml new file mode 100644 index 00000000..de7505e6 --- /dev/null +++ b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.upstream.status.yaml @@ -0,0 +1,128 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + opt_in_flag: null + +target: + library: vllm + component: moriio + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref. The patch's own header + records verification against vllm 0.23.x. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: null + evidence: Not established — main was not read, and no search hit exists as of 2026-08-05. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (vllm-dsv4-patches loop) + effect: unverified + required_to_run: false + evidence: >- + Verified load-bearing on vllm 0.23.x for DSv4-Pro 1P1D. Not re-verified on v0.25.1. The + anchor is matched verbatim so drift skips rather than mis-patches. + +alive_because: + reason: no-upstream-pr + detail: Nothing upstream and nothing of ours. + consumers: + - Dockerfile.vllm, for DeepSeek-V4 sparse MLA MoRIIO PD + drop_when: >- + A base vLLM's ROCm backend selector recognises fp8_ds_mla sparse MLA, or MoRIIO stops + routing a handshake tag through the full backend-selection path. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + The patch hardcodes backend_name for one model family. If a future base changes what + backend_name means — from a P/D handshake tag to something the transfer path actually + dispatches on — a hardcoded value would be wrong rather than merely cosmetic. That + assumption is the load-bearing part of this fix and should be re-checked on a base bump. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged across two base bumps. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + MoRIIO asks the generic ROCm backend selector for a backend name purely to fill a P/D + handshake tag, and for DSv4 sparse MLA that query raises and kills the prefill worker. + why: >- + The generic selector returns ROCM_AITER_MLA_SPARSE, which does not support fp8_ds_mla, so + get_attn_backend() raises. The irony is that the return value is only used as a handshake + STRING — nothing dispatches on it — so a fatal query is being made for a cosmetic purpose. + how: >- + For DSv4 sparse MLA specifically (use_mla and hf_config.index_topk and cache_dtype + fp8_ds_mla), skip the generic query and set backend_name directly to + ROCM_FLASHMLA_SPARSE_DSV4. Every other model keeps the original path. The old code is + matched verbatim so drift skips loudly rather than mis-patching. + before_fix: >- + get_attn_backend() raises during MoRIIO setup and the prefill worker dies, so DSv4 sparse + MLA PD never starts. + after_fix: DSv4-Pro serves over MoRIIO 1P1D with the correct handshake tag. + context: >- + DeepSeek-V4-Pro sparse MLA (fp8_ds_mla, index_topk) with MoRIIOConnector 1P1D on ROCm, + verified on vllm 0.23.x. + call_chain: + - "moriio_connector.py :: setup -> get_attn_backend(...) to obtain backend_name" + - "generic ROCm selector returns ROCM_AITER_MLA_SPARSE, which lacks fp8_ds_mla -> raises" + - "prefill worker dies" + symptom_signature: get_attn_backend raising for fp8_ds_mla during MoRIIO handshake setup + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm 1P1D + software: vllm 0.23.x ROCm, MoRIIOConnector + workload: DeepSeek-V4-Pro sparse MLA MoRIIO PD + result: Prefill worker survives setup and PD serves. + notes: null + +scope_limits: >- + DSv4 sparse MLA only, gated on use_mla + index_topk + fp8_ds_mla. Other models are untouched. + +open_actions: + - action: Re-verify on the v0.25.1 base together with the other three DSv4 MoRIIO patches. + owner: unassigned + blocked_on: null + - action: >- + Confirm backend_name is still only a handshake tag on the current base — the whole fix + rests on that. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_defer_kv_register.upstream.status.yaml b/deploy/docker/patches/vllm/patch_defer_kv_register.upstream.status.yaml new file mode 100644 index 00000000..7840fc66 --- /dev/null +++ b/deploy/docker/patches/vllm/patch_defer_kv_register.upstream.status.yaml @@ -0,0 +1,162 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_defer_kv_register.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: mooncake-connector + repo: vllm-project/vllm + files: + - vllm/v1/worker/gpu_model_runner.py + - vllm/v1/worker/gpu_worker.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: true + evidence: >- + Weakest evidence of any record here. No upstream issue or PR was found by search, and + main was NOT read for this one, so "affected" is an inference from the defect never + having been reported rather than an observation. Treat as unverified until someone reads + compile_or_warm_up_model on main. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: op + required_to_run: false + evidence: >- + Load-bearing at high GPU utilisation with bare ibv_reg_mr registration; at lower util + the crash it avoids does not fire, so the reordering is inert in effect. + - engine: vllm + surface: deploy/overlay/infera-exec + effect: op + required_to_run: false + evidence: Same patch on the runtime-overlay path. + +alive_because: + reason: no-upstream-pr + detail: >- + Nothing upstream, and nothing of ours. This one is also the hardest to upstream honestly: + what we carry is a reordering that AVOIDS a crash whose mechanism is not fully explained + — the None return from one TP worker's compile_or_warm_up_model is a symptom we routed + around rather than diagnosed. + consumers: + - >- + Dockerfile.vllm and the overlay payload, for high-util Mooncake PD with bare + ibv_reg_mr registration (MOONCAKE_HIP_DMABUF=0) + drop_when: >- + The high-util boot crash is diagnosed properly and fixed at its cause, or a base vLLM + stops aggregating compilation_times in a way that a None can kill. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + The patch reorders when registration happens. If a future base changes the warmup + sequence such that the deferred point is no longer before the engine reports ready, P/D + pairing could break in a way this record's "registration still completes before ready" + claim would no longer cover. The anchors are narrow enough that a rewrite of that + function drops the patch rather than misplacing it, but the invariant is worth re-checking + on a base bump rather than assumed. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + With bare ibv_reg_mr registration, registering the Mooncake KV pool at its normal point — + inside gpu_model_runner.initialize_kv_cache, before compile_or_warm_up_model — trips a + decode-boot crash at high GPU utilisation. + why: >- + Not fully established, and this record should not pretend otherwise. The observable is + that one TP worker's compile_or_warm_up_model returns None, and vLLM's + `max(t.language_model for t in compilation_times)` aggregation then dies on it. Why + early registration makes that worker return None is unexplained; the working theory is + memory pressure from the pinned pool during warmup, which is consistent with it being + util-dependent but is not proven. + how: >- + Move the Mooncake register_kv_caches call to the very END of compile_or_warm_up_model, + after all warmup, cudagraph capture and the post-capture sampler/pooler dummy runs. + gpu_model_runner stashes the caches and the transfer group on self instead of registering; + gpu_worker performs the registration at the end. Registration still completes before the + engine reports ready, so P<->D pairing is unaffected. + before_fix: >- + "AttributeError: 'NoneType' object has no attribute 'language_model'" during decode boot + at high util. + after_fix: >- + Kimi-K2.6 at --gpu-memory-utilization 0.8 boots stably and DeepSeek-V4-Pro output is + correct, confirming the deferral does not break pairing or transfer. + context: >- + Reachable with bare ibv_reg_mr GPU registration (MOONCAKE_HIP_DMABUF=0, the default, + with host-libionic injection) at high --gpu-memory-utilization. Not observed at lower + util. + call_chain: + - "gpu_model_runner.py :: initialize_kv_cache -> kv_transfer_group.register_kv_caches(kv_caches)" + - "gpu_worker.py :: compile_or_warm_up_model — one TP worker returns None" + - "max(t.language_model for t in compilation_times) -> AttributeError on the None" + symptom_signature: "AttributeError: 'NoneType' object has no attribute 'language_model'" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm multi-node PD + software: vLLM with the Mooncake connector, bare ibv_reg_mr registration + workload: Kimi-K2.6 at --gpu-memory-utilization 0.8, and DeepSeek-V4-Pro for correctness + result: >- + Boot crash cleared and DSv4-Pro output correct, which is the check that the deferral did + not break pairing. + notes: >- + Validating correctness as well as boot was the right instinct: a reordering of KV + registration is exactly the kind of change that could boot fine and transfer nothing. + +scope_limits: >- + A workaround, not a fix — it changes WHEN registration happens, not why early registration + breaks warmup. + +open_actions: + - action: >- + Diagnose why one TP worker's compile_or_warm_up_model returns None under early + registration. Without that, this cannot be upstreamed and cannot be safely dropped. + owner: unassigned + blocked_on: null + - action: >- + Read compile_or_warm_up_model on upstream main to replace this record's inferred + upstream_main_affected with an observed one. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.upstream.status.yaml b/deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.upstream.status.yaml new file mode 100644 index 00000000..48bf45d6 --- /dev/null +++ b/deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.upstream.status.yaml @@ -0,0 +1,161 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: mooncake-connector + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref, so no upstream commit + is recorded; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: mooncake_connector.py still has + `conv, _ = cache_or_caches` at line 1678 — the exact line and line number from the + Kimi-K3 traceback. Main is affected. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: op + required_to_run: true + evidence: >- + Kimi-K3 PD cannot register KV without it — every rank raises at once during + registration. + - engine: vllm + surface: deploy/overlay/infera-exec (applied at container start over an unpatched base) + effect: op + required_to_run: true + evidence: >- + This is the surface the patch was written for: the overlay applies patches/vllm/*.py + at container start so an unpatched vendor image can still serve. + +alive_because: + reason: no-upstream-pr + detail: >- + No upstream issue and no upstream PR — including none of ours, for a crash that takes + down every rank. The nearest PR, #47638, edits the very next line and leaves the + arity assumption intact, so it will not fix this even if it merges. + consumers: + - Dockerfile.vllm and the overlay payload, for Kimi-K3 (KDA linear attention) PD + drop_when: >- + Upstream stops destructuring MambaSpec caches as a fixed 2-tuple. The patch then + reports "already applied" or fails to find its anchor and no-ops. + drop_signal: self-guard-marker-skips + silent_misapply_risk: null + +history: + born: + date: 2026-08-02 + commit: 5b31fe0 + subject: "feat(overlay): apply vendor engine patches at container start" + last_modified: + date: 2026-08-02 + commit: 5b31fe0 + subject: "feat(overlay): apply vendor engine patches at container start" + reason: >- + Born with the overlay mechanism it was written for. It was never indexed on the + previous status page, which is the gap this record system exists to prevent. + +upstream_issues: null + +upstream_prs: null + +related_refs: + - ref: vllm-project/vllm#47638 + url: https://github.com/vllm-project/vllm/pull/47638 + kind: pr + title: "[KV Connector] Normalize list/tuple KV cache input in register_kv_caches" + state: OPEN + state_reason: null + author: jianzs + relevance: adjacent-site + note: >- + Edits the very next line — it appends an `elif isinstance(cache_or_caches, (list, + tuple))` branch for Ascend compressed MLA — and leaves the MambaSpec arity + assumption alone. Our three-line anchor survives it, so the two can coexist; do not + read this PR as an incoming fix. + +problem: + what: >- + The Mooncake connector's register_kv_caches() destructures every MambaSpec layer as a + 2-tuple, so any Mamba-family layer that carries a different number of state tensors + kills KV registration on every rank. + why: >- + `conv, _ = cache_or_caches` encodes Mamba2's shape (conv state, SSM state) as if it + were the contract. It is not: MambaSpec.shapes is already a variable-length tuple of + shapes, so the two-tensor assumption belongs to the connector, not the spec. Kimi-K3 + uses KDA linear attention, whose layers carry a different count. + how: >- + Take the first state tensor — exactly what the original code kept — without + constraining how many follow: `cache_list = [cache_or_caches[0]]`. + before_fix: >- + "mooncake_connector.py:1678 in register_kv_caches / conv, _ = cache_or_caches / + ValueError: too many values to unpack (expected 2)", on every rank simultaneously + during KV registration, so PD never starts. + after_fix: Kimi-K3 registers KV and PD starts. + context: >- + Kimi-K3 with KDA linear attention over Mooncake PD. Mamba2 models were never + affected, which is why the assumption survived. + call_chain: + - "mooncake_connector.py :: register_kv_caches" + - "isinstance(layer_spec, MambaSpec) -> conv, _ = cache_or_caches" + - "KDA layer carries != 2 state tensors -> ValueError on every rank" + symptom_signature: "ValueError: too many values to unpack (expected 2)" + silent: false + +verification: + date: 2026-08-02 + hardware: AMD ROCm multi-node PD + software: vLLM v0.25.1 base with the Mooncake connector + workload: Kimi-K3 PD KV registration + result: Registration completes; the all-rank ValueError is gone. + notes: null + +scope_limits: >- + Keeps the first state tensor and no more, which is what the original code did. It does + not teach the connector to transfer the additional KDA state tensors — if those turn + out to matter for correctness, that is a separate and larger fix. + +open_actions: + - action: >- + Submit upstream. Main is affected (confirmed by source read), the fix is three + lines, and there is no competing PR. + owner: unassigned + blocked_on: null + - action: >- + Establish whether dropping the remaining KDA state tensors from the transfer is + correct, or merely non-crashing. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_moriio_pagelen.upstream.status.yaml b/deploy/docker/patches/vllm/patch_moriio_pagelen.upstream.status.yaml new file mode 100644 index 00000000..d0b06b9f --- /dev/null +++ b/deploy/docker/patches/vllm/patch_moriio_pagelen.upstream.status.yaml @@ -0,0 +1,165 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_moriio_pagelen.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: moriio + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_layout.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: >- + v0.25.1 refactored the MoRIIO connector into moriio_layout.py, which is why + three sibling patches became no-ops and were dropped while this one stayed + load-bearing. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: moriio_layout.py's MLA branch still computes + `block_len = block_size * slot_size_bytes` and `block_stride = stride[0]` from the + tensor shape. Main is affected. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: op + required_to_run: true + evidence: >- + Required for DeepSeek-V4 and GLM-5.1 over MoRIIO PD — without it the output is wrong + while direct prefill is correct. Dockerfile.vllm's step-2 header calls it out by name + as ACTIVE ON v0.25.1. + - engine: vllm + surface: deploy/overlay/infera-exec + effect: op + required_to_run: true + evidence: Same patch on the runtime-overlay path. + +alive_because: + reason: no-upstream-pr + detail: >- + No upstream issue and no upstream PR, ours included, for a silent correctness bug on + main. Searched again on 2026-08-05 across vLLM on both the symptom and the symbol. + MoRIIO attracts little third-party traffic, so read this as genuinely unreported rather + than as a stale search. + consumers: + - Dockerfile.vllm and the overlay payload, for DSv4 and GLM-5.1 MoRIIO PD + drop_when: >- + A base vLLM derives the MLA per-block geometry from spec.page_size_bytes instead of the + tensor shape. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-07-23 + commit: 78fd0ff + subject: "Fix MoRIIO MLA page-length transfer geometry on v0.25.1 (DSv4 / GLM-5.1 PD)" + last_modified: + date: 2026-07-23 + commit: 78fd0ff + subject: "Fix MoRIIO MLA page-length transfer geometry on v0.25.1 (DSv4 / GLM-5.1 PD)" + reason: >- + Written against v0.25.1's refactored moriio_layout.py and unchanged since. Its three + GLM-5.1 siblings were dropped in the same era for becoming no-ops on this base; this + one survived because the refactor did not fix the geometry. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + MoRIIO's MLA transfer path derives the per-block transfer SIZE and STRIDE from the + tensor shape rather than from the authoritative page size, so block-scaled fp8 MLA + caches move the wrong bytes and PD output is silently wrong. + why: >- + get_layer_transfer_geometry's MLA (3-dim) branch computes slot_size_bytes = latent_dim * + element_size, block_len = block_size * slot_size_bytes and block_stride = stride[0]. + The authoritative per-scheduler-block page is spec.page_size_bytes, and the + shape-derived values disagree with it in two independent ways. For DeepSeek-V4 + fp8_ds_mla (UE8M0 block-scaled, 576-byte aligned) the page is PADDED, so page_size_bytes + > block_size*inner*es and the dropped tail is exactly the per-block scale — decode + dequantizes with a stale scale, so facts garble while structure survives. For GLM-5.1 + the cache is laid out per KERNEL block of size 1 (shape[1] == 1, ~1.25M blocks) while + the scheduler pages at block_size 16, so the shape-derived values are 16x too small and + only 1/16 of each block moves, to the wrong offset — total garbage. + how: >- + Use spec.page_size_bytes, which is both alignment-aware and kernel/logical-block-ratio + aware. block_stride is in ELEMENTS, so it becomes page_size_bytes // element_size. + This is what Mooncake already does, which is why Mooncake PD is correct on the same + nodes and model — that differential is how the bug was localised. + before_fix: >- + DSv4 gives right structure with wrong facts ("The capital of France is" -> "a good + idea..."); GLM-5.1 gives total garbage ("is is is is..."). READ and WRITE modes fail + identically, and prefill queried directly is correct. + after_fix: >- + Verified 2026-07-22 on TP4 2-node MoRIIO PD at temp=0: DSv4-Pro France->Paris, + China->Beijing, PD output equal to prefill-direct. GLM-5.1-FP8 likewise correct. + context: >- + TP4 2-node MoRIIO PD, DeepSeek-V4 and GLM-5.1 (block-scaled fp8 MLA + DSA lightning + indexer, --kv-cache-dtype fp8). Byte-for-byte no-op for contiguous matched-block caches + (fp16/bf16 K/V — Qwen, Kimi), where page_size_bytes == stride[0]*es == + block_size*inner*es. The K/V (5-dim) branches are untouched; they already handle the + ratio via kernel_blocks_per_block. + call_chain: + - "moriio_layout.py :: get_layer_transfer_geometry (MLA 3-dim branch)" + - "slot_size_bytes = latent_dim * element_size; block_len = block_size * slot_size_bytes; block_stride = stride[0]" + - "MoRIIO transfer moves block_len bytes at block_stride elements — disagrees with spec.page_size_bytes" + - "decode attends over partially-transferred / mis-scaled KV" + symptom_signature: >- + PD output wrong while prefill-direct is correct; DSv4 right-structure/wrong-fact, + GLM-5.1 degenerate repetition + silent: true + +verification: + date: 2026-07-22 + hardware: TP4 across 2 nodes + software: vLLM v0.25.1, MoRIIO connector + workload: DSv4-Pro and GLM-5.1-FP8 MoRIIO PD, temp=0 + result: >- + DSv4-Pro France->Paris and China->Beijing with PD == prefill-direct; GLM-5.1-FP8 + correct. Diagnosed by differential against Mooncake on the same nodes plus live + per-layer geometry instrumentation. + notes: >- + The Mooncake differential is the transferable technique here: two connectors on the + same nodes and model, one correct, isolates the defect to the transport layer. + +scope_limits: >- + MLA (3-dim) branch only. The K/V (5-dim) branches already handle the kernel/logical ratio + and are deliberately untouched. + +open_actions: + - action: >- + Submit upstream. This is a silent correctness bug present on main with no issue and no + PR, and the fix is to use a value upstream already computes. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_moriio_write.upstream.status.yaml b/deploy/docker/patches/vllm/patch_moriio_write.upstream.status.yaml new file mode 100644 index 00000000..229eec8c --- /dev/null +++ b/deploy/docker/patches/vllm/patch_moriio_write.upstream.status.yaml @@ -0,0 +1,164 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - upstream-source-read + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_moriio_write.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: moriio + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: >- + Whether THIS base is one of the ROCm images that still hardcodes + is_producer=True is the unresolved question — see applies_to. + +upstream_main_affected: + value: false + evidence: >- + Read from upstream main on 2026-08-05: moriio_connector.py has is_producer=False at + BOTH call sites. Main is fixed. The patch's docstring records the same for + v0.22.1rc0 source. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: unverified + required_to_run: false + evidence: >- + The patch's docstring enumerates v0.20.2 / v0.21.0 / v0.22.0 / v0.22.1 as ROCm images + that still hardcode is_producer=True, but v0.25.1 postdates all of them and main was + fixed, so this base plausibly carries the fix already. Not checked. If it does, the + patch prints "already patched" and exits 0. One grep inside the image settles it. + - engine: vllm + surface: deploy/overlay/infera-exec + effect: unverified + required_to_run: false + evidence: Same question, same base. + +alive_because: + reason: released-but-infera-not-bumped + detail: >- + Fixed upstream on source (main and v0.22.1rc0) but historically absent from every + published AMD ROCm image, which is why the patch exists at all. Our base has since moved + to v0.25.1, so the likely truth is that this is now redundant — a drop candidate whose + only blocker is that nobody has looked. + consumers: + - >- + possibly none on the v0.25.1 base; required on any ROCm image at v0.22.1 or earlier + running MoRIIO WRITE (push) mode + drop_when: >- + The pinned base is confirmed to carry is_producer=False. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + The guard is `if new in src` — i.e. it keys on the FIXED text, so on a fixed base it + reports "already patched" and exits 0. Safe, but the build log line reads like success + rather than like redundancy. Note the sharper edge: if upstream ever ships a base with + a DIFFERENT count of is_producer=True occurrences, the script exits 1, which the + Dockerfile's patch loop swallows into "[vllm-patch] skipped" — so a real failure is + indistinguishable from a benign skip in the log. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged. The base has moved twice under it without the patch being re-examined, + which is precisely how a patch becomes invisible dead weight. + +upstream_issues: + - ref: internal#67 + url: https://github.com/AMD-AGI/Infera/issues/67 + kind: issue + title: "MoRIIO WRITE mode: decode addresses itself for block-allocate notification" + state: OPEN + state_reason: null + author: null + note: >- + Our tracker item, quoted in the patch docstring. Resolves against an internal tracker, + so a 404 is expected. + +# Deliberately null rather than a placeholder ref. Upstream reaches the same end state — +# is_producer=False at both call sites on main and on v0.22.1rc0 source — but the PR that +# did it was never identified, and a guessed number is worse than none. The source read +# under upstream_main_affected is the citation. +upstream_prs: null + +related_refs: null + +problem: + what: >- + In MoRIIO WRITE (push) mode the decode sends the block-allocate notification to ITSELF, + the consumer-side handler asserts, the notify thread dies and the request hangs. + why: >- + The decode calls get_peer_zmq_from_request_id(..., is_producer=True), which resolves to + the producer — itself. The receiving handler asserts get_role() == PRODUCER ("Only + prefill can get block messages") and raises, killing the moriio-notify thread. The + decode must address the PREFILL, i.e. is_producer=False. + how: >- + One-token change: is_producer=True -> is_producer=False at the single call site, guarded + on the occurrence count being exactly 1 so a layout change fails rather than guesses. + before_fix: The request hangs and the client sees HTTP 000; the notify thread is dead. + after_fix: WRITE-mode PD requests complete. + context: >- + MoRIIO WRITE (push) mode only. PULL (read) mode never takes this branch and does not + need the patch. + call_chain: + - "moriio_connector.py :: decode sends block-allocate notification" + - "get_peer_zmq_from_request_id(request.request_id, is_producer=True) -> resolves to self" + - "consumer handler asserts get_role() == PRODUCER -> moriio-notify thread dies" + - "request hangs, HTTP 000" + symptom_signature: '"Only prefill can get block messages" followed by a hung request / HTTP 000' + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm 2-node PD + software: vLLM ROCm image, MoRIIO WRITE mode + workload: MoRIIO push-mode PD request + result: Hang cleared; the notification reaches the prefill. + notes: null + +scope_limits: WRITE (push) mode only. + +open_actions: + - action: >- + Grep the pinned v0.25.1 base for is_producer= and drop this patch if it is already + False. Main is fixed and the base has moved twice since the patch was written. + owner: unassigned + blocked_on: null + - action: >- + Identify the upstream PR that fixed this, so the record cites a change rather than a + source state. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_sched_guard.upstream.status.yaml b/deploy/docker/patches/vllm/patch_sched_guard.upstream.status.yaml new file mode 100644 index 00000000..97a647d7 --- /dev/null +++ b/deploy/docker/patches/vllm/patch_sched_guard.upstream.status.yaml @@ -0,0 +1,183 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_sched_guard.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: scheduler + repo: vllm-project/vllm + files: + - vllm/v1/core/sched/scheduler.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: v1/core/sched/scheduler.py still has the bare + `assert req_id in self.requests` at both call sites, with three open issues against it + and no PR. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: op + required_to_run: false + evidence: >- + Keeps the decode EngineCore alive under concurrent PD load. Deliberately NOT + required to run: it is applied for throughput runs and should be considered + unwanted for correctness testing — see scope_limits. + - engine: vllm + surface: deploy/overlay/infera-exec + effect: op + required_to_run: false + evidence: Same patch on the runtime-overlay path. + +alive_because: + reason: no-upstream-pr + detail: >- + Three independent upstream issues describe this exact assert and all three are OPEN + with no PR against any of them. Nobody has fixed it upstream, ourselves included — + though in our case that is defensible, because what we carry is a symptom guard and + not a fix, and upstreaming a symptom guard would be the wrong contribution. + consumers: + - Dockerfile.vllm and the overlay payload, for PD throughput runs + drop_when: >- + Upstream replaces the assert with real handling of late KV-transfer-finished events — + which means fixing the race, not guarding it. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + This patch converts a crash into a warning while the underlying race still loses the + racing request's KV transfer. Carrying it on a correctness run therefore HIDES a + correctness defect. That is the risk here — not that the patch becomes stale, but that + it is applied on a run whose purpose is to surface what it suppresses. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: + - ref: vllm-project/vllm#43226 + url: https://github.com/vllm-project/vllm/issues/43226 + kind: issue + title: "assert req_id in self.requests in _update_from_kv_xfer_finished for an aborted/freed request" + state: OPEN + state_reason: null + author: null + note: The most discussed of the three (5 comments). Same assert, aborted-request trigger. + - ref: vllm-project/vllm#46240 + url: https://github.com/vllm-project/vllm/issues/46240 + kind: issue + title: "Scheduler assert when finished_recving and finished_sending arrive in one step" + state: OPEN + state_reason: null + author: null + note: Same assert, different trigger — both events in a single step. + - ref: vllm-project/vllm#49089 + url: https://github.com/vllm-project/vllm/issues/49089 + kind: issue + title: "Late KV-transfer-finished event for an already-failed request trips the scheduler assert" + state: OPEN + state_reason: null + author: null + note: >- + Same assert again. Three independent reporters and no PR is the useful signal here: + the race is real and unowned upstream. + - ref: internal#69 + url: https://github.com/AMD-AGI/Infera/issues/69 + kind: issue + title: "PD: decode EngineCore dies on assert req_id in self.requests under concurrent load" + state: OPEN + state_reason: null + author: null + note: >- + Our tracker item, quoted in the patch docstring. Resolves against an internal + tracker, so a 404 is expected rather than stale. + +upstream_prs: null + +related_refs: null + +problem: + what: >- + Under concurrent PD load the decode EngineCore dies on + `assert req_id in self.requests` in scheduler._update_from_kv_xfer_finished — a + KV-transfer-finished event arrives for a request that has already been removed. + why: >- + The scheduler assumes a KV-transfer-finished event can only refer to a live request. + It can't guarantee that: the request can be aborted, freed, or completed between the + transfer being issued and the event arriving, and both the recving and sending paths + have the same assumption. + how: >- + Skip the stale event with a warning instead of asserting, at both call sites. This is + explicitly a symptom guard, not a repair — the racing request still loses its KV + transfer. + before_fix: The decode EngineCore process dies, taking the deployment down mid-run. + after_fix: >- + The engine stays up and logs "KV xfer finished for unknown req (recving/sending); + skipping stale event". The affected request is still wrong. + context: >- + Concurrent PD load on vLLM. Applied for THROUGHPUT runs where correctness checking is + off; it must not be applied to correctness tests, which should surface the race. + call_chain: + - "vllm/v1/core/sched/scheduler.py :: _update_from_kv_xfer_finished" + - "finished_recving / finished_sending event for a req_id already removed from self.requests" + - "bare assert -> EngineCore process death" + symptom_signature: "assert req_id in self.requests" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm multi-node PD + software: vLLM + workload: concurrent PD throughput + result: EngineCore survives; the stale event is logged and skipped. + notes: >- + "Works" here means the process stays up, which is the whole intent and also the whole + limitation. + +scope_limits: >- + Does NOT fix the underlying correctness race — the racing request still loses its KV + transfer. Apply for throughput runs; leave it off for correctness runs so the race + stays visible. + +open_actions: + - action: >- + Decide whether to pursue the real fix upstream. Three open issues and no PR means + the race is unowned; a proper repair would be a genuine contribution where this + guard would not. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.upstream.status.yaml b/deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.upstream.status.yaml new file mode 100644 index 00000000..b2231929 --- /dev/null +++ b/deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.upstream.status.yaml @@ -0,0 +1,180 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: mooncake-connector + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + No upstream commit is recorded here: the base is consumed as a published image + digest, not a git ref, so the exact vLLM commit is whatever v0.25.1 was cut from. + The digest below is the real pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: >- + vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. The tag in the ref is for readability; + the @sha256 is what pins it. + +upstream_main_affected: + value: false + evidence: >- + vllm#46807 is MERGED (2026-06-30) and landed the marker name + _physical_blocks_per_logical_kv_block on main — seven hits in its diff. Main is fixed. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: unverified + required_to_run: false + evidence: >- + Whether the pinned v0.25.1 base already carries #46807 has not been checked + directly, and it decides between op and no-op. If it does, the patch prints + "already patched" and exits; if not, it is still doing the work. Checking is one + grep inside the image and is the open action below. + - engine: vllm + surface: deploy/overlay/infera-exec (patches applied at container start) + effect: unverified + required_to_run: false + evidence: Same question, same base. + +alive_because: + reason: released-but-infera-not-bumped + detail: >- + Upstream fixed this in passing on 2026-06-30 as part of #46807 (GDN + MLA PD + support). Our own PR proposed the identical repair first and was closed as + superseded. So nothing is outstanding upstream — this is the cleanest drop + candidate in the tree, waiting only on confirmation that the pinned base carries it. + consumers: + - >- + possibly none — this is the next patch to drop, pending a grep of the pinned base + drop_when: >- + The pinned vLLM base is confirmed to carry #46807. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + Note the inversion that makes this safe: the guard marker + _physical_blocks_per_logical_kv_block is what UPSTREAM added, not what we edit. So on + a fixed base the patch self-skips and cannot double-apply — but equally there is no + `git apply` failure and no build-log complaint to tell you it has become dead weight. + The build log's "already patched" line is the only signal, and it reads like success. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged. The upstream conversation moved (our PR was superseded) without the + local patch needing an edit. + +upstream_issues: null + +upstream_prs: + - ref: vllm-project/vllm#46807 + url: https://github.com/vllm-project/vllm/pull/46807 + title: Support GDN + MLA PD with the Mooncake connector + state: MERGED + review_decision: null + merged_at: 2026-06-30 + author: null + ours: false + same_approach: true + approach_note: >- + Fixes this in passing as part of a larger change, using the same logical/kernel + ratio handle we did — the name _physical_blocks_per_logical_kv_block is literally + the marker this patch guards on. + requested_action: null + in_pinned_base: null + - ref: vllm-project/vllm#46334 + url: https://github.com/vllm-project/vllm/pull/46334 + title: "Mooncake connector: register KV at logical-page granularity" + state: CLOSED + review_decision: null + merged_at: null + author: llying-001 + ours: true + same_approach: true + approach_note: >- + Ours, proposing the identical repair, and it went in FIRST. Closed 2026-07-01 as + superseded by #46807 rather than rejected — the outcome we wanted, by a different + PR. + requested_action: null + in_pinned_base: null + +related_refs: null + +problem: + what: >- + Some attention backends force a KERNEL block size of 1 even when the user pins + --block-size 16. The KV tensor is then allocated at per-token granularity while the + scheduler still hands the connector LOGICAL page block_ids, so the connector's + address arithmetic is off by the ratio. + why: >- + The stock connector registers num_blocks = cache.shape[0] and block_len = + stride(0)*elemsize. When cache.shape[0] == num_logical_pages * ratio, the address + base + page_id*block_len lands `ratio` times too early. Affected backends are ROCm + Aiter MLA and the DeepSeek-V3.2 / GLM-5.1 DSA lightning indexer. + how: >- + Mirror vLLM's NIXL connector: compute ratio = logical_block_size // kernel_block_size + in _sync_block_size_with_kernel and register at logical-page granularity + (num_blocks //= ratio, block_len *= ratio). ratio == 1 for dense and matched-block + models, so this is byte-for-byte inert there. + before_fix: >- + RDMA transfers empty rows, the decode engine attends over all-zero prompt KV and + emits garbage from the first token. The transfer also overflows, producing + "destination transfer region exceeds remote KV block size". + after_fix: GLM-5.1 fp8 mooncake PD produces correct output from the first token. + context: >- + Observed with GLM-5.1 fp8 over mooncake PD. Applies to any backend that forces + kernel block size 1; no-op for dense or matched-block models. + call_chain: + - "mooncake_connector.py :: register_kv_caches -> num_blocks = cache.shape[0], block_len = stride(0)*elemsize" + - "scheduler hands LOGICAL page block_ids" + - "base + page_id*block_len lands ratio x too early -> empty rows transferred" + symptom_signature: "destination transfer region exceeds remote KV block size" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm multi-node PD + software: vLLM with the Mooncake connector, GLM-5.1 fp8 + workload: mooncake PD, --block-size 16 with a kernel-block-1 backend + result: Garbage-from-first-token cleared. + notes: null + +scope_limits: null + +open_actions: + - action: >- + Grep the pinned v0.25.1 base for _physical_blocks_per_logical_kv_block and, if + present, delete this patch. It is the least contentious retirement available: + merged upstream, our own repair, nothing outstanding. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.upstream.status.yaml b/deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.upstream.status.yaml new file mode 100644 index 00000000..6eb747ed --- /dev/null +++ b/deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.upstream.status.yaml @@ -0,0 +1,193 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: mooncake-connector + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: mooncake_connector.py still has NO + build_prom_metrics. Main is affected; #50374 is the candidate fix and is unreviewed. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: op + required_to_run: true + evidence: >- + Required for the PD + kvd-L3 recipe specifically: MultiConnector(InferaKvdConnector + + MooncakeConnector) with engine metrics enabled dies on the first stats collection + after the first request without it. + - engine: vllm + surface: deploy/overlay/infera-exec + effect: op + required_to_run: true + evidence: Same condition on the runtime-overlay path. + +alive_because: + reason: upstream-pr-open + detail: >- + Upstream now has a candidate, #50374, opened 2026-07-30 and still REVIEW_REQUIRED, plus + a second PR (#43836) that would solve it from the other side by making MultiConnector + skip children without metrics. Either would retire this. Neither has moved. + consumers: + - >- + Dockerfile.vllm and the overlay payload, for PD + kvd-L3 with engine metrics enabled + (i.e. without --disable-log-stats) + drop_when: >- + A base vLLM gives MooncakeConnector a build_prom_metrics (#50374), or makes + MultiConnector tolerant of children without one (#43836). + drop_signal: self-guard-marker-skips + silent_misapply_risk: null + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: null + +upstream_prs: + - ref: vllm-project/vllm#50374 + url: https://github.com/vllm-project/vllm/pull/50374 + title: Add MooncakePromMetrics and wire via build_prom_metrics + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: pavithranrao + ours: false + same_approach: true + approach_note: >- + Same shape as ours — give the connector a build_prom_metrics — but a real metrics + class rather than our no-op adapter. Better; ours only has to satisfy the + registration contract because Mooncake's telemetry already flows through + get_kv_connector_stats. + requested_action: null + in_pinned_base: false + - ref: internal#178 + url: https://github.com/AMD-AGI/Infera/pull/178 + title: "Give InferaKvdConnector build_prom_metrics" + state: MERGED + review_decision: null + merged_at: null + author: null + ours: true + same_approach: true + approach_note: >- + Ours, and the direct ancestor of this patch: it fixed the InferaKvd half of the same + bug and stopped there, so KVD=1 with stats still died. That is why the Kimi-K2.6 PD + benchmarks had to run with L3 off or stats off, and why kvd's own L3 telemetry was + invisible. Worth recording as the shape of the mistake: fixing one child of a + MultiConnector fixes nothing, because the assert is over all of them. + requested_action: null + in_pinned_base: null + +related_refs: + - ref: vllm-project/vllm#43836 + url: https://github.com/vllm-project/vllm/pull/43836 + kind: pr + title: "MultiConnector: skip children without Prometheus metrics support" + state: OPEN + state_reason: null + author: null + relevance: alternative-repair + note: >- + The opposite side of the same fix — make the container tolerant instead of making + every child comply. Either landing retires this patch. + +problem: + what: >- + MooncakeConnector implements get_kv_connector_stats but not build_prom_metrics, so + under MultiConnector the Prometheus registration contract is violated and the engine + dies on the first stats collection after the first request. + why: >- + Without its own build_prom_metrics, MooncakeConnector inherits the base classmethod + that returns None. MultiConnector.build_prom_metrics only registers children whose + build_prom_metrics returns non-None, so Mooncake never lands in the registry — and + then MultiKVConnectorPromMetrics.observe asserts that every child IS registered. + Reachable only under MultiConnector; a single-connector setup never takes that path. + how: >- + Add a build_prom_metrics returning a no-op observe() adapter, mirroring + InferaKvdPromMetrics. It invents no counters and alters no existing metric — Mooncake's + transfer telemetry already flows through get_kv_connector_stats, so only the + registration contract needs satisfying. Doing this at image-build time rather than + monkey-patching from infera's Python at run time is deliberate: the runtime approach + loses an import-order race, because MultiConnector calls Mooncake's build_prom_metrics + before the infera module that would patch it is imported. Editing the vendored file + removes the race entirely. + before_fix: >- + "AssertionError: MooncakeConnector is not contained in the list of registered + connectors with Prometheus metrics support: dict_keys([...])" and the engine dies. + after_fix: >- + PD + kvd-L3 runs with engine metrics on, so the vllm:*prefix_cache_* counters and + kvd's L3 telemetry are visible — they were never missing, just gated behind engine + metrics being enabled at all. + context: >- + Exactly the PD + kvd-L3 recipe: MultiConnector(InferaKvdConnector + MooncakeConnector) + with engine metrics enabled. Unreachable with a single connector, and unreachable with + --disable-log-stats, which is how the Kimi-K2.6 benchmarks worked around it. + call_chain: + - "MultiConnector.build_prom_metrics -> registers only children returning non-None" + - "MooncakeConnector inherits the base build_prom_metrics -> None -> never registered" + - "MultiKVConnectorPromMetrics.observe asserts every child is registered -> engine dies" + symptom_signature: "MooncakeConnector is not contained in the list of registered connectors with Prometheus metrics support" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm PD with kvd L3 + software: vLLM with MultiConnector(InferaKvd + Mooncake), engine metrics enabled + workload: PD + kvd-L3 with stats on + result: >- + The first-request assert is gone and L3 telemetry is visible for the first time. + notes: null + +scope_limits: >- + A no-op adapter: it satisfies registration and emits nothing of its own. If real + Mooncake Prometheus counters are wanted, that is #50374's job, not this patch's. + +open_actions: + - action: >- + Support #50374 with our datapoint rather than opening a competing PR, and note on + the thread that #43836 would also resolve it. + owner: unassigned + blocked_on: upstream review diff --git a/pyproject.toml b/pyproject.toml index e71ed3ed..ea45ce4f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -32,6 +32,9 @@ dev = [ "pre-commit>=3", "pytest>=8", "pytest-asyncio>=0.23", + # scripts/validate-patch-status.py (deploy/docker/patches/ status records) + "jsonschema>=4.18", + "PyYAML>=6", ] # all = ["amd-infera[sglang]", "amd-infera[vllm]", "amd-infera[atom]"] diff --git a/scripts/validate-patch-status.py b/scripts/validate-patch-status.py new file mode 100644 index 00000000..07595b7a --- /dev/null +++ b/scripts/validate-patch-status.py @@ -0,0 +1,288 @@ +#!/usr/bin/env python3 +"""Validate the patch ↔ upstream status records under deploy/docker/patches/. + +Three schemas, three kinds of file: + + _schema/patch.upstream.status.schema.json -> .upstream.status.yaml + _schema/patch.upstream.index.schema.json -> deploy/docker/patch.upstream.status.yaml + _schema/patch.archived.schema.json -> patches/archived/patch.archived.yaml + +Schema validation alone would let the set drift out of agreement with the tree, so +this also cross-checks them against what is actually on disk: + + * every applied patch file has a record, unless the index lists it under not_patches + * every record points at a patch file that exists, and only speaks for its own directory + * every record file on disk belongs to a patch — a rename cannot strand one + * every index entry points at a record that exists, and every record is indexed + * the index totals match the files found + * archived entries point at a file that exists, or say `deleted` and name the commit + +Dates are written unquoted in the YAML, so PyYAML hands us `datetime.date`; those +are normalised to ISO strings before validation, which means the schema's date +pattern still rejects anything written as a malformed string. + +Usage: + python3 scripts/validate-patch-status.py [--repo-root .] [-v] +""" + +from __future__ import annotations + +import argparse +import datetime as dt +import json +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + sys.exit("validate-patch-status: PyYAML is required (pip install -e '.[dev]')") + +try: + from jsonschema import Draft202012Validator, FormatChecker +except ImportError: # pragma: no cover + sys.exit("validate-patch-status: jsonschema is required (pip install -e '.[dev]')") + +PATCH_ROOT = Path("deploy/docker/patches") +SCHEMA_DIR = PATCH_ROOT / "_schema" +INDEX_PATH = Path("deploy/docker/patch.upstream.status.yaml") +ARCHIVED_PATH = PATCH_ROOT / "archived" / "patch.archived.yaml" + +RECORD_SUFFIX = ".upstream.status.yaml" +# Extensions that can carry a fix and therefore need a record. +PATCH_EXTS = {".py", ".diff", ".patch", ".sh"} + + +def normalise(node): + """Turn YAML's native dates back into ISO strings so the schema can check them.""" + if isinstance(node, dict): + return {k: normalise(v) for k, v in node.items()} + if isinstance(node, list): + return [normalise(v) for v in node] + if isinstance(node, dt.datetime): + return node.date().isoformat() + if isinstance(node, dt.date): + return node.isoformat() + return node + + +class Report: + def __init__(self, verbose: bool) -> None: + self.errors: list[str] = [] + self.verbose = verbose + + def fail(self, where: str, message: str) -> None: + self.errors.append(f"{where}: {message}") + + def ok(self, message: str) -> None: + if self.verbose: + print(f" ok {message}") + + +def load_yaml(path: Path, report: Report): + try: + return normalise(yaml.safe_load(path.read_text())) + except yaml.YAMLError as exc: + report.fail(str(path), f"not valid YAML: {exc}") + return None + + +def validate_against(schema_path: Path, doc, where: str, report: Report) -> None: + schema = json.loads(schema_path.read_text()) + validator = Draft202012Validator(schema, format_checker=FormatChecker()) + for err in sorted(validator.iter_errors(doc), key=lambda e: list(e.path)): + loc = "/".join(str(p) for p in err.path) or "(root)" + report.fail(where, f"{loc}: {err.message}") + + +def patch_files(root: Path) -> list[Path]: + """Every file under patches/ that could carry a fix, excluding schemas and archive.""" + found = [] + for path in sorted((root / PATCH_ROOT).rglob("*")): + if not path.is_file() or path.suffix not in PATCH_EXTS: + continue + rel = path.relative_to(root) + parts = rel.parts + if "_schema" in parts or "archived" in parts: + continue + found.append(rel) + return found + + +def record_for(patch: Path) -> Path: + """patches/vllm/patch_x.py -> patches/vllm/patch_x.upstream.status.yaml""" + return patch.with_suffix("").with_name(patch.stem + RECORD_SUFFIX) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--repo-root", default=".", type=Path) + ap.add_argument("-v", "--verbose", action="store_true") + args = ap.parse_args() + root = args.repo_root.resolve() + report = Report(args.verbose) + + record_schema = root / SCHEMA_DIR / "patch.upstream.status.schema.json" + index_schema = root / SCHEMA_DIR / "patch.upstream.index.schema.json" + archived_schema = root / SCHEMA_DIR / "patch.archived.schema.json" + for schema in (record_schema, index_schema, archived_schema): + if not schema.is_file(): + report.fail(str(schema.relative_to(root)), "schema missing") + if report.errors: + print("\n".join(f"FAIL {e}" for e in report.errors)) + return 1 + + # ---- index ------------------------------------------------------------- + index = load_yaml(root / INDEX_PATH, report) + if index is None: + print("\n".join(f"FAIL {e}" for e in report.errors)) + return 1 + validate_against(index_schema, index, str(INDEX_PATH), report) + + not_patches = {Path(e["path"]) for e in index.get("not_patches", [])} + indexed: dict[Path, dict] = {} + for lib, block in index.get("libraries", {}).items(): + for entry in block.get("patches", []): + indexed[Path(entry["patch"])] = {"library": lib, **entry} + + for entry in index.get("not_patches", []): + if not (root / entry["path"]).is_file(): + report.fail(str(INDEX_PATH), f"not_patches lists a missing file: {entry['path']}") + + # ---- per-patch records ------------------------------------------------- + found = patch_files(root) + records_seen: set[Path] = set() + + # A patch can be several files (a .patch plus the fragment a script appends). The + # record names them under extra_files, and they must not also demand a record of + # their own — so collect those before deciding what is missing one. + owned_extras: set[Path] = set() + on_disk_records: set[Path] = set() + for record in sorted((root / PATCH_ROOT).rglob(f"*{RECORD_SUFFIX}")): + rel_record = record.relative_to(root) + on_disk_records.add(rel_record) + doc = load_yaml(record, report) or {} + for entry in doc.get("patch", {}).get("extra_files", []): + extra = Path(entry) + # extra_files is the rest of ONE patch, so a record may only name files + # beside it. Unbounded, one line in any record excuses a patch anywhere + # in the tree from having its own — which is the whole gate. + if extra.parent != rel_record.parent: + report.fail( + str(rel_record), + f"patch.extra_files entry {extra} is outside {rel_record.parent}", + ) + continue + owned_extras.add(extra) + + for patch in found: + if patch in not_patches: + report.ok(f"{patch} (declared not-a-patch)") + continue + if patch in owned_extras: + report.ok(f"{patch} (covered as extra_files)") + continue + record = record_for(patch) + if not (root / record).is_file(): + report.fail( + str(patch), + f"no status record — expected {record}, or list it under not_patches in the index", + ) + continue + records_seen.add(record) + doc = load_yaml(root / record, report) + if doc is None: + continue + validate_against(record_schema, doc, str(record), report) + + declared = Path(doc.get("patch", {}).get("path", "")) + if declared != patch: + report.fail(str(record), f"patch.path is {declared}, file is {patch}") + for extra in doc.get("patch", {}).get("extra_files", []): + if not (root / extra).is_file(): + report.fail(str(record), f"patch.extra_files entry missing: {extra}") + if patch not in indexed: + report.fail(str(patch), "has a record but is not listed in the index") + elif Path(indexed[patch]["record"]) != record: + report.fail( + str(INDEX_PATH), + f"{patch}: record is {indexed[patch]['record']}, expected {record}", + ) + else: + report.ok(f"{patch} <-> {record}") + + for patch in indexed: + if patch not in found: + report.fail(str(INDEX_PATH), f"indexes a patch that is not on disk: {patch}") + + # The walk above only goes patch -> record, so a record whose patch was renamed or + # deleted is never loaded and never checked. Close the loop the other way. + for stale in sorted(on_disk_records - records_seen): + report.fail( + str(stale), + "does not correspond to any patch file — delete it, or restore the patch", + ) + + # ---- archived ---------------------------------------------------------- + archived_declared = Path(index["archived_record"]) + if archived_declared != ARCHIVED_PATH: + report.fail(str(INDEX_PATH), f"archived_record should be {ARCHIVED_PATH}") + archived = load_yaml(root / ARCHIVED_PATH, report) + if archived is not None: + validate_against(archived_schema, archived, str(ARCHIVED_PATH), report) + for entry in archived.get("patches", []): + src = entry.get("source", {}) + current = src.get("current_path") + where = f"{ARCHIVED_PATH}[{entry.get('name')}]" + if current == "deleted": + if not src.get("last_commit_with_file"): + report.fail( + where, + "current_path is 'deleted' but last_commit_with_file is unset, " + "so the file cannot be recovered", + ) + elif not (root / current).is_file(): + report.fail(where, f"current_path does not exist: {current}") + for extra in src.get("extra_files", []): + if not (root / extra).is_file(): + report.fail(where, f"extra_files entry missing: {extra}") + if entry.get("retired_reason") == "upstream-fixed-and-in-base" and not entry.get( + "upstream_fix" + ): + report.fail(where, "retired as upstream-fixed but upstream_fix is null") + + # ---- totals ------------------------------------------------------------ + totals = index.get("totals", {}) + active = len([p for p in found if p not in not_patches and p not in owned_extras]) + if totals.get("active_patches") != active: + report.fail( + str(INDEX_PATH), + f"totals.active_patches is {totals.get('active_patches')}, found {active}", + ) + if totals.get("records") != len(records_seen): + report.fail( + str(INDEX_PATH), + f"totals.records is {totals.get('records')}, found {len(records_seen)}", + ) + n_archived = len((archived or {}).get("patches", [])) + if totals.get("archived_patches") != n_archived: + report.fail( + str(INDEX_PATH), + f"totals.archived_patches is {totals.get('archived_patches')}, found {n_archived}", + ) + + # ---- result ------------------------------------------------------------ + if report.errors: + for err in report.errors: + print(f"FAIL {err}") + print(f"\n{len(report.errors)} problem(s)") + return 1 + print( + f"patch status OK — {len(records_seen)} record(s), " + f"{len(indexed)} indexed, {n_archived} archived" + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/unit/test_patch_status_records.py b/tests/unit/test_patch_status_records.py new file mode 100644 index 00000000..1f9d431a --- /dev/null +++ b/tests/unit/test_patch_status_records.py @@ -0,0 +1,175 @@ +############################################################################### +# Copyright (c) 2026, Advanced Micro Devices, Inc. All rights reserved. +# +# SPDX-License-Identifier: MIT +############################################################################### +"""The patch status records under deploy/docker/patches/ hold, and the gate bites. + +A patch that outlives its upstream fix is not obviously broken: it keeps applying +cleanly and the build log says nothing. The records exist so that state is written +down, and scripts/validate-patch-status.py exists so the writing-down cannot be +skipped. A validator that passes on a tree with a missing record would be worse +than none, so these tests check the REFUSALS as well as the happy path: + + * the real tree validates + * a patch added without a record fails + * a record whose date is not a date fails + * an index whose totals disagree with the tree fails + * an archived entry that claims `deleted` without a recovery commit fails + * a record that reaches outside its own directory to cover a patch fails + * a record left behind by a renamed or deleted patch fails +""" + +from __future__ import annotations + +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] +VALIDATOR = REPO_ROOT / "scripts" / "validate-patch-status.py" +PATCH_ROOT = Path("deploy/docker/patches") +INDEX = Path("deploy/docker/patch.upstream.status.yaml") +ARCHIVED = PATCH_ROOT / "archived" / "patch.archived.yaml" + +pytestmark = pytest.mark.skipif( + not VALIDATOR.is_file(), reason="validator script not present in this checkout" +) + + +def run(root: Path) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [sys.executable, str(VALIDATOR), "--repo-root", str(root)], + capture_output=True, + text=True, + ) + + +@pytest.fixture +def tree(tmp_path: Path) -> Path: + """A throwaway copy of just the parts the validator reads.""" + root = tmp_path / "repo" + (root / PATCH_ROOT.parent).mkdir(parents=True) + shutil.copytree(REPO_ROOT / PATCH_ROOT, root / PATCH_ROOT) + shutil.copy(REPO_ROOT / INDEX, root / INDEX) + (root / "scripts").mkdir() + shutil.copy(VALIDATOR, root / "scripts" / VALIDATOR.name) + return root + + +def test_repo_records_validate() -> None: + """The tree as committed passes, so a later failure means a real regression.""" + done = run(REPO_ROOT) + assert done.returncode == 0, done.stdout + done.stderr + + +def test_copied_tree_validates(tree: Path) -> None: + """Guards the fixture itself — the mutations below only mean something from a clean base.""" + done = run(tree) + assert done.returncode == 0, done.stdout + done.stderr + + +def test_patch_without_record_is_rejected(tree: Path) -> None: + (tree / PATCH_ROOT / "vllm" / "patch_brand_new.py").write_text("# no record\n") + done = run(tree) + assert done.returncode == 1 + assert "no status record" in done.stdout + + +def test_record_with_non_date_is_rejected(tree: Path) -> None: + record = tree / PATCH_ROOT / "vllm" / "patch_sched_guard.upstream.status.yaml" + record.write_text( + record.read_text().replace("status_updated: 2026-08-05", 'status_updated: "soon"') + ) + done = run(tree) + assert done.returncode == 1 + assert "status_updated" in done.stdout + + +def test_record_not_in_index_is_rejected(tree: Path) -> None: + index = tree / INDEX + text = index.read_text() + # Drop the whole entry for one patch, keeping the file valid YAML. + start = text.index(" - patch: deploy/docker/patches/vllm/patch_sched_guard.py") + end = text.index(" - patch:", start + 1) + index.write_text(text[:start] + text[end:]) + done = run(tree) + assert done.returncode == 1 + assert "not listed in the index" in done.stdout + + +def test_wrong_totals_are_rejected(tree: Path) -> None: + index = tree / INDEX + index.write_text(index.read_text().replace("active_patches: 25", "active_patches: 24")) + done = run(tree) + assert done.returncode == 1 + assert "totals.active_patches" in done.stdout + + +def test_deleted_archive_entry_needs_a_recovery_commit(tree: Path) -> None: + archived = tree / ARCHIVED + archived.write_text( + archived.read_text().replace( + "last_commit_with_file: 89c86fb", "last_commit_with_file: null", 1 + ) + ) + done = run(tree) + assert done.returncode == 1 + assert "cannot be recovered" in done.stdout + + +def test_record_pointing_at_the_wrong_patch_is_rejected(tree: Path) -> None: + record = tree / PATCH_ROOT / "vllm" / "patch_sched_guard.upstream.status.yaml" + record.write_text( + record.read_text().replace( + "path: deploy/docker/patches/vllm/patch_sched_guard.py", + "path: deploy/docker/patches/vllm/patch_somewhere_else.py", + ) + ) + done = run(tree) + assert done.returncode == 1 + assert "patch.path is" in done.stdout + + +def test_extra_files_cannot_reach_into_another_directory(tree: Path) -> None: + """extra_files is for the rest of one patch, not a way to vouch for someone else's. + + Left unbounded it is a hole straight through the gate: the new patch below needs a + record, and one line in an unrelated record would otherwise be enough to excuse it. + """ + (tree / PATCH_ROOT / "vllm" / "patch_brand_new.py").write_text("# no record\n") + record = tree / PATCH_ROOT / "atom" / "patch_gdn_pd_state_transfer.upstream.status.yaml" + record.write_text( + record.read_text().replace( + "patch:\n", + "patch:\n extra_files:\n - deploy/docker/patches/vllm/patch_brand_new.py\n", + 1, + ) + ) + done = run(tree) + assert done.returncode == 1 + assert "is outside" in done.stdout + # and the patch it tried to cover is still asked for a record of its own + assert "no status record" in done.stdout + + +def test_orphan_record_is_rejected(tree: Path) -> None: + """Nothing walks record -> patch, so an unclaimed record is never even parsed.""" + (tree / PATCH_ROOT / "vllm" / "patch_ghost.upstream.status.yaml").write_text( + "status_updated: not-a-date\n" + ) + done = run(tree) + assert done.returncode == 1 + assert "does not correspond to any patch file" in done.stdout + + +def test_renamed_patch_may_not_leave_its_record_behind(tree: Path) -> None: + """The realistic version of the above: `git mv` the patch, forget the record.""" + vllm = tree / PATCH_ROOT / "vllm" + (vllm / "patch_sched_guard.py").rename(vllm / "patch_sched_guard_v2.py") + done = run(tree) + assert done.returncode == 1 + assert "does not correspond to any patch file" in done.stdout From bb102210d07b497f2bc983ca86dcdb71ecf089a1 Mon Sep 17 00:00:00 2001 From: yihou Date: Fri, 7 Aug 2026 10:09:24 +0000 Subject: [PATCH 2/3] docs(patches): record the three upstream PRs opened for the sglang GLM-5.2 patches work.todo.md inventories all seven sglang GLM-5.2 patches against live upstream and finds exactly three with no PR of anyone's: the ROCm hicache allocator, the mooncake early-send wait event, and the DSA decode host-sync deadlock. Those are now sglang#33968, #33970 and #33973, all drafts. pr.done.md records the outcome. The three records move from `no-upstream-pr` to `upstream-pr-open`, gain an upstream_prs entry, and have their open_actions rewritten to say what is still owed. by_status moves 11/5 -> 8/8. Adapting each patch onto upstream main corrected the records that described them: - The hicache fix is NOT a one-line override in the shape of the merged MUSA PR #23361, as its header claimed. get_device() returns "cuda" on ROCm, so HIP has no key of its own; and memory_pool_host.py keys the table twice with a torch.device OBJECT, which is not dict-key-equal to "cuda" (measured: different hash), so those two pools always take the defaultdict default. Adding only a key misses them. - The DSA record credits base_spec_worker for supplying extend_seq_lens_cpu. It is eagle_worker_common.prepare_for_draft_extend:105. Also found a better upstream argument than the record had: the backend already declares needs_cpu_seq_lens = False to opt out of the D2H sync, so its eager fallback contradicts itself. - The mooncake patch's prefill.py edit also fixes mori, which reads the barrier but was never handed one on the overlap path. Neither header said so. Only the 2a half of the DSA diff went upstream. 2b overlaps sglang#32209 and rests on a concurrency-32 failure whose cause is unidentified, so that patch outlives its PR -- recorded in the index rather than left to be rediscovered. Every PR is a draft, and the records say why in the field that decides it: the MI355X and multi-node clusters were unreachable, so none has been re-run against the original fault on hardware that reproduces it. This session established equivalence and scope against the proven local fix on MI300X and nothing more. The index's verification note warns not to read these rows as "fixed upstream". One figure was corrected after the fact: `.max().item()` was first measured at 44 ms, but that was a first-call measurement carrying lazy-init cost. Warmed, it is 0.5-3 ms. Fixed here and in sglang#33973's body. The claim the PR rests on is unchanged and qualitative -- the call blocks at all, on a branch only some DP ranks take. work/ is gitignored: the validation scripts and working log are useful to re-run when the cluster returns, but they are not repo deliverables. Their conclusions are in the records and in the two reports. validate-patch-status.py passes and the edited .diff still applies --fuzz=0 to the pinned base. Signed-off-by: yihou --- .gitignore | 7 + deploy/docker/patch.upstream.status.yaml | 38 +++-- .../patch_mooncake_early_send_wait_event.py | 33 +++- ...early_send_wait_event.upstream.status.yaml | 58 +++++-- ...a_backend_dp_sync_and_page_table_rows.diff | 36 ++++- ...c_and_page_table_rows.upstream.status.yaml | 54 +++++-- .../patch_hicache_rocm_host_alloc.py | 27 +++- ...cache_rocm_host_alloc.upstream.status.yaml | 45 ++++-- pr.done.md | 150 ++++++++++++++++++ work.todo.md | 142 +++++++++++++++++ 10 files changed, 528 insertions(+), 62 deletions(-) create mode 100644 pr.done.md create mode 100644 work.todo.md diff --git a/.gitignore b/.gitignore index 1c6b4a8e..8d1a92f3 100644 --- a/.gitignore +++ b/.gitignore @@ -60,3 +60,10 @@ CLAUDE.md .spur_job_*.sh .spur_ns_*.sh spur-*.out + +# Per-task agent workspaces. These hold the working log, validation scripts and +# scratch copies for one piece of work — useful while it is in flight and to +# re-run later, but they are not repo deliverables. The conclusions that ARE +# deliverables get written into the patch records and into the reports at the +# repo root instead. +work/ diff --git a/deploy/docker/patch.upstream.status.yaml b/deploy/docker/patch.upstream.status.yaml index 3e3a75bc..fda8d661 100644 --- a/deploy/docker/patch.upstream.status.yaml +++ b/deploy/docker/patch.upstream.status.yaml @@ -15,7 +15,7 @@ # # Replaces patch.upstream.status.md (removed in the same change). schema_version: 1 -status_updated: 2026-08-05 +status_updated: 2026-08-07 verification_note: >- Every row was re-established on 2026-08-05: patch header read, `gh` queried for the @@ -24,6 +24,13 @@ verification_note: >- or inspected, so the four rows whose effect depends on what the pinned base already carries are marked `unverified` in their records rather than guessed at. Treat any row whose status_updated has drifted far from today as a prompt to re-check, not as fact. + UPDATED 2026-08-07: the three sglang rows that had no PR of ours now have one — + #33968 (hicache allocator), #33970 (mooncake wait event), #33973 (DSA host-sync, the + 2a half only). All three are DRAFTS: they were re-validated for equivalence and scope + against the proven local fix on MI300X, but the MI355X / multi-node cluster was + unreachable, so none has been re-run against the original fault on the hardware that + reproduces it. Do not read `carry-upstream-pr-open` on those rows as "fixed upstream". + Full record: pr.done.md and work.todo.md at the repo root. pinned_bases: - surface: Dockerfile.sglang @@ -128,10 +135,12 @@ libraries: record: deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml component: disaggregation engines: [sglang] - status: carry-no-upstream-fix - alive_because: not submitted upstream; main confirmed affected by source read + status: carry-upstream-pr-open + alive_because: >- + our sglang#33970 is a draft pending 2-node PD re-validation; main confirmed + affected by source read drop_signal: self-guard-marker-skips - ours_upstream_pr: null + ours_upstream_pr: sgl-project/sglang#33970 summary: >- SILENT CORRECTNESS BUG. Chunked prefill over mooncake PD RDMA-reads KV pages while the forward is still writing them, because the barrier prefill.py records is only ever read @@ -154,12 +163,13 @@ libraries: record: deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml component: dsa engines: [sglang] - status: carry-no-upstream-fix + status: carry-upstream-pr-open alive_because: >- - no upstream PR for the DP host-sync deadlock at all; the one PR touching the row - mismatch takes an approach that fails here at concurrency 32 + only HALF is upstream — our sglang#33973 carries the host-sync deadlock (2a); + the row mismatch (2b) has no PR of ours because #32209's approach fails here at + concurrency 32, so this patch outlives #33973 drop_signal: anchor-drift-fails-loudly - ours_upstream_pr: null + ours_upstream_pr: sgl-project/sglang#33973 summary: >- Two independent defects. A blocking device-to-host sync on a DP-divergent branch deadlocks the group; and the decode page table is per-request while top-k is per-token @@ -183,10 +193,12 @@ libraries: record: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml component: hicache engines: [sglang] - status: carry-no-upstream-fix - alive_because: no upstream issue and no PR, ours included; main confirmed affected + status: carry-upstream-pr-open + alive_because: >- + our sglang#33968 is a draft pending gfx950 re-validation; no third-party issue + or PR; main confirmed affected drop_signal: anchor-drift-fails-loudly - ours_upstream_pr: null + ours_upstream_pr: sgl-project/sglang#33968 summary: >- On ROCm, hipHostRegister maps pages at a DIFFERENT device address than the host VA, but hicache stores host VAs in a device-side pointer table a kernel dereferences — GPU @@ -500,8 +512,8 @@ totals: records: 25 archived_patches: 6 by_status: - carry-no-upstream-fix: 11 - carry-upstream-pr-open: 5 + carry-no-upstream-fix: 8 + carry-upstream-pr-open: 8 drop-candidate: 6 internal-engine: 3 diff --git a/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py b/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py index 0a67d8f9..2a2a2279 100644 --- a/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py +++ b/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py @@ -43,13 +43,32 @@ logs confirm the failing prompt is still really split into 4 chunks afterwards. The anchors below are present in both v0.5.15.post1 and v0.5.16. -UPSTREAM: not submitted. The closest existing report, -sgl-project/sglang#25583 (GLM-5-FP8 + NSA + 70k prompt, identical symptom), was -auto-closed with no follow-up; the aggregated-vs-PD A/B above is what it was -missing. Worth measuring when upstreaming: the new `synchronize()` blocks the -transfer worker, trading some transfer overlap for correctness. DROP THIS PATCH -once the base sglang waits on the event in mooncake — this script then reports -"already present" and no-ops. +UPSTREAM: sgl-project/sglang#33970 (OPEN, DRAFT, filed 2026-08-07 by dorado269) +-- "[PD] Make the mooncake KV transfer wait on the prefill forward that wrote +the pages". Same three edits, re-anchored onto main (which has since gained +`TransferKVChunk.staging_counted` and `_prepare_send_indices`, so the anchors +moved). The closest existing report, sgl-project/sglang#25583 (GLM-5-FP8 + NSA + +70k prompt, identical symptom), was auto-closed with no follow-up; the +aggregated-vs-PD A/B above is what it was missing, and is cited in the PR. + +STILL A DRAFT: the 2-node PD pair was unreachable when it was opened, so it has +NOT been re-run end-to-end against this branch. What WAS checked on a single +MI300X: `wait_event` defaults to None so existing construction sites are +unaffected; `send()` forwards it on BOTH arms and clears the sender's copy; and +the wait is a real barrier (event pending after record, `synchronize()` +measurably blocks, a post-barrier read on another stream sees the writes). +STILL UNMEASURED: the new `synchronize()` blocks the transfer worker, trading +some transfer overlap for correctness. The PR says so plainly rather than +implying it is free -- it is the first thing a reviewer will ask. + +WORTH KNOWING when reading the prefill.py edit: that file is transport-agnostic, +so recording the event on the overlap non-final-chunk send also closes the same +gap for MORI, which reads the field but was never handed one on that path. +nixl / ascend / fake / base do not read `_early_send_wait_event` and no sender +uses __slots__, so setting the attribute is inert for them. + +DROP THIS PATCH once the base sglang waits on the event in mooncake -- this +script then reports "already present" and no-ops. Self-locating and idempotent. All three files or none: a half-patched tree still corrupts long prompts, so an anchor that is missing or no longer unique writes diff --git a/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml b/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml index fd0c06d5..d926dece 100644 --- a/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml +++ b/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml @@ -1,6 +1,6 @@ # yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json schema_version: 1 -status_updated: 2026-08-05 +status_updated: 2026-08-07 verified_by: - gh-pr-issue-state - gh-search @@ -72,12 +72,14 @@ applies_to: 5/9 -> 9/9. alive_because: - reason: no-upstream-pr + reason: upstream-pr-open detail: >- - Not submitted upstream. The closest existing report, #25583, describes the identical - corruption shape but on an AGGREGATED server with no PD and no mooncake, so a shared - root cause is unestablished; it was auto-closed inactive with no follow-up. The - aggregated-vs-PD A/B this patch rests on is exactly what that issue was missing. + Ours is now upstream as sglang#33970 (DRAFT, opened 2026-08-07). The closest existing + report, #25583, describes the identical corruption shape but on an AGGREGATED server + with no PD and no mooncake, so a shared root cause is unestablished; it was + auto-closed inactive with no follow-up. The aggregated-vs-PD A/B this patch rests on + is exactly what that issue was missing, and is the core of the PR's argument. + Was `no-upstream-pr` until #33970. consumers: - Dockerfile.sglang (any PD deployment with chunked prefill over mooncake) - Dockerfile.sglang.gfx942 (same) @@ -113,7 +115,29 @@ upstream_issues: Same corruption shape on GLM-5, but aggregated: no PD and no mooncake. So it may or may not share this root cause. Treat as suggestive, not confirming. -upstream_prs: null +upstream_prs: + - ref: sgl-project/sglang#33970 + url: https://github.com/sgl-project/sglang/pull/33970 + title: "[PD] Make the mooncake KV transfer wait on the prefill forward that wrote the pages" + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: dorado269 + ours: true + same_approach: true + approach_note: >- + Opened 2026-08-07 as a DRAFT. Same three edits as this patch, re-anchored onto + main (which has since gained TransferKVChunk.staging_counted and + _prepare_send_indices, so the anchors moved). One behaviour worth knowing that the + local patch header does not spell out: prefill.py is transport-agnostic, so + recording the event on the overlap non-final-chunk send also closes the same gap + for MORI, which reads the field but was never handed one on that path. Checked + that nixl / ascend / fake / base do not read _early_send_wait_event and that no + sender class uses __slots__, so setting the attribute is inert for them. + requested_action: >- + Draft until re-validated on a 2-node PD pair, and until the added synchronize()'s + cost on prefill throughput is measured — the PR states plainly that it is not. + in_pinned_base: false related_refs: null @@ -177,12 +201,18 @@ scope_limits: null open_actions: - action: >- - Submit upstream. The aggregated-vs-PD A/B is the evidence #25583 lacked, and no - upstream PR exists for a correctness bug that is silent by construction. - owner: unassigned - blocked_on: null + Re-validate sglang#33970 on a 2-node PD pair and take it out of draft. What WAS + checked on a single MI300X against the PR's exact diff: the wait_event field + defaults to None so existing construction sites are unaffected; send() forwards it + on both the last-chunk and non-last-chunk arms and clears the sender's copy; and + the wait is a real barrier (event pending after record, synchronize() measurably + blocks, a post-barrier read on another stream sees the writes). What was NOT: that + the corruption is gone end-to-end. + owner: dorado269 + blocked_on: 2-node RDMA cluster access (unreachable 2026-08-07) - action: >- - Measure the added synchronize()'s cost on prefill throughput before upstreaming — - it sits on the hot chunk path. + Measure the added synchronize()'s cost on prefill throughput — it sits on the hot + chunk path. Still unmeasured; #33970 says so explicitly rather than implying it is + free, and it is the first thing a reviewer will ask. owner: unassigned - blocked_on: null + blocked_on: same cluster access diff --git a/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff b/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff index b1c9b64c..5cf671fa 100644 --- a/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff +++ b/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff @@ -50,10 +50,38 @@ third-party PR NONE for either half. Per-diff greps: #31683 does not touch Weaker than it sounds: `gh search` matches titles and bodies, not diff content, so an upstream PR could touch either site without naming it. -own PR NONE. Not upstreamed. 2a's evidence is runtime-state, not a - revert, so it is harder to present upstream than patch 01 was; - 2b's upstream counterpart would be #32209's other half, which we - could not make work here (see below). +own PR 2a ONLY -- #33973 (OPEN, DRAFT, filed 2026-08-07 by dorado269), + "[DSA] Remove the device-to-host syncs on the decode + DP-divergent branch". Carries 2a AND 2a2. 2b is deliberately + NOT in it: one fix, one PR, and 2b overlaps #32209 while resting + on a concurrency-32 failure whose cause is unidentified (below). + So THIS PATCH OUTLIVES #33973 -- landing it retires only half. + + Still a draft: gfx950 was unreachable when it was opened, so it + has NOT been re-run against the deadlock on hardware that + reproduces it. What WAS checked, on MI300X: `.max().item()` + measurably blocks (~0.5-3 ms run to run, behind 30 queued + 4096^2 matmuls) while `.shape[1]` does not (~3 us) -- the point + is that it blocks AT ALL on a branch only some ranks take, not + the magnitude; widening the page table leaves + every REAL top-k selection and score bit-identical; and + DRAFT_EXTEND_V2 is not is_extend(), so every consumer of the two + removed mirrors is unreachable. + + Two things the PR says that this header did not: + * the strongest argument for 2a is that the backend ALREADY + declares `needs_cpu_seq_lens = False`, with a comment saying + it opts out of the D2H sync -- so the eager fallback + contradicts its own contract; and + * the CPU mirror this branch relies on comes from + eagle_worker_common.prepare_for_draft_extend (line 105), on + its gpu_only branch and for this same reason -- NOT from + base_spec_worker, as stated further down in this file. + A caveat found while validating: a row with seq_len < topk does + return indices past its own seq_len, because top-k must return + topk entries. Those slots carry -inf and occur identically at + BOTH widths, so they are a consequence of seq_len < topk, not of + widening -- the property that holds is about the REAL selections. HOW 2a WAS ESTABLISHED -------------------------------------------------------------------------------- diff --git a/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml b/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml index 67abcd7d..bfdc3955 100644 --- a/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml +++ b/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml @@ -1,6 +1,6 @@ # yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json schema_version: 1 -status_updated: 2026-08-05 +status_updated: 2026-08-07 verified_by: - gh-pr-issue-state - gh-search @@ -61,12 +61,15 @@ applies_to: address 2a at all — 2a's deadlock has not been observed on that base. alive_because: - reason: no-upstream-pr + reason: upstream-pr-open detail: >- - 2a has no upstream issue and no upstream PR. 2b has one third-party PR (#32209) - that solves the same row mismatch by trimming q/top-k instead of expanding the page - table; porting that half here fails at concurrency 32 and is unresolved, so we - cannot simply adopt it. + SPLIT, and only half is upstream. 2a is now ours as sglang#33973 (DRAFT, opened + 2026-08-07) — it had no upstream issue and no PR of any kind before that. 2b is NOT + in that PR and has no PR of ours: the one third-party PR (#32209) solves the same + row mismatch by trimming q/top-k instead of expanding the page table, and porting + that half here fails at concurrency 32 with the cause unidentified, so we can + neither adopt it nor justify a competing PR. This patch therefore stays until BOTH + halves land. consumers: - Dockerfile.sglang (GLM-5.2 DSA PD decode leg with DP-attention + MTP) drop_when: >- @@ -92,6 +95,27 @@ history: upstream_issues: null upstream_prs: + - ref: sgl-project/sglang#33973 + url: https://github.com/sgl-project/sglang/pull/33973 + title: "[DSA] Remove the device-to-host syncs on the decode DP-divergent branch" + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: dorado269 + ours: true + same_approach: true + approach_note: >- + Opened 2026-08-07 as a DRAFT, carrying 2a AND 2a2 — and deliberately NOT 2b, which + overlaps #32209 and rests on an unexplained concurrency-32 failure. One fix, one + PR. Adapting it surfaced a stronger upstream argument than this record had: the + backend already declares `needs_cpu_seq_lens = False` with a comment saying it + opts out of the D2H sync, so the eager fallback contradicts its own contract. + Also corrects a citation in this record — the CPU mirror the DRAFT_EXTEND_V2 branch + relies on comes from eagle_worker_common.prepare_for_draft_extend (line 105), on + its gpu_only branch and for this same reason, not from base_spec_worker. + requested_action: >- + Draft until re-validated on gfx950 with PD + DP-attention + MTP. + in_pinned_base: false - ref: sgl-project/sglang#32209 url: https://github.com/sgl-project/sglang/pull/32209 title: Fix PD decode hang with DP attention and GLM-5.2 MTP @@ -186,11 +210,21 @@ scope_limits: >- around there with a runtime flag rather than fixed. open_actions: + - action: >- + Re-validate sglang#33973 (2a + 2a2) on gfx950 with PD + DP-attention + MTP and take + it out of draft. What WAS checked on MI300X against the PR's exact diff: that + `.max().item()` measurably blocks (~0.5-3 ms run to run, behind 30 queued 4096^2 + matmuls) while `.shape[1]` does not (~3 us) — the point being that it blocks at + all on a branch only some ranks take, not the absolute figure; that widening the + page table leaves every REAL top-k + selection and score bit-identical; and that DRAFT_EXTEND_V2 is not is_extend(), so + every consumer of the two removed mirrors is unreachable. What was NOT: that the + group actually stops deadlocking. + owner: dorado269 + blocked_on: MI355X cluster access (unreachable 2026-08-07) - action: >- Establish why #32209's trimming approach fails at concurrency 32 here, so 2b can - converge upstream instead of diverging. - owner: unassigned - blocked_on: null - - action: File an upstream issue for 2a — the DP host-sync deadlock has no upstream record at all. + converge upstream instead of diverging. Until then 2b keeps this patch alive on + its own, even once #33973 lands. owner: unassigned blocked_on: null diff --git a/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py index 2e318135..e416ec22 100644 --- a/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py +++ b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py @@ -105,10 +105,29 @@ (host VA is not the device VA on that platform). #32503 / #32792 (both OPEN) add Intel XPU HiCache and will touch the same dict -- a merge conflict risk for this anchor, not a fix for it. - own PR NONE. Not filed. It should be: upstream main is affected, the - one-line form matches an already-merged precedent (#23361), and - the device-pointer measurements above are the evidence. Blocked - only on someone opening it. + own PR #33968 (OPEN, DRAFT, filed 2026-08-07 by dorado269) -- + "[ROCm] Fix HiCache host-pool allocator: hipHostRegister's + device pointer is not the host VA". Same fix as this script, + minus the GLM52_ROCM_HOST_ALLOC bytecode marker. + + STILL A DRAFT because the MI355X cluster was unreachable when it + was opened, so it has NOT been re-run against the original fault + on hardware that reproduces it. What was checked, on MI300X: the + dispatch resolves to alloc_with_pin_memory for every key incl. a + torch.device one, a real 8 MiB allocation comes back pinned with + devPtr == host, and nothing is removed from the module. Do not + read "PR open" as "fixed upstream". + + CORRECTION This header used to call the fix a one-line dispatch override in + the shape of #23361. It is not, and cannot be: + * get_device() returns "cuda" on ROCm (measured), so HIP has no + key of its own to add the way "npu"/"musa" do; and + * memory_pool_host.py:768 and :1257 key the table with a + torch.device OBJECT. torch.device("cuda:0") is NOT + dict-key-equal to "cuda" (measured: different hash), so those + two pools always resolve through the defaultdict default. + That is why both the key and the default move below. A fix that + only added a "cuda" key would silently miss those two pools. Idempotent and self-locating. Run inside the container, then delete stale .pyc. """ diff --git a/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml index 03303897..3141a489 100644 --- a/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml +++ b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml @@ -1,6 +1,6 @@ # yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json schema_version: 1 -status_updated: 2026-08-05 +status_updated: 2026-08-07 verified_by: - gh-pr-issue-state - gh-search @@ -73,11 +73,11 @@ applies_to: seen on gfx942. alive_because: - reason: no-upstream-pr + reason: upstream-pr-open detail: >- - No upstream issue and no upstream PR — and none of ours either, which is the gap - worth closing here. Upstream main is affected, established by reading the file, so - this is not a case of "probably already fixed". + Ours is now upstream as sglang#33968 (DRAFT, opened 2026-08-07). No third-party + issue or PR exists. Upstream main is affected, established by reading the file, so + this is not a case of "probably already fixed". Was `no-upstream-pr` until #33968. consumers: - Dockerfile.sglang (kvd / hierarchical cache on gfx950 — required) - Dockerfile.sglang.gfx942 (preventive) @@ -102,7 +102,29 @@ history: upstream_issues: null -upstream_prs: null +upstream_prs: + - ref: sgl-project/sglang#33968 + url: https://github.com/sgl-project/sglang/pull/33968 + title: "[ROCm] Fix HiCache host-pool allocator: hipHostRegister's device pointer is not the host VA" + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: dorado269 + ours: true + same_approach: true + approach_note: >- + Opened 2026-08-07 as a DRAFT. Same fix as this patch, minus the local + GLM52_ROCM_HOST_ALLOC bytecode marker. Adapting it corrected the record's own + framing twice: get_device() returns "cuda" on ROCm, so HIP cannot take a key of + its own the way #23361's "musa" did; and memory_pool_host.py:768 and :1257 key + the table with a torch.device OBJECT, which is not dict-key-equal to the string + "cuda" (measured: different hash), so those two pools always resolve through the + defaultdict default. Changing only the key would have missed them — which is why + both the key and the default move. + requested_action: >- + Draft until re-validated on gfx950. The MI355X cluster was unreachable when it was + opened, so the write-back repro in the PR body is historical, not from that branch. + in_pinned_base: false related_refs: - ref: sgl-project/sglang#23361 @@ -197,7 +219,10 @@ scope_limits: >- open_actions: - action: >- - File the upstream PR. This is the only crash-class fix in the tree with no upstream - PR at all, upstream main is affected, and #23361 is the merged precedent to copy. - owner: unassigned - blocked_on: null + Re-validate sglang#33968 on gfx950 and take it out of draft. The equivalence and + scope of the PR's diff were checked on MI300X (dispatch for every key including a + torch.device one, a real 8 MiB allocation with its device pointer measured, module + exports), but MI300X is the NEGATIVE control — it cannot reproduce the fault. Run + the write-back repro on MI355X against that exact branch. + owner: dorado269 + blocked_on: MI355X cluster access (unreachable 2026-08-07) diff --git a/pr.done.md b/pr.done.md new file mode 100644 index 00000000..3fb14fd6 --- /dev/null +++ b/pr.done.md @@ -0,0 +1,150 @@ +# sglang GLM-5.2 patches — upstream PRs opened + +Companion to `work.todo.md`, which inventories all seven sglang GLM-5.2 patches +and establishes that exactly three needed a new upstream PR. This file records +what was opened, on what evidence, and what is still outstanding. + +Date: 2026-08-07. Upstream base: `sgl-project/sglang` `main` @ `7395ee833e`. +All three are **drafts** — see "Deferred validation" for why. + +## Opened + +| PR | Title | Patch it upstreams | State | +|----|-------|--------------------|-------| +| [#33968](https://github.com/sgl-project/sglang/pull/33968) | `[ROCm] Fix HiCache host-pool allocator: hipHostRegister's device pointer is not the host VA` | `sglang_rocm/patch_hicache_rocm_host_alloc.py` | draft | +| [#33970](https://github.com/sgl-project/sglang/pull/33970) | `[PD] Make the mooncake KV transfer wait on the prefill forward that wrote the pages` | `sglang_disagg/patch_mooncake_early_send_wait_event.py` | draft | +| [#33973](https://github.com/sgl-project/sglang/pull/33973) | `[DSA] Remove the device-to-host syncs on the decode DP-divergent branch` | `sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff` — **2a half only** | draft | + +Branches live on `dorado269/sglang`. Working folder with the adapted diffs, +validation scripts and PR bodies: `work/upstream-glm52-sglang-prs/`. + +## What each PR changes, and what was found while adapting it + +### #33968 — HiCache ROCm allocator + +`ALLOC_MEMORY_FUNCS` defaults to `alloc_with_host_register` (anonymous `mmap` + +`hipHostRegister`), but the host pools hand host `data_ptr()`s to GPU kernels +through a device-side pointer table. On ROCm those addresses are not the same, +so the first write-back faults at the host VA. + +**The record described this as a one-line override in the shape of the merged +MUSA PR #23361. It cannot be**, for two reasons established first-hand: + +1. `get_device()` returns `"cuda"` on ROCm — measured in the container — so HIP + has no key of its own to add, unlike `"npu"`/`"musa"`. +2. `memory_pool_host.py:768` and `:1257` key the table with a `torch.device` + **object**. Measured: `torch.device("cuda:0") == "cuda"` is `False` and the + hashes differ, so those two pools *always* resolve through the `defaultdict` + default. Adding only a `"cuda"` key would silently miss them. + +So the default has to change too. That widens behaviour on a ROCm host to any +other device string (e.g. `"xpu"`); raised with the user, who chose to keep +equivalence with the proven fix rather than ship an unvalidated narrower one. + +### #33970 — mooncake early-send wait event + +Silent correctness bug: chunked prefill over mooncake RDMA-reads KV pages while +the forward writing them is still running, so long prompts come back partially +wrong with no crash and no log line. + +Confirmed on `main`: `mooncake/conn.py` contains `wait_event` **0 times** while +`mori/conn.py` contains it **6 times**, and the overlap non-final-chunk send in +`prefill.py` records no event at all. + +**Cross-transport effect worth knowing:** `prefill.py` is transport-agnostic, so +recording the event there also closes the same gap for `mori`, which reads the +field but was never handed one on that path. Checked that `nixl`, `ascend`, +`fake` and `base` do not read `_early_send_wait_event` and that no sender uses +`__slots__`, so setting the attribute is inert for them. + +### #33973 — DSA decode DP host-sync (2a only) + +`seq_lens.max().item()` is a D2H sync on a branch only some DP ranks enter, so +the group deadlocks on the first routed request. + +The strongest upstream argument turned out to be one the local record did not +use: the backend **already declares** `needs_cpu_seq_lens = False`, with a +comment saying it opts out of the D2H sync — so the eager fallback contradicts +its own contract. Also corrected a citation: the CPU mirror this path relies on +comes from `eagle_worker_common.prepare_for_draft_extend` (line 105), not +`base_spec_worker` as the record said. + +**Scope deliberately narrowed.** The local diff bundles a second, independent +defect (2b, the per-request vs per-token page-table row mismatch). It is **not** +in this PR: it overlaps #32209, and porting that PR's trimming approach here +fails reproducibly at concurrency 32 with the cause unidentified. One fix, one +PR; 2b stays local until that is understood. + +## Validation + +Everything below was run **this session**, on 8× MI300X (gfx942), ROCm 7.2.0, +torch 2.9.1, inside `lmsysorg/sglang-rocm:v0.5.15-rocm720-mi30x-20260713` with +the adapted upstream tree on `PYTHONPATH`. Scripts are in the working folder. + +| Script | What it establishes | Result | +|--------|--------------------|--------| +| `probe_host_devptr.py` | host VA vs device pointer per allocation strategy | PASS — all four strategies **equal** on MI300X, reproducing the documented negative control | +| `validate_A.py` | #33968 equivalence + scope vs the proven local fix | PASS | +| `validate_B.py` | #33970 plumbing + that the barrier is real, against live CUDA events | PASS | +| `validate_C.py` | #33973 — the sync is real; widening does not change top-k; the removed mirrors are unreachable | PASS | + +Two findings from those runs that changed the PRs rather than just confirming +them: + +- `.max().item()` measurably blocks — **~0.5–3 ms** run to run, behind 30 queued + 4096² matmuls — versus **~3 µs** for `.shape[1]`. First-hand evidence that the + branch really does desynchronize ranks, quoted in #33973. The number moved + during cleanup: an earlier revision measured 44 ms because the timed call was + also the first one, so it carried lazy-init cost. Warmed, it is 0.5–3 ms. The + PR was corrected; what matters is that it blocks at all, not the magnitude. +- `validate_C.py` initially **failed** an assertion I had written. Diagnosis: a + row with `seq_len < topk` legitimately returns indices past its own `seq_len`, + because top-k must return `topk` entries; those slots carry `-inf` and occur + identically at both widths. My test's model was wrong, not the fix — but the + correct property (every *real* selection is bit-identical narrow vs wide) is + now asserted, and stated in the PR. + +## Deferred validation — why all three are drafts + +The original faults need hardware this session does not have. The vultr MI355X +cluster is **unreachable**: `149.28.124.225` answers neither ping (100% loss) +nor `:22`; `chi2865` / `chi2866` the same. The local box is gfx942, which is the +*negative control* for #33968 and cannot reproduce any of the three faults. + +So this session established **equivalence and scope** against the proven local +fixes, and nothing more. Each PR body says so explicitly and marks which evidence +is historical. + +**TODO when the cluster returns** — re-run against the exact PR diffs, then flip +each draft to ready: + +| PR | Needs | Run | +|----|-------|-----| +| #33968 | 1× gfx950 | HiCache write-back repro: stock must fault at the host VA, patched must not. Re-run `probe_host_devptr.py` there — expect `same=False`, the positive control. | +| #33970 | 2 nodes with RDMA | GLM-5.2-FP8 1P1D over mooncake, `--chunked-prefill-size 131072`, overlap on. Needle retrieval, expect 5/9 → 9/9. **Plus** the added `synchronize()`'s cost on prefill throughput — flagged as unmeasured in the PR, and the first thing a reviewer will ask. | +| #33973 | 1× gfx950, PD + DP-attention + MTP | Group must not deadlock on the first routed request; `py-spy` should show no rank inside `dsa_backend`. | + +## CI + +All three show `pr-gate` red. Cause verified via the job steps API: the failing +step is **"Block draft PR"** — a repository policy that fails the gate for any +draft, and the `*-finish` jobs just aggregate it. Not a lint, format, or compile +failure; `pre-commit` passes clean locally on every changed file. These go green +when the drafts are marked ready. + +## Not upstreamed, and why + +From `work.todo.md`, four of the seven sglang patches correctly get no PR: + +- `patch_glm52_nextn_quark_exclude.py` — #30265 already **merged**; ours is a + backport onto a frozen release base. +- `patch_dsa_indexer_hip_dp_padded_rows.py` — already ours as **#33059** (open). +- `draft_cuda_graph_dp_vote.diff` — **#32209** carries the same fix with the same + strategy; a competing PR would be noise. Add a ROCm datapoint to that thread + and to #32527 instead. +- `patch_hicache_rocm_staged_write_back.py` — **#30350** is the upstream repair + and is better than ours (covers the DeepSeekV4 pools too). It is + `CHANGES_REQUESTED` with the conflict long cleared; the action is a re-review + nudge. Note the inversion: the *merged* #28534 is what introduced the defect. +- `dsa_backend_dp_sync_and_page_table_rows.diff` **2b half** — see #33973's scope + note above. diff --git a/work.todo.md b/work.todo.md new file mode 100644 index 00000000..d3971f1c --- /dev/null +++ b/work.todo.md @@ -0,0 +1,142 @@ +# sglang GLM-5.2 patches — upstream PR inventory + +Scope: every patch in `deploy/docker/patches/` whose `target.library` is `sglang` +and whose problem was found on, or verified against, GLM-5.2. That is all seven +sglang patches in the tree — no sglang patch in this repo is unrelated to GLM-5.2. + +Source of truth: each patch's `.upstream.status.yaml`, re-checked against +live upstream on 2026-08-07 (`gh` for PR/issue state, `git show origin/main` for +source). vLLM / mooncake / aiter / atom patches are out of scope here even where +they mention GLM. + +Upstream `main` read at `3ed2a0adf3` (2026-08-07). + +## The table + +| # | Patch | Component | Existing upstream PR | State (2026-08-07) | Ours? | main affected? | Needs a NEW PR | +|---|-------|-----------|----------------------|--------------------|-------|----------------|----------------| +| 1 | `sglang/patch_glm52_nextn_quark_exclude.py` | quantization | sglang#30265 | MERGED 2026-07-08 | no | **no** — fixed | **no** | +| 2 | `sglang_disagg/patch_mooncake_early_send_wait_event.py` | disaggregation | *none* | — | — | **yes** | **YES** | +| 3 | `sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.py` | dsa | sglang#33059 | OPEN, REVIEW_REQUIRED | **yes** | yes | no — already filed | +| 4 | `sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff` (2a) | dsa | *none* | — | — | **yes** | **YES** | +| 4b | `sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff` (2b) | dsa | sglang#32209 | OPEN, REVIEW_REQUIRED | no | yes | **no** — see below | +| 5 | `sglang_dsa/draft_cuda_graph_dp_vote.diff` | speculative-decoding | sglang#32209 | OPEN, REVIEW_REQUIRED | no | yes | no — deliberate | +| 6 | `sglang_rocm/patch_hicache_rocm_host_alloc.py` | hicache | *none* | — | — | **yes** | **YES** | +| 7 | `sglang_rocm/patch_hicache_rocm_staged_write_back.py` | hicache | sglang#30350 | OPEN, CHANGES_REQUESTED | no | yes | no — nudge #30350 | + +Three patches need a new upstream PR: **#2, #4 (2a half only), #6**. + +## Per-patch notes + +### 1 — `patch_glm52_nextn_quark_exclude.py` — no PR needed +sglang#30265 ("[AMD] Fix GLM-5.2 MTP Quark excludes", wangjiaxin99) is MERGED and +is a superset of ours: it gives GLM-5.2 a dedicated `GlmMoeDsaForCausalLMNextN`. +Our patch is a one-line backport onto the frozen `v0.5.15.post1` base, which was +cut from `release/v0.5.15` without it. Nothing to upstream — it is upstream. + +Carry-forward warning (already in the record): the anchor string +`ckpt_prefix = f"model.layers.{config.num_hidden_layers}"` is STILL on main +(`deepseek_nextn.py:328`), so the patch does not self-disable on a fixed base. +Decide the drop from the base version, not from the build log. + +### 2 — `patch_mooncake_early_send_wait_event.py` — NEEDS A PR +No upstream PR, and no upstream issue that establishes the root cause (#25583 is +the same corruption shape on an *aggregated* GLM-5-FP8 server, auto-closed +inactive — suggestive only). + +Re-verified on main today: +- `disaggregation/mooncake/conn.py` — `wait_event` occurs **0 times**, while + `disaggregation/mori/conn.py` has **6**. The barrier exists only for mori. +- `disaggregation/prefill.py:1113` records `_early_send_wait_event` on the + radix early-send path; the overlap non-final-chunk send at line 812 records + nothing at all. + +So main is affected on both halves. This is a **silent** correctness bug — no +crash, no log line, output partially wrong past the first prefill chunk. Best +candidate in the tree for an upstream PR. + +Before opening: the record's own open action asks for the cost of the added +`synchronize()` on prefill throughput. Not measured. Will be stated as an open +question in the PR rather than claimed either way. + +### 3 — `patch_dsa_indexer_hip_dp_padded_rows.py` — already filed +Ours: sglang#33059, OPEN, `REVIEW_REQUIRED`, `MERGEABLE`/`BLOCKED`, last touched +2026-08-07. No action beyond review chasing. + +Note the file moved upstream: `layers/attention/dsa_indexer.py` → +`layers/attention/dsa/dsa_indexer.py`. #33059 is already against the new path. + +### 4 — `dsa_backend_dp_sync_and_page_table_rows.diff` — SPLIT +This diff bundles two independent defects. They upstream differently. + +**2a — DP host-sync deadlock. NEEDS A PR.** No upstream issue, no upstream PR, +ours or anyone's. Confirmed live on main: +- `dsa_backend.py:794` — `max_seqlen_k = int(forward_batch.seq_lens.max().item())`, + a blocking D2H sync on a branch only some DP ranks take. +- `dsa_backend.py:854-861` — the two further unconditional `.cpu()` mirrors + (2a2) are still there. With 2a alone the hang persists, so both go together. + +**2b — page-table row mismatch. NO new PR.** sglang#32209 (HZY-Wade) is OPEN and +already addresses this row mismatch, by TRIMMING q/top-k where we EXPAND the page +table. Our record documents a negative result: porting #32209's trimming approach +onto this HIP/tilelang path fails reproducibly at concurrency 32 (0/32 across +seven runs), with the root cause **not identified**. Opening a competing PR on an +unexplained failure would be noise. Correct move is to finish that investigation +first; keep 2b local until then. + +### 5 — `draft_cuda_graph_dp_vote.diff` — no PR, deliberate +sglang#32209 carries this exact fix with the same strategy (group decision rather +than per-rank), and our diff adopts its placement verbatim so the two converge. +sglang#32527 (Xavier1994) reports the same deadlock independently on 8× Blackwell +— not ROCm-specific — and is still OPEN with no activity. + +The value we can add is a datapoint on those threads, not a fourth PR. + +### 6 — `patch_hicache_rocm_host_alloc.py` — NEEDS A PR +No upstream issue and no upstream PR. Confirmed live on main +(`mem_cache/pool_host/common.py:177-183`): + +```python +ALLOC_MEMORY_FUNCS = defaultdict( + lambda: alloc_with_host_register, + {"npu": alloc_with_pin_memory, "musa": alloc_with_pin_memory}, +) +``` + +No HIP entry. The merged precedent to copy is sglang#23361 ("[MUSA][19/N] Support +HiCache with pin_memory allocator") — same one-line dispatch override, same +reason. This is the clearest missing-PR gap in the whole tree: a crash-class fix, +main affected by direct source read, and an already-merged PR of the same shape. + +Anchor-collision risk when it lands: #32503 and #32792 (both OPEN, Intel XPU) +touch the same dict. + +### 7 — `patch_hicache_rocm_staged_write_back.py` — no new PR +sglang#30350 (Emmanuel0612) is the upstream repair and is strictly better than +ours: it flips the three CUDA-only gates via `_is_cuda_alike`, covers the +DeepSeekV4 pools we do not, and teaches `staged_write_back.cuh` to accept +`kDLROCM`. It is OPEN with `CHANGES_REQUESTED` (HaiShaw, 2026-07-13); the +conflict was cleared the same day and no re-review has been requested. + +Our MI300X datapoint is already on the thread (llying-001, 2026-08-04). Correct +action is a re-review nudge, not a competing PR. Note the inversion: the MERGED +PR here (#28534) is what *introduced* the disagreement. + +## Plan + +1. Open an upstream PR for **#6** (hicache ROCm allocator) — smallest, strongest + evidence, merged precedent. +2. Open an upstream PR for **#2** (mooncake early-send wait event) — silent + correctness bug, three files, mirrors what mori already does. +3. Open an upstream PR for **#4 / 2a** (DP host-sync deadlock) — 2a + 2a2 only. + 2b stays local. +4. All three as **drafts**, via the `open-source-pr` workflow: check upstream + main, strip local-repo semantics (`GLM52_*` markers, the `MARKER = "applied"` + bytecode literal, infera path references), re-validate the adapted patch + locally for equivalence and scope, then open. +5. Record the outcome in `pr.done.md` and update each patch's + `upstream_prs` / `open_actions` in its `.upstream.status.yaml`. + +Not doing, and why: #1 is already merged upstream; #3 is already our open PR; +#5 and #7 have a better third-party PR in flight that we should support rather +than compete with; #4 / 2b rests on an unexplained concurrency-32 failure. From 9235c26f7b236b79580f7aec1b27a9983630def8 Mon Sep 17 00:00:00 2001 From: yihou Date: Wed, 19 Aug 2026 07:39:50 +0000 Subject: [PATCH 3/3] docs(pr-verify): hand-off record for the unfinished upstream PR validation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gfx950 hardware validation for #33968/#33970/#33973 is incomplete on this box and has to move to another machine. Record what was established first-hand, what failed and why, and what the next session must fix — so none of it is re-derived. Key items that would otherwise be lost: MC_GID_INDEX is 1 on this fabric (and a passing ib_write_bw -x 3 does NOT validate it), the ZMQ port collision that kills a second same-host leg, and that DP-attention divides chunked-prefill-size by dp_size so the needle probe's chunk math was wrong. Signed-off-by: yihou --- temp.working.process.pr.verify.md | 233 ++++++++++++++++++++++++++++++ 1 file changed, 233 insertions(+) create mode 100644 temp.working.process.pr.verify.md diff --git a/temp.working.process.pr.verify.md b/temp.working.process.pr.verify.md new file mode 100644 index 00000000..62cdf8e1 --- /dev/null +++ b/temp.working.process.pr.verify.md @@ -0,0 +1,233 @@ +# Working process — verifying the three upstream sglang GLM-5.2 PRs + +**Status: INCOMPLETE.** Hardware validation is unfinished; this file exists so the work +can be resumed on a different machine. Session 2026-08-19 on n06-33 (8x MI355X / gfx950). + +Task spec: `pr.verify.md`. Prior session's record: `pr.done.md` (second-hand — two of its +claims turned out stale, see below). Session config: `.claude/CLAUDE.md`. +Scratch workspace on n06-33: `/data/yihou/workspace.temp/pr-verify-20260819/`. + +## Where this stands + +| # | Stage | State | +|---|-------|-------| +| 1 | Upstream status + scope selection | done | +| 2 | Rebase all three branches onto current `main` | done | +| 3 | Code review of each PR | not started | +| 4 | **gfx950 hardware validation** | **in progress — blocked, see below** | +| 5 | Deep review (code read + LSP + serena) | not started | +| 6 | Flip draft -> ready, update records | not started | + +All three PRs are still **draft**, with **zero** comments and **zero** reviews since they +were opened on 2026-08-07. + +## The three PRs + +| PR | Fix | Branch (on `dorado269/sglang`) | Rebased onto `c863760ae1` | +|----|-----|--------------------------------|---------------------------| +| [#33968](https://github.com/sgl-project/sglang/pull/33968) | HiCache ROCm host-pool allocator | `fix-hicache-rocm-pin-memory-allocator` | `33f0ea6cd3` | +| [#33970](https://github.com/sgl-project/sglang/pull/33970) | mooncake KV transfer waits on prefill forward | `fix-mooncake-pd-chunked-prefill-kv-race` | `780fbb3018` | +| [#33973](https://github.com/sgl-project/sglang/pull/33973) | DSA decode DP-divergent D2H syncs | `fix-dsa-decode-dp-host-sync-deadlock` | `2b3c9ea7a3` | + +Upstream checkout: `/home/yihou/dev/git.16-10/sglang` (`origin`=sgl-project, `fork`=dorado269). + +## Completed, with first-hand evidence + +- **Rebase.** All three were 609 commits behind. Rebased onto `c863760ae1`, all three + **without textual conflict**. +- **Defects re-confirmed present on current `main`** by source read, not by assuming the + clean rebase implied it: `ALLOC_MEMORY_FUNCS` still has no HIP entry; `mooncake/conn.py` + still contains `wait_event` 0 times vs `mori/conn.py` 6; `dsa_backend.py:796` still has + `seq_lens.max().item()`. +- **PR C semantic gap closed.** `dsa_backend.py:1055` constructs + `DSAMetadata(seq_lens_sum=forward_batch.seq_lens_sum)` on a path DRAFT_EXTEND_V2 reaches. + Checked whether the PR's removal of the `.cpu()` mirrors could strand that field: + `metadata.seq_lens_sum` has **zero** readers repo-wide (checked `dsa_indexer_metadata.py`, + `dsa_topk_backend.py`, `nsa_backend.py`; no `asdict` / `astuple` / `replace` either). +- **Image built and saved.** `infera-local:sglang-prverify-20260819`, 80 GB, saved to + `/data/yihou/images.backup/`. Base pinned by `Dockerfile.sglang` is + `lmsysorg/sglang:v0.5.17-rocm720-mi35x` — a *different repo/tag* from the local + `lmsysorg/sglang-rocm:...-20260809`; do not substitute, the DSA context diffs apply at + `--fuzz=0` only against the pinned one. +- **`validate_A/B/C` all PASS** on gfx950 against current `main`. +- **The image's sglang is a git checkout** (`/sgl-workspace/sglang`, HEAD `2948168546`) + with the infera patches applied as working-tree modifications. This is useful: a stock + arm is `git checkout --` of the three PR-B files inside the container, so the two arms of + an A/B differ by exactly those files and nothing else — no rebuild needed. +- **The applied mooncake diff matches PR #33970 exactly** (verified by `git diff` in the + container against the PR body). + +## Established environment facts (n06-33) + +- 8x MI355X gfx950, 288 GB/GPU. ROCm 7.2.0, torch 2.9.1, amdgpu **6.14.14**. +- Model: **`/data/models/GLM-5.2-MXFP4` is a complete local copy** (408 GB, 282 shards). + Prefer it over `/apps/data/models/...`, which is NFS at 716 MB/s and 100% full. +- Rails `benic{1..8}p1` = `192.168.{1..8}.14/31`; HCA map is **not** in numeric order: + `ionic_0->benic1p1, ionic_1->benic2p1, ionic_2->benic4p1, ionic_3->benic3p1, + ionic_4->benic5p1, ionic_5->benic6p1, ionic_6->benic8p1, ionic_7->benic7p1`. +- **`MC_GID_INDEX` must be 1 on this fabric, not 3.** `ionic_0` port 1 exposes only gid[0] + (link-local) and gid[1] (`::ffff:c0a8:010e` = 192.168.1.14, RoCE v2). With 3, mooncake + dies with "GID is NULL ... No available RNIC". + **Trap: `ib_write_bw -x 3` works anyway** — perftest and mooncake index GIDs differently, + so a passing perftest run does **not** validate mooncake's GID setting. +- **`RDMAV_FORK_SAFE=1` is required**, else "RDMA context setup failed: fork compatibility: + Invalid argument". +- Rails route only to `192.168.N.12` (= n06-25). **n01-33 has no rail route to n06-33 and + vice versa** (ARP INCOMPLETE) — this is what blocks the 2-node #33970 run. +- **A measurement error to not repeat:** `ping -I ` only sets the source address; + traffic still egresses over the `fenic` management NIC, which made all 8 rails look + healthy when none were. Use `ping -I `. + +## Errors hit and how they were resolved + +| Symptom | Root cause | Resolution | +|---|---|---| +| `docker pull` log showed "Pull complete" but no image | `nohup ... &` inside the background wrapper truncated it | run `docker pull` as the foreground command of a background task | +| Build failed: `Could not resolve host: index.crates.io` (exit 101) | **Hypothesised IPv6-only resolution with no v6 route — then tested it and the hypothesis was wrong.** A dedicated BuildKit probe with `--network=host --no-cache` showed DNS and IPv4 egress both fine (`http=200`) inside a RUN step | transient. **Retried; it succeeded. Dockerfile deliberately not modified.** | +| `validate_A.py`: "ALLOC_MEMORY_FUNCS block not found" | it resolved the container's already-patched sglang; the validator needs a *stock* tree to patch itself | `git worktree add ... origin/main --detach`, mount that | +| `validate_B.py` FAILED sections [1] and [2] | pointed at the stock tree, which correctly lacks the fix | worktrees at the rebased branches | +| `ib_write_bw` to n01-33 hung | no rail route between the two nodes | 2-node path abandoned; switched to single-node TP4+TP4 | +| `ib_write_bw` produced no data rows even to the routed peer | needs `-R` (rdma_cm) | with `-R`: 330–360 Gb/s on all 8 rails. mooncake also uses rdma_cm, so not a blocker | +| mooncake MVP: `GID is NULL`, `No available RNIC`, fork-compat error | GID index 3 does not exist on this fabric | `MC_GID_INDEX=1` + `RDMAV_FORK_SAFE=1` | +| `AttributeError: ... no attribute 'get_session_id'` | this mooncake build exposes `get_rpc_port`, not `get_session_id` | session id built as `f"{ip}:{eng.get_rpc_port()}"` | + +## #33968 — negative result, stays draft + +`pr.done.md` predicted gfx950 would measure `same=False` (host VA != device pointer), i.e. +the positive control that gfx942 lacks. Measured here: + +``` +device: AMD Instinct MI355X gcn=gfx950:sramecc+:xnack- +torch: 2.9.1+rocm7.2.0 hip=7.2.26015 + [pin_memory] / [mmap+hipHostRegister] / [+Mapped] / [+Portable|Mapped] -> same=True (all four) +``` + +Suspected buffer size was the uncontrolled variable (the original fault report used a +7.33 GB indexer buffer; the probe uses 8 MiB), so swept 8 MiB / 256 MiB / 1 GiB / 4 GiB / +7.33 GB x 4 strategies: **all `same=True`**. Size is ruled out. + +Only known remaining difference is amdgpu **6.14.14**, which the patch record attributes to +the MI300X *negative* control; the original gfx950 fault report does not record its driver +version. **No mechanism is claimed.** The statement that survives: this machine cannot +reproduce the fault, so it is a negative control like gfx942, and #33968's write-back +evidence remains historical. + +`validate_A.py` still PASSes here — it tests equivalence and scope against the proven local +fix, which does not require the fault to reproduce. It also independently confirms the PR's +core argument: stock dispatch for a `torch.device('cuda:0')` key resolves to +`alloc_with_host_register`, so the two pools that key with a device object do fall through +the defaultdict. + +**Action: needs a machine that reproduces `same=False`. User is sourcing one.** + +## r01 — single-node TP4+TP4 premise checks (all passed) + +Hypothesis: the #33970 race is between the mooncake transfer *thread* and the CUDA stream, +not between two hosts. If so, a single-node 1P1D with two TP4 legs over loopback RDMA +reproduces it, and the blocked 2-node path is not required. + +1. **No local/loopback shortcut in mooncake.** `mooncake/conn.py` on current main has no + `is_local` / `same_host` / `loopback` / `local_transfer` branch and no assertion that + prefill and decode are on different hosts. +2. **The race is thread-vs-stream.** `conn.py:252` starts + `threading.Thread(target=self.transfer_worker)`; that worker calls + `engine.batch_transfer_sync` (`:657`, `:1112`). A CPU thread reads GPU memory outside the + CUDA stream — exactly what the fix gates with an event. Nothing there depends on the peer + being remote. +3. **RDMA loopback works at the verb layer.** `ib_write_bw -R`: cross-HCA (ionic_0 -> + ionic_4) **348.59 Gb/s**, same-HCA **335.92 Gb/s**. +4. **mooncake itself works loopback** — the decisive check, with the real engine rather than + perftest and deliberately without sglang: two containers on this host, + `transfer_sync_read` of 8 MiB, `rc=0`, every byte 0xAB, **PASS**. Log also showed + "HIP transport installed for intra-node GPU P2P". + Script: `scripts/mvp_mooncake_loopback.py`. +5. **Capacity.** 408 GB model, TP4 -> ~102 GB/GPU on 288 GB cards. + +**What this configuration can and cannot establish — must go in the PR:** +- CAN: the correctness claim (needle retrieval degraded -> clean) and the `synchronize()` + cost, which is the reviewer's obvious first question. +- CANNOT: behaviour under real cross-node RDMA latency. Loopback is *faster*, so the race + window is *narrower* and reproduction is *harder*. A positive reproduction here implies + the cross-node case is at least as bad; a failure to reproduce here would **not** clear + the cross-node case. + +## r02 — stock positive control: FAILED to launch, two causes found + +Both are artifacts of running two legs on one host. **Neither is a defect in the patch.** + +**1. ZMQ port collision -> prefill SIGKILLed.** `ZMQ_TCP_PORT_DELTA = 233`, and both +containers are `--network=host` so they share 127.0.0.1: + +| leg | `--port` | `port_base` | reserved | +|---|---|---|---| +| prefill | 30000 | 30234 | 30234–30240 | +| decode | 30001 | **30235** | 30235–30241 | + +prefill's detokenizer wants 30235; decode already holds it -> +`zmq.error.ZMQError: Address already in use (addr='tcp://127.0.0.1:30235')` -> +`sglang subprocess exited with code -9`. + +Fix for the next attempt: separate the two legs' `--port` by at least +`NUM_DERIVED_PORTS + ZMQ_TCP_PORT_DELTA` (e.g. 30000 / 31000). Note decode also binds +`0.0.0.0:5557` and `0.0.0.0:8801` **even with `KVAWARE=0`** (infera's +`--kv-events-bind` defaults to `tcp://0.0.0.0:5557` in +`/opt/infera/infera/engine/sglang/args.py:180`), so those must be moved too. + +**2. The effective chunk size was 32768, not the 131072 that was passed.** From the +resolved `server_args`: `chunked_prefill_size=32768`. With DP-attention on, sglang divides +the global budget by `dp_size` (=4). `leg.sh` warns about exactly this. + +Consequence: the needle probe's chunk-index arithmetic was **wrong** and must be recomputed +against the *resolved* value, not the requested one. Silver lining — a smaller chunk means a +200k-token prompt splits into ~7 chunks instead of ~2, so there are more non-final chunks +and the race window is easier to hit. **The needle must land in a non-final chunk**: the +final chunk goes through the sampling path, which already has a real `copy_done.synchronize()`, +so a final-chunk needle is retrieved correctly even on a broken build and would read as a +false PASS. + +**Discipline for the retry: the positive control comes first.** Stock (unpatched) sglang +must reproduce a degraded needle score. Without that, a clean score on the patched tree +proves nothing. + +## Scratch artifacts on n06-33 + +Under `/data/yihou/workspace.temp/pr-verify-20260819/`: + +- `working_process.md` — the in-workspace log (this file supersedes it for hand-off) +- `scripts/mvp_mooncake_loopback.py` — the loopback MVP that proved premise 4 +- `scripts/probe_host_devptr_sizes.py` — the #33968 size sweep +- `rounds/r02-stock-positive-control/scripts/up_singlenode.sh` — single-node 1P1D bring-up; + takes `ARM=stock|patched` and **guards on `grep -c wait_event` (stock=0, patched=9)**, + refusing to run a mislabelled experiment. Needs the two port fixes above. +- `rounds/r02-stock-positive-control/scripts/needle.py` — needle probe; tokenizes with the + real tokenizer so each needle's chunk index is known rather than assumed, and reports + non-final-chunk and final-chunk scores separately. Needs the chunk-size fix above. +- `sglang-stock/`, `sglang-B/`, `sglang-C/` — git worktrees used by the validators +- Image tar in `/data/yihou/images.backup/` + +## Open items for whoever resumes this + +1. **#33970**: fix the two r02 causes (leg ports 30000/31000 + move 5557/8801; recompute + needle chunk math against the resolved 32768), run the **stock** arm to get a degraded + score, then the **patched** arm. Also measure the `synchronize()` cost on prefill + throughput — unmeasured, and the first thing a reviewer will ask. +2. **#33973**: not started. Needs 1x gfx950 with PD + DP-attention + MTP; the group must not + deadlock on the first routed request, and `py-spy` should show no rank inside + `dsa_backend`. +3. **#33968**: blocked on a machine that reproduces `same=False`. +4. **Stale record to correct:** `pr.done.md` says upstream **#30350** is the better fix for + `patch_hicache_rocm_staged_write_back` and that the action is a re-review nudge. + **#30350 was CLOSED unmerged on 2026-08-17 by its author** (Emmanuel0612). Per the user's + decision this is a TODO only — they will look at it themselves; do not open a replacement + PR. `pr.done.md` has not yet been updated. +5. CI red on all three is the `Block draft PR` repo policy, not a real failure. +6. **DCO**: every commit needs `-s`, signed off as the actual author — never a bot or + assistant identity, and upstream rejects assistant `Co-Authored-By` trailers. + +## Note on n06-33 machine state at hand-off + +All containers from this session (`glm52_p`, `glm52_d`, `glm52-etcd`) were removed and their +GPUs released. A container `sla-decode-limou` belonging to **another user** (started +2026-08-19 07:08, running `gpt-oss-120b`, using this session's image +`infera-local:sglang-prverify-20260819`) holds ~286 GB on GPUs 1 and 2. It was left +untouched.