diff --git a/.gitignore b/.gitignore index 1c6b4a8e..8d1a92f3 100644 --- a/.gitignore +++ b/.gitignore @@ -60,3 +60,10 @@ CLAUDE.md .spur_job_*.sh .spur_ns_*.sh spur-*.out + +# Per-task agent workspaces. These hold the working log, validation scripts and +# scratch copies for one piece of work — useful while it is in flight and to +# re-run later, but they are not repo deliverables. The conclusions that ARE +# deliverables get written into the patch records and into the reports at the +# repo root instead. +work/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 5abd08ba..b50e15fc 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -28,6 +28,19 @@ repos: - id: ruff-check args: [--fix] + # Style gate for the hand-written YAML under deploy/docker/ only. check-yaml + # above answers "does this parse"; yamllint answers "does it say what its + # author meant" — an unquoted ` #` opens a comment, so a PR number mid-sentence + # silently truncates the value. Scoped by `files:` rather than turned loose on + # the repo: deploy/operator/ holds a much larger body of existing k8s YAML and + # is left alone deliberately. Config in .yamllint.yaml. + - repo: https://github.com/adrienverge/yamllint + rev: v1.38.0 + hooks: + - id: yamllint + args: [--strict] # warnings fail too; a hook that exits 0 gets ignored + files: ^deploy/docker/.*\.ya?ml$ + # Rust formatting gate (needs a local toolchain: rustup + rustfmt). clippy and # tests are heavier, so they run in CI (.github/workflows/ci.yml, job `rust`). - repo: local @@ -39,6 +52,21 @@ repos: files: ^rust/.*\.rs$ pass_filenames: false + # Every patch under deploy/docker/patches/ must have a status record that validates + # against its schema and agrees with the index, so a patch cannot be added — or + # silently outlive its upstream fix — without saying where it stands. + # language: python (not system) so pre-commit builds an env holding the two + # deps — the CI lint job installs pre-commit alone, never the project, so a + # system hook has no jsonschema. Kept in sync with pyproject's [dev] extra, + # which is what the pytest job uses. + - id: validate-patch-status + name: patch upstream status (schema + index cross-check) + entry: python scripts/validate-patch-status.py + language: python + additional_dependencies: ["jsonschema>=4.18", "PyYAML>=6"] + files: ^deploy/docker/(patch\.upstream\.status\.yaml|patches/) + pass_filenames: false + # Refuse a commit whose git email is a machine-generated local hostname # (git's fallback when user.email is unset), so an internal build-host # name can't leak into permanent public history. Skipped in CI. diff --git a/.yamllint.yaml b/.yamllint.yaml new file mode 100644 index 00000000..cfe7da0e --- /dev/null +++ b/.yamllint.yaml @@ -0,0 +1,42 @@ +# Style gate for the hand-written YAML under deploy/docker/ (the patch status +# records and their index). check-yaml already answers "does this parse"; this +# answers "does it say what its author meant". +# +# Scoped by the pre-commit hook's `files:`, not from here — the k8s manifests and +# Helm charts under deploy/operator/ are a much larger body of existing YAML and +# are deliberately left alone for now. +# +# Run by hand over the same set: +# yamllint $(git ls-files 'deploy/docker/**/*.yaml') + +extends: default + +rules: + # The records quote upstream verbatim — issue titles, engine error strings, + # call chains — and those have to stay byte-exact to stay greppable, so the + # limit is set where it catches a runaway line without asking anyone to reflow + # a fact. 80 (the default) is unusable here; the longest line today is 156. + line-length: + max: 160 + allow-non-breakable-words: true # a bare URL should not have to wrap + + # These are single-document files by construction; the schemas have no notion + # of a multi-doc record, so a leading `---` would be noise on all 27 of them. + document-start: disable + + # This one is load-bearing, not cosmetic. In YAML an unquoted ` #` opens a + # comment, so `action: Get #2725 reviewed; ...` silently parses as `Get` and + # the rest of the sentence is discarded. Eleven values were losing text that + # way before this hook went in — every one of them a PR number mid-sentence. + comments: + require-starting-space: true + min-spaces-from-content: 2 + + # Warnings are failures: a hook that prints and exits 0 teaches people to + # ignore it. + truthy: + level: error + braces: + level: error + brackets: + level: error diff --git a/deploy/docker/Dockerfile.vllm b/deploy/docker/Dockerfile.vllm index 92a6ab0e..52bfbd67 100644 --- a/deploy/docker/Dockerfile.vllm +++ b/deploy/docker/Dockerfile.vllm @@ -33,7 +33,9 @@ RUN if [ "${BUILD_AITER}" = "1" ]; then \ # Apply patches/vllm/ (general PD-transport / scheduler fixes) then patches/ # vllm-dsv4/ (DSv4-specific: moriio_dsv4 {hybrid_blocksize, sparse_backend, # noncontig_register} + aiter flydsl). Each patch no-ops if its anchor is absent, -# so a base bump degrades gracefully. legacy/ is upstream on this base, not applied. +# so a base bump degrades gracefully. The retired legacy/ patches have moved to +# patches/archived/vllm-dsv4/ (recorded in patches/archived/patch.archived.yaml) and are +# no longer copied into the image at all. # # Removed on v0.25.1 (were no-op here): the three GLM-5.1 moriio patches # patch_moriio_dsa_write.py, patch_moriio_hetero.py and patch_vllm_moriio_blocksize.py. diff --git a/deploy/docker/patch.upstream.status.md b/deploy/docker/patch.upstream.status.md deleted file mode 100644 index 812c1226..00000000 --- a/deploy/docker/patch.upstream.status.md +++ /dev/null @@ -1,230 +0,0 @@ -# Patch ↔ upstream status - -Every patch under `deploy/docker/patches/`, and where it stands relative to the -project it patches. Kept here so "why do we still carry this?" has one answer -per row, and so a patch that upstream has since merged gets dropped instead of -quietly outliving its reason. - -**Verified with `gh` on 2026-08-01**, except the `patches/sglang_rocm/` section: -2026-08-03 for the host allocator, 2026-08-04 for the staged write-back. -State drifts; re-check before relying on a row. `gh search` -matches titles and bodies, **not diff content**, so "no upstream PR" means "none -found by search", not "none exists" — where a row could be checked by reading -upstream source instead, it says so. - -Column meanings: - -- **ours?** — was the upstream PR opened by a contributor of this repo? - `yes` = us; `no` = a third party; `—` = no PR. -- **PR state** — of the upstream PR named in the same row. - -## sglang — `patches/sglang_dsa/` (baked by `Dockerfile.sglang` and `Dockerfile.sglang.gfx942`, `APPLY_SGLANG_DSA_PATCHES=1`) - -Only patch 01 is baked by both: it is an anchor script, while 02 and 04 are -`--fuzz=0` diffs pinned to v0.5.15.post1 and cannot apply to the gfx942 image's -v0.5.16 base. That image substitutes 02b and 04 at runtime with -`--json-model-override-args '{"index_share_for_mtp_iteration":false}'` and does -not address 02a at all — `patches/sglang_dsa/README.md` carries the reasoning. - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.py` | HIP/aiter paged-MQA sizes its output from DP-padded rows while `lengths` is sized to real rows → `Expected lengths.size(0) == B` | none found | [sglang#33059](https://github.com/sgl-project/sglang/pull/33059) | **yes** (`dorado269`) | OPEN, `REVIEW_REQUIRED` | -| ″ (same bug class, other platform) | — | none found | [sglang#32762](https://github.com/sgl-project/sglang/pull/32762) `[NPU] Fix DSA eager padding mismatch` — our diff is written in its shape | no (`stellaxcpeng`) | OPEN | -| ″ (anchor collision, **not** a fix) | — | — | [sglang#32738](https://github.com/sgl-project/sglang/pull/32738) pads heads for DeepGEMM at the same two aiter call sites; [#31480](https://github.com/sgl-project/sglang/pull/31480) extracts the paged-MQA backend and restructures the `is_aiter()` dispatch | no | both OPEN (re-read 2026-08-03) | -| `sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff` (2a) | `seq_lens.max().item()` is a host sync on a branch only *some* DP ranks take → DP collectives desync → deadlock | none found | none found | — | — | -| `sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff` (2b) | page table has one row per **request**, top-k one per **token** under MTP → `assert page_table.shape[0] == topk_indices.shape[0]` | none found | [sglang#32209](https://github.com/sgl-project/sglang/pull/32209) solves the same row mismatch by **trimming q/top-k**; porting that half here fails at conc=32 and is unresolved | no (`HZY-Wade`) | OPEN, `REVIEW_REQUIRED` | -| `sglang_dsa/draft_cuda_graph_dp_vote.diff` | draft graph/eager choice is per-rank and diverges on the PD decode leg → group deadlock | [sglang#32527](https://github.com/sgl-project/sglang/issues/32527) (independent report, 8× Blackwell) | [sglang#32209](https://github.com/sgl-project/sglang/pull/32209) — same defect, same strategy; **this diff adopts its placement** | no (`HZY-Wade`) | OPEN, `REVIEW_REQUIRED` | - -Prerequisite for the set, applied earlier in the same Dockerfile and **asserted** -by `scripts/apply_sglang_dsa_patches.sh`: - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `sglang/patch_glm52_nextn_quark_exclude.py` | GLM-5.2 MTP `eh_proj` is bf16 but the quark-exclude check probes the bare layer prefix → draft weight-load dies `3072 vs 6144` | none found | [sglang#30265](https://github.com/sgl-project/sglang/pull/30265) `[AMD] Fix GLM-5.2 MTP Quark excludes` — a **superset** (dedicated `GlmMoeDsaForCausalLMNextN`); ours is a narrow backport | no (`wangjiaxin99`) | **MERGED** 2026-07-08 | - -> Our base `v0.5.15.post1` (`0b3bb0c`) predates #30265 — the release line was cut -> without it. **Drop this patch** when the base sglang carries #30265; the anchor -> disappears and the script no-ops. - -Background, already present in the base and **not** patched by us: -[sglang#30378](https://github.com/sgl-project/sglang/pull/30378) / -[#30427](https://github.com/sgl-project/sglang/pull/30427) (MERGED) clamp padded-row -seq_lens **values**; our patch 01 fixes the HIP-side row **count**. -[#30839](https://github.com/sgl-project/sglang/pull/30839) / -[#31083](https://github.com/sgl-project/sglang/pull/31083) (MERGED 2026-07-14) -introduced the guard patch 04 repairs — so that deadlock is a regression in this -baseline, not a legacy wart. -[#32722](https://github.com/sgl-project/sglang/pull/32722) (OPEN) adds a test for -PD + DP-attention + MTP, i.e. **no CI covers this topology today**. - -## sglang PD — `patches/sglang_disagg/` (baked by `Dockerfile.sglang` and `Dockerfile.sglang.gfx942`) - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `sglang_disagg/patch_mooncake_early_send_wait_event.py` | `mooncake/conn.py` never waits on the forward's completion event and the overlap path records none, so chunked prefill hands a non-final chunk to the decode leg while the forward writing those pages is still running → prompts longer than one chunk come back **partially wrong**, with nothing in any log | [sglang#25583](https://github.com/sgl-project/sglang/issues/25583) reports the same corruption shape on GLM-5, but **aggregated** — no PD, no mooncake — so a shared root cause is unestablished; closed **inactive** 2026-07-18, no follow-up | none found | — | — | - -> `prefill.py` already records the completion event this needs; only the `mori` -> backend ever read it, and the patch mirrors what `mori` does. **Drop it** once a -> base sglang synchronizes on that event itself — the script then reports "already -> present" and no-ops. -> -> Unlike the rest of this page, this row was **not** verified with `gh`: the issue -> state was read from the web UI on 2026-08-03, and no upstream PR search was run. -> "none found" here is weaker than elsewhere on the page. - -## sglang ROCm — `patches/sglang_rocm/` (baked by `Dockerfile.sglang`, `Dockerfile.sglang.gfx942`) - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `sglang_rocm/patch_hicache_rocm_host_alloc.py` | hicache allocates host pools with `mmap` + `hipHostRegister`, which on ROCm maps the pages at a device address ≠ the host VA, but the pools hand raw host `data_ptr()`s to GPU kernels via device-side pointer tables → `Memory access fault by GPU node-N on address ` on the first kvd write-back | none found | none found | — | — | -| `sglang_rocm/patch_hicache_rocm_staged_write_back.py` | `pool_host/mla.py` enables the staged write-back JIT on HIP while `DSAIndexerPoolHost` — in the same `HostPoolGroup` for any DSA model — still gates it on `_is_cuda`. The group ANDs the flag, so the controller puts the destination indices on the GPU; the anchor MLA pool then reads its own flag and launches the JIT anyway → `Tensor match failed … device=rocm:0 … allowed options: [cpu, rocm_host]`, scheduler exit −3 on the first write-back | none found | [sglang#28534](https://github.com/sgl-project/sglang/pull/28534) `[AMD] Enable JIT staged HiCache write-back and fix CPU-index crash` — added the HIP enablement and aligned `cache_controller.py`, `memory_pool_host.py`, `pool_host/mha.py`; **never touched `pool_host/mla.py`** | no (`AMD-yanfeiwang`) | **MERGED** 2026-07-09 | - -**`patch_hicache_rocm_host_alloc.py`** — the fault is **gfx950-only so far**: MI300X -(amdgpu 6.14.14, ROCm 7.2.0) measures the two addresses equal, so -`Dockerfile.sglang.gfx942` carries this one preventively rather than to fix a crash. -Don't read its row above as evidence the fault was seen on both arches. - -> Verified on 2026-08-03 by **reading upstream `main` directly** (contents API, -> `pool_host/common.py`): `ALLOC_MEMORY_FUNCS` still overrides only `"npu"` and -> `"musa"`, with no HIP entry — so main is affected too, and this is stronger -> than the usual "no search hit". [sglang#23361](https://github.com/sgl-project/sglang/pull/23361) -> (**MERGED**, MUSA) is the same one-line dispatch override for the same reason -> and is the shape this patch copies. -> -> **No PR of ours has been filed** — it should be. **Drop this patch** when a -> base sglang routes HIP to `alloc_with_pin_memory`; the anchor stops matching -> and the script exits non-zero, so the drop is not silent. Note -> [#32503](https://github.com/sgl-project/sglang/pull/32503) / -> [#32792](https://github.com/sgl-project/sglang/pull/32792) (OPEN, Intel XPU -> HiCache) touch this same dict — expect an anchor conflict, not a fix. - -**`patch_hicache_rocm_staged_write_back.py`** — **not** preventive: without it the -v0.5.16 gfx942 base kills the prefill scheduler on the first reused prefix, so that -image needs it to run kvd at all. A **no-op on the mi35x image**, which has nothing to -fix — every `can_use_write_back_jit` gate at v0.5.15.post1 is still `_is_cuda` (MHA, -MLA, both V4 pools and `DSAIndexerPoolHost`, all in `memory_pool_host.py` before -MLA/MHA moved into `pool_host/`), and `_is_hip` appears only in the kernel import -guard, so the group's AND and its anchor agree on False. #28534 introduced the -disagreement after that tag; the absent `pool_host/mla.py` is only how the script -notices. Both `_is_cuda or _is_hip` in `pool_host/mla.py` and the CUDA-only gates on -the other pools are still on upstream `main` (read from the raw files, so stronger -than a search miss), i.e. **main is affected** — and #28534's own reasoning argues for -the opposite repair, teaching the remaining pools the JIT rather than gating MLA down, -so expect upstream to close this differently than we did. - -> **That repair is already in flight (checked 2026-08-04):** -> [sglang#30350](https://github.com/sgl-project/sglang/pull/30350) `Add HiCache JIT -> test and benchmark for ROCm/HIP CI support` (**OPEN**, `Emmanuel0612`) adds -> `_is_cuda_alike = _is_cuda or _is_hip` and flips exactly the three CUDA-only gates -> (`DSAIndexerPoolHost`, `DeepSeekV4PagedHostPool`, `DeepSeekV4StateHostPool`), so the -> group AND stops reading False on ROCm — **including the V4 stack our patch does not -> cover**. It also teaches `staged_write_back.cuh` to accept kDLROCM/kDLROCMHost (the -> TensorMatcher check that emits our crash) and adds an AMD CI lane for the HiCache -> JIT; the author reports 47/47 on MI355X. Stalled rather than rejected: amd-bot -> called its AMD suites green on 07-09 alongside a merge conflict, conflicts were -> cleared 07-13, nothing since 07-16, and #28534 landed in between. **Our leverage is -> a gfx942 reproduction on that thread, not a competing PR** — it has no MI300X -> datapoint. -> -> **Anchor drift cannot be the drop signal.** #30350 never touches -> `pool_host/mla.py`, so our anchor would keep matching and the patch would keep -> applying on top of the fix — not a crash, since both gates read False again, but a -> silent forfeit of the staged kernel #30350 enables. `check_group_still_poisoned()` -> checks the *precondition* instead: once `DSAIndexerPoolHost` stops gating on -> `_is_cuda` alone, the script refuses and exits 1 telling the operator to drop it. -> -> **Exercised 2026-08-04** against the v0.5.16 *and* current-`main` copies of both -> files, on throwaway trees. Stock: applies, exit 0. Re-run: "already applied", exit -> 0. #30350 simulated by flipping the three CUDA-only gates: refuses, exit 1, with -> `mla.py` byte-identical to pristine. `pool_host/mla.py` removed (the v0.5.15.post1 -> shape): tolerated, exit 0. Anchor or pool renamed: exit 1. -> -> Scope: `DSAIndexerPoolHost` is not the only CUDA-only member that can poison the -> group's AND, and `build_deepseek_v4_hicache_stack` puts `DeepSeekV4PagedHostPool` -> in a group anchored by `LogicalHostPool` (flag unconditionally True). Expect the -> same crash on a V4 hicache stack on gfx942; **this patch gates `mla.py` only**. No -> V4 stack runs on this branch, so that gate would be untested — the script's SCOPE -> section records it for whoever gets there. - -## Mooncake C++ — `patches/mooncake_cpp/` (built by `Dockerfile.sglang`, `Dockerfile.vllm`, `Dockerfile.atom`) - -Pinned to Mooncake `main @ 747003c`; `git apply` fails loudly on ref drift. - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `mooncake_cpp/rdma_auto_chunk_mr_2017.diff` | buffers over the device `max_mr_size` are silently truncated by `ibv_reg_mr` while `BufferDesc.length` advertises the full size → `IBV_WC_REM_ACCESS_ERR` past the boundary | [Mooncake#2017](https://github.com/kvcache-ai/Mooncake/issues/2017) | [Mooncake#2644](https://github.com/kvcache-ai/Mooncake/pull/2644) | **yes** (`jiejingzhangamd`) | **MERGED** 2026-07-28 | -| `mooncake_cpp/rdma_transport_dmabuf_cmake.diff` | `USE_HIP_DMABUF` is defined only on the `transfer_engine` target, so the `ibv_reg_dmabuf_mr` branch compiles **out** of `rdma_transport` — where it is actually called → GPU buffers fall back to bare `ibv_reg_mr`, which cannot pin VRAM without `ib_peer_mem` | none found | none found | — | — | -| `mooncake_cpp/transfer_engine_impl.diff` | `installTransport("hip")` runs unconditionally, so GPU buffers become intra-node HIP IPC segments a cross-node peer cannot open (`Corrupted segment descriptor hipbuffer`) | none found | none found | — | — | - -> #2644 is merged upstream but **not** in the pinned `747003c` tree, so the local -> diff is still applied. Drop it when the pin advances past the merge. - -## vLLM — `patches/vllm/` (baked by `Dockerfile.vllm`) - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `vllm/patch_defer_kv_register.py` | registering the Mooncake KV pool before warmup trips a decode-boot crash at high util (`compile_or_warm_up_model` returns None → `AttributeError: 'NoneType' … language_model`); defer to the end of warmup | none found | none found | — | — | -| `vllm/patch_moriio_pagelen.py` | MoRIIO MLA derives per-block transfer size/stride from tensor **shape**, so block-scaled fp8 MLA + DSA transfers the wrong geometry → PD output is wrong while direct prefill is correct | none found | none found | — | — | -| `vllm/patch_moriio_write.py` | in WRITE (push) mode the decode addresses **itself** (`is_producer=True`), the consumer handler asserts, the notify thread dies and the request hangs | `AMD-AGI/Infera#67` | fixed on vLLM `main` / `v0.22.1rc0` **source**, but no AMD ROCm image ships it | no | n/a (source-only) | -| `vllm/patch_sched_guard.py` | decode EngineCore dies on `assert req_id in self.requests` when a KV-xfer-finished event arrives for an already-removed request | `AMD-AGI/Infera#69` | none found | — | — | -| `vllm/patch_vllm_mooncake_blocksize.py` | backends that force a kernel block size of 1 make the connector index logical pages at kernel granularity → RDMA moves empty rows, decode attends over zeros | none found | none found | — | — | -| `vllm/patch_vllm_mooncake_prom_metrics.py` | `MooncakeConnector` lacks `build_prom_metrics`, so `MultiKVConnectorPromMetrics.observe` asserts and kills the engine under `MultiConnector` | none found | internal PR #178 (the sibling fix for `InferaKvdConnector`) | **yes** | **MERGED** | - -> The `AMD-AGI/Infera#NN` references above are quoted verbatim from the patches' -> own docstrings, which is the citation form this repo already uses. They resolve -> against an internal tracker, so treat a `404` as expected rather than as a -> stale number. - -## vLLM DSv4 — `patches/vllm-dsv4/` (baked by `Dockerfile.vllm`) - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.py` | aiter 0.1.16's `fx.ptrtoint` rejects flydsl memrefs → `MLIRError` in the 2-stage MoE GEMM (Kimi-K2.6 int4 W4A16) | none found | none found | — | — | -| `vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.py` | a global `block_size` equality check kills the prefill worker at KV registration, though DSv4 registers per-layer caches with different block sizes and offsets already use the per-layer map | none found | none found | — | — | -| `vllm-dsv4/patch_moriio_dsv4_noncontig_register.py` | `register_torch_tensor` rejects DSv4's 576B-aligned non-contiguous fp8_ds_mla KV view, and `.contiguous()` would detach from the buffer the forward writes into | none found | none found | — | — | -| `vllm-dsv4/patch_moriio_dsv4_sparse_backend.py` | the generic ROCm selector returns `ROCM_AITER_MLA_SPARSE` (no `fp8_ds_mla`) and raises, killing the prefill worker, though `backend_name` is only a P/D handshake tag | none found | none found | — | — | - -### `patches/vllm-dsv4/legacy/` — archival, **not executed by any Dockerfile** - -Kept for provenance; both are no-ops on the current verified stack -(vLLM `0.23.1rc1.dev748`, `amd-aiter 0.1.16.post2`). - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `vllm-dsv4/legacy/patch_dsv4_aiter_moe.py` | DSv4 MXFP4 MoE gate-mode plumbing, before aiter carried it | none found | [aiter#3123](https://github.com/ROCm/aiter/pull/3123) `[MoE] Align Swiglu MXFP4 fused quant paths` | no (`XiaobingSuper`) | **MERGED** 2026-05-12 | -| `vllm-dsv4/legacy/patch_dsv4_mhc_aiter.py` + `.diff` | `mhc_pre_gemm_sqrsum_kernel` store race → EngineCore dies | none found | [aiter#3033](https://github.com/ROCm/aiter/pull/3033) `Fix sqrsum store race condition` | no (`kkHuang-amd`) | **MERGED** 2026-05-06 | - -## ATOM — `patches/atom/` (baked by `Dockerfile.atom`) - -ATOM is an internal engine; there is no public upstream to file against. - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `atom/patch_gdn_pd_state_transfer.py` | hybrid GDN models never transfer the GatedDeltaNet recurrent state in PD, so decode starts from a zero state and cannot recall prompt context (5/5 mixed vs 0/5 PD) | n/a — internal engine | n/a | — | — | -| `atom/patch_minimax_m2_qknorm_rope.py` | stock `minimax_m2.py` fails to load: `get_rope()` rejects `dtype=`, and the TP fused QK-norm kernel needs a batch guard | n/a | n/a | — | — | -| `atom/patch_mooncake_consumer_slot.py` | `UnboundLocalError: consumer_staging_pool_idx` crashes the decode worker on the first PD request for models with slot state but no `slot_regions` | n/a | n/a | — | — | - -## hipFile — `patches/hipfile_async/` (baked by `Dockerfile.vllm`) - -| patch | fixes | upstream issue | upstream PR | ours? | PR state | -|---|---|---|---|---|---| -| `hipfile_async/hipfile_async.patch` (+ `file_async_fragment.py`, `patch_hipfile_async.sh`) | the ROCm hipFile binding exposes no async API; adds `write_async`/`read_async`, `Stream`, `supports_async()`. Cython stack-locals passed as `&local` die before the driver dereferences them → `bytes_done` reads 0 and intermittent `HipFileException 5022`, so the wrapper calls `libhipfile.so` via ctypes with heap-allocated slots | none found | none found | — | — | - -## Not patches - -Every file under `patches/` is covered above except these, which carry no fix of -their own: `mooncake_cpp/apply_mooncake_cpp_patches.sh` (applies the three -Mooncake diffs), `sglang_dsa/README.md`, `sglang_disagg/README.md` and -`vllm-dsv4/legacy/README.md`. - -## Maintenance - -A row is ready to delete when its upstream PR is **merged and present in the -pinned base**. Merged-but-not-in-base still needs the local patch — sglang#30265 -and Mooncake#2644 are exactly that case. The staged write-back row is the -exception: its `MERGED` PR (sglang#28534) is what *introduced* the defect, so that -patch drops on sglang#30350 instead. - -When adding a patch, add its row here in the same commit, and put the full -argument — evidence, alternatives, how it differs from our own upstream PR — in -the patch's own header. This table is the index, not the record. diff --git a/deploy/docker/patch.upstream.status.yaml b/deploy/docker/patch.upstream.status.yaml new file mode 100644 index 00000000..fda8d661 --- /dev/null +++ b/deploy/docker/patch.upstream.status.yaml @@ -0,0 +1,562 @@ +# yaml-language-server: $schema=patches/_schema/patch.upstream.index.schema.json +# +# Every third-party patch this repo carries, grouped by the library it patches. +# +# This file is an INDEX. It holds only what you need to decide whether to open the +# per-patch record: which engines see the patch, why it is still here, and how you will +# find out when it can go. The argument — root cause, upstream state, evidence, call +# chain, verification — lives in the referenced .upstream.status.yaml, and the +# patch's own header carries the code-level detail. +# +# Validated by scripts/validate-patch-status.py against +# patches/_schema/patch.upstream.index.schema.json, which also cross-checks that every +# patch on disk has a record and that the totals below match. A patch added without a +# record fails the lint gate. +# +# Replaces patch.upstream.status.md (removed in the same change). +schema_version: 1 +status_updated: 2026-08-07 + +verification_note: >- + Every row was re-established on 2026-08-05: patch header read, `gh` queried for the + live state of each PR and issue, and for the rows where a search found nothing, upstream + `main` read directly through the contents API. What is NOT covered: no image was built + or inspected, so the four rows whose effect depends on what the pinned base already + carries are marked `unverified` in their records rather than guessed at. Treat any row + whose status_updated has drifted far from today as a prompt to re-check, not as fact. + UPDATED 2026-08-07: the three sglang rows that had no PR of ours now have one — + #33968 (hicache allocator), #33970 (mooncake wait event), #33973 (DSA host-sync, the + 2a half only). All three are DRAFTS: they were re-validated for equivalence and scope + against the proven local fix on MI300X, but the MI355X / multi-node cluster was + unreachable, so none has been re-run against the original fault on the hardware that + reproduces it. Do not read `carry-upstream-pr-open` on those rows as "fixed upstream". + Full record: pr.done.md and work.todo.md at the repo root. + +pinned_bases: + - surface: Dockerfile.sglang + library: sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + image: + tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: >- + gfx950 / MI355X. On release/v0.5.15, not main — so upstream main fixes do not arrive + by a patch-level bump. release/v0.5.17 already exists upstream. + - surface: Dockerfile.sglang.gfx942 + library: sglang + ref: v0.5.16 + commit: fdebc938f7f4 + pinned_ref_on_main: false + image: + tag: lmsysorg/sglang:v0.5.16-rocm720-mi30x + digest: null + digest_source: unresolved + note: >- + gfx942 / MI325X. On release/v0.5.16. The two sglang bases disagree about which + patches they need, which is why several records carry two target versions. + - surface: Dockerfile.vllm + library: vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + image: + tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: >- + The only digest-pinned base in the tree. vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + - surface: Dockerfile.atom + library: atom + ref: atom0.1.4 + commit: null + pinned_ref_on_main: null + image: + tag: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612 + digest: null + digest_source: unresolved + note: Internal engine — no public repo, so there is no commit and no notion of main. + - surface: all images (MOONCAKE_GIT_REF) + library: mooncake + ref: main @ 747003c + commit: 747003c058015c4077a266e7ccd7549bbc9baede + pinned_ref_on_main: true + image: null + note: >- + Rebuilt in place in every engine image. The only base pinned to a real upstream commit, + and the only one on main. It is 2026-06-26 and two of our own merged fixes are newer, + so advancing it retires two patches at once. + - surface: Dockerfile.vllm (BUILD_AITER=1) + library: aiter + ref: v0.1.16.post1 + commit: null + pinned_ref_on_main: null + image: null + note: >- + Built in-container from a tag. The flydsl patch was verified against 0.1.16.post2, one + patch level above this — they have drifted. + - surface: Dockerfile.vllm (BUILD_HIPFILE=1) + library: hipfile + ref: unpinned-default-branch + commit: null + pinned_ref_on_main: true + image: null + note: >- + NOT PINNED. build_hipfile.sh clones HEAD of ROCm/rocm-systems, so image content is a + function of the build DATE. Two images from the same infera commit built either side of + 2026-07-16 contain different hipfile code. Worth fixing independently of any patch. + +libraries: + sglang: + repo: sgl-project/sglang + note: >- + Two bases, and they need different patch sets. The two --fuzz=0 context diffs apply only + to v0.5.15.post1; the gfx942 image substitutes a runtime flag for one of them and does + not address the other. + patches: + - patch: deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.py + record: deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.upstream.status.yaml + component: quantization + engines: [sglang] + status: drop-candidate + alive_because: >- + sglang#30265 merged 2026-07-08 but release/v0.5.15 was cut without it + drop_signal: base-version-only + ours_upstream_pr: null + summary: >- + GLM-5.2's bf16 MTP layer is built as MXFP4 because the quark exclude probe tests the + bare layer prefix instead of the eh_proj submodule, so the draft weight-load asserts. + A narrow one-line backport of #30265, applied to the mi35x base only. WARNING: the + anchor does NOT disappear on a fixed base — #30265 left the matched string on main — + so decide the drop from the base version, not from the build log. + - patch: deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py + record: deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml + component: disaggregation + engines: [sglang] + status: carry-upstream-pr-open + alive_because: >- + our sglang#33970 is a draft pending 2-node PD re-validation; main confirmed + affected by source read + drop_signal: self-guard-marker-skips + ours_upstream_pr: sgl-project/sglang#33970 + summary: >- + SILENT CORRECTNESS BUG. Chunked prefill over mooncake PD RDMA-reads KV pages while the + forward is still writing them, because the barrier prefill.py records is only ever read + by mori/conn.py — mooncake/conn.py has no wait_event at all. Long prompts come back + partially wrong with no log line. Needle retrieval 5/9 -> 9/9. + - patch: deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.py + record: deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.upstream.status.yaml + component: dsa + engines: [sglang] + status: carry-upstream-pr-open + alive_because: our sglang#33059 is open and unreviewed; both bases are release-branch pins + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: sgl-project/sglang#33059 + summary: >- + The aiter/HIP paged-MQA branch sizes logits and lengths from different row counts, which + diverge in BOTH directions under DP-attention. Reconciles to min(real, padded). The only + sglang_dsa patch that runs on both bases, because it is an anchor script rather than a + --fuzz=0 diff. + - patch: deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff + record: deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml + component: dsa + engines: [sglang] + status: carry-upstream-pr-open + alive_because: >- + only HALF is upstream — our sglang#33973 carries the host-sync deadlock (2a); + the row mismatch (2b) has no PR of ours because #32209's approach fails here at + concurrency 32, so this patch outlives #33973 + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: sgl-project/sglang#33973 + summary: >- + Two independent defects. A blocking device-to-host sync on a DP-divergent branch + deadlocks the group; and the decode page table is per-request while top-k is per-token + under MTP, which asserts. gfx950 only. + - patch: deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.diff + record: deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.upstream.status.yaml + component: speculative-decoding + engines: [sglang] + status: carry-upstream-pr-open + alive_because: >- + sglang#32209 carries this exact fix and is unreviewed; we deliberately opened no + competing PR + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + The draft-CUDA-graph choice is made per rank from two rank-dependent terms, so the DP + group deadlocks on the first PD request. Made a group decision via one extra int64 slot + in the existing all-gather. A regression from merged #30839/#31083, independently + reported upstream as #32527 on Blackwell — not ROCm-specific. + - patch: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py + record: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml + component: hicache + engines: [sglang] + status: carry-upstream-pr-open + alive_because: >- + our sglang#33968 is a draft pending gfx950 re-validation; no third-party issue + or PR; main confirmed affected + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: sgl-project/sglang#33968 + summary: >- + On ROCm, hipHostRegister maps pages at a DIFFERENT device address than the host VA, but + hicache stores host VAs in a device-side pointer table a kernel dereferences — GPU + memory access fault at the host address. Routes ROCm to pin_memory, exactly as the merged + MUSA entry (#23361) does. The clearest missing-PR gap in the tree. + - patch: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.py + record: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.upstream.status.yaml + component: hicache + engines: [sglang] + status: carry-upstream-pr-open + alive_because: >- + drops on sglang#30350 (OPEN, CHANGES_REQUESTED) — note its MERGED sibling #28534 is what + INTRODUCED the defect + drop_signal: precondition-check-refuses + ours_upstream_pr: null + summary: >- + Two gates decide one thing and disagree on ROCm: the MLA pool opts HIP into the staged + JIT while the pools sharing its group do not, so the controller puts indices on the GPU + and the kernel demands them on the host. Kills the scheduler on the first write-back. + Uses a PRECONDITION CHECK rather than an anchor check, because #30350 never touches our + anchor and would let the patch silently outlive its reason. + + vllm: + repo: vllm-project/vllm + note: >- + One digest-pinned base. Patches are baked by Dockerfile.vllm and, for patches/vllm/, also + applied at container start by the overlay. Note the patch loop swallows a patch's + sys.exit(1) into "skipped", so a real failure and a benign skip look identical in the log. + patches: + - patch: deploy/docker/patches/vllm/patch_moriio_pagelen.py + record: deploy/docker/patches/vllm/patch_moriio_pagelen.upstream.status.yaml + component: moriio + engines: [vllm] + status: carry-no-upstream-fix + alive_because: no upstream issue and no PR; main confirmed affected by source read + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + SILENT CORRECTNESS BUG. MoRIIO derives MLA per-block transfer size and stride from tensor + SHAPE rather than spec.page_size_bytes, which disagrees two independent ways: DSv4's + fp8_ds_mla page is padded (the dropped tail holds the scale), and GLM-5.1 pages 16x + larger than its kernel block. PD output wrong while prefill-direct is correct. Localised + by differential against Mooncake on the same nodes. + - patch: deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.py + record: deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.upstream.status.yaml + component: mooncake-connector + engines: [vllm] + status: carry-no-upstream-fix + alive_because: no upstream issue and no PR; main confirmed affected at line 1678 + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + `conv, _ = cache_or_caches` treats Mamba2's two-tensor shape as the contract, so Kimi-K3's + KDA linear attention raises "too many values to unpack" on every rank during KV + registration. Three-line fix, no competing PR. Existed since 2026-08-02 and was never + indexed on the old status page — the gap this system closes. + - patch: deploy/docker/patches/vllm/patch_sched_guard.py + record: deploy/docker/patches/vllm/patch_sched_guard.upstream.status.yaml + component: scheduler + engines: [vllm] + status: carry-no-upstream-fix + alive_because: >- + three open upstream issues describe this exact assert and none has a PR; ours is a + symptom guard we should not upstream as-is + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + Skips a KV-xfer-finished event for an already-removed request instead of asserting, so the + decode EngineCore survives concurrent PD load. Deliberately NOT a fix — the racing request + still loses its transfer — so apply for throughput runs and leave it OFF for correctness + runs, which should surface the race. + - patch: deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.py + record: deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.upstream.status.yaml + component: mooncake-connector + engines: [vllm] + status: carry-upstream-pr-open + alive_because: "vllm#50374 open and unreviewed; #43836 would also resolve it" + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + MooncakeConnector has no build_prom_metrics, so under MultiConnector the Prometheus + registration assert kills the engine on the first request — i.e. exactly the PD + kvd-L3 + recipe with stats on. Patched at build time, not run time, because the runtime approach + loses an import-order race. Our internal #178 fixed only the sibling connector, which + fixed nothing. + - patch: deploy/docker/patches/vllm/patch_defer_kv_register.py + record: deploy/docker/patches/vllm/patch_defer_kv_register.upstream.status.yaml + component: mooncake-connector + engines: [vllm] + status: carry-no-upstream-fix + alive_because: >- + nothing upstream; also the hardest to upstream, because the crash it avoids was routed + around rather than diagnosed + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + Defers Mooncake KV registration to the end of warmup, which avoids a high-util boot crash + where one TP worker's compile_or_warm_up_model returns None. A workaround with an + unexplained mechanism — the weakest-evidence record here, and it says so. + - patch: deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.py + record: deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.upstream.status.yaml + component: mooncake-connector + engines: [vllm] + status: drop-candidate + alive_because: >- + vllm#46807 MERGED 2026-06-30 with the identical repair; only unconfirmed whether the + pinned base carries it + drop_signal: self-guard-marker-skips + ours_upstream_pr: vllm-project/vllm#46334 + summary: >- + THE NEXT ONE TO DELETE. Registers KV at logical-page granularity for backends that force + kernel block size 1. Our #46334 proposed this first and was closed as superseded by + #46807. The guard marker is the name UPSTREAM added, so on a fixed base it self-skips — + safe, but "already patched" in the log reads like success rather than redundancy. + - patch: deploy/docker/patches/vllm/patch_moriio_write.py + record: deploy/docker/patches/vllm/patch_moriio_write.upstream.status.yaml + component: moriio + engines: [vllm] + status: drop-candidate + alive_because: >- + fixed on upstream source (main and v0.22.1rc0); written when no shipped ROCm image had + it, and the base has moved twice since without anyone re-checking + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + One token: in WRITE mode the decode addressed itself (is_producer=True), the consumer + handler asserted, the notify thread died and the request hung at HTTP 000. Main has + is_producer=False at both call sites; v0.25.1 postdates every image the header lists as + affected, so this is probably already redundant. + + # ---- DSv4-specific, from patches/vllm-dsv4/ (Dockerfile.vllm's second loop) ---- + # Grouped here rather than under a directory-shaped key, because the library they patch + # is vLLM. All three were verified on vllm 0.23.x and none was re-verified on the v0.25.1 + # base — the same refactor that retired three sibling patches. This is the single largest + # unverified area in the index. + - patch: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.py + record: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.upstream.status.yaml + component: moriio + engines: [vllm] + status: carry-no-upstream-fix + alive_because: nothing upstream; effect on the current base unverified + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + Drops a global block_size equality check that kills the prefill worker at KV registration, + although DSv4 legitimately registers per-layer caches at different block sizes and the + offset path already consults the per-layer map. + - patch: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.py + record: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.upstream.status.yaml + component: moriio + engines: [vllm] + status: carry-no-upstream-fix + alive_because: nothing upstream; effect on the current base unverified + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + Registers DSv4's 576B-aligned non-contiguous fp8_ds_mla KV view by storage span, as + sglang's mori connector already does. The trap worth knowing: .contiguous() would have + "worked" and silently moved stale KV, since it copies away from the buffer the forward + writes into. + - patch: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.py + record: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.upstream.status.yaml + component: moriio + engines: [vllm] + status: carry-no-upstream-fix + alive_because: nothing upstream; effect on the current base unverified + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + The generic ROCm selector returns a backend that lacks fp8_ds_mla and raises, killing the + prefill worker — for a value used only as a P/D handshake tag. Sets it directly for DSv4 + sparse MLA. The whole fix rests on backend_name still being cosmetic. + + aiter: + repo: ROCm/aiter + note: Built in-container by Dockerfile.vllm from AITER_GIT_REF. + patches: + - patch: deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.py + record: deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.upstream.status.yaml + component: flydsl-moe + engines: [vllm] + status: carry-no-upstream-fix + alive_because: >- + nothing upstream; and verified against aiter 0.1.16.post2 while the Dockerfile now builds + v0.1.16.post1 + drop_signal: self-guard-marker-skips + ours_upstream_pr: null + summary: >- + aiter 0.1.16's fx.ptrtoint rejects the flydsl memrefs vLLM's tensors become, so the + 2-stage MoE GEMM dies with an MLIRError. Reverts gemm1/gemm2 to the memref-friendly + buffer_ops calls. Needed by Kimi-K2.6 int4 W4A16; no-op for DSv4 MXFP4. + + mooncake: + repo: kvcache-ai/Mooncake + note: >- + Rebuilt in place in every engine image from MOONCAKE_GIT_REF and patched with three C++ + diffs. The pin is on main but two of our own merged fixes are newer than it, so ONE pin + advance retires two of these three — sequence them together, because the same advance also + pulls #2682, which is what makes the third patch necessary. + patches: + - patch: deploy/docker/patches/mooncake_cpp/transfer_engine_impl.diff + record: deploy/docker/patches/mooncake_cpp/transfer_engine_impl.upstream.status.yaml + component: transfer-engine + engines: [sglang, vllm, atom] + status: carry-upstream-pr-open + alive_because: our Mooncake#2725 is open and unreviewed + drop_signal: pin-advance-only + ours_upstream_pr: kvcache-ai/Mooncake#2725 + summary: >- + Upstream #2682 installs the HIP (hipIpc) transport unconditionally and prefers it over + RDMA, so cross-node PD dies with hipIpcOpenMemHandle 201. Gates it behind an env var. The + risk here is INVERTED — a build that silently skips the rebuild ships stock #2682 — which + is why every image asserts the gate is present in the .so it will load. + - patch: deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.diff + record: deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.upstream.status.yaml + component: transfer-engine-rdma + engines: [sglang, vllm, atom] + status: drop-candidate + alive_because: our Mooncake#2644 MERGED 2026-07-28; the pin is 2026-06-26 + drop_signal: pin-advance-only + ours_upstream_pr: kvcache-ai/Mooncake#2644 + summary: >- + Buffers larger than the device max_mr_size were silently truncated while BufferDesc kept + advertising the full length, so remote operations past the boundary hit + IBV_WC_REM_ACCESS_ERR. Chunks at registration time and makes the truncation site a hard + error. Ours, merged; only the pin holds it here. + - patch: deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.diff + record: deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.upstream.status.yaml + component: transfer-engine-rdma + engines: [sglang, vllm, atom] + status: drop-candidate + alive_because: Mooncake#2543 MERGED 2026-07-24; the pin is 2026-06-26 + drop_signal: pin-advance-only + ours_upstream_pr: null + summary: >- + Adds dma-buf GPUDirect registration (ibv_reg_dmabuf_mr), which is the ONLY way to pin VRAM + on a fabric without ib_peer_mem — required on crusoe amd-spur. Opt-in via + MOONCAKE_HIP_DMABUF=1, default off. + + atom: + repo: internal + note: >- + ATOM is an internal engine with no public upstream, so "file it upstream" means landing it in + ATOM itself. These three cannot be retired by any external event. + patches: + - patch: deploy/docker/patches/atom/patch_gdn_pd_state_transfer.py + record: deploy/docker/patches/atom/patch_gdn_pd_state_transfer.upstream.status.yaml + component: attention-metadata + engines: [atom] + status: internal-engine + alive_because: internal engine — no public upstream to file against + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + SILENT CORRECTNESS BUG. Hybrid GDN models never transfer the GatedDeltaNet recurrent state + in PD, so decode starts from a zero mamba state and cannot recall anything from the + prompt. Masked because weight-stored world knowledge still answers correctly: 5/5 recall + single-node mixed versus 0/5 in PD. + - patch: deploy/docker/patches/atom/patch_mooncake_consumer_slot.py + record: deploy/docker/patches/atom/patch_mooncake_consumer_slot.upstream.status.yaml + component: mooncake-connector + engines: [atom] + status: internal-engine + alive_because: internal engine — no public upstream to file against + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + consumer_staging_pool_idx is bound only inside the slot_regions branch but read + unconditionally, so models with cache-group slot state and no registered slot_regions + (Qwen3.5 GDN) crash the decode worker with UnboundLocalError on the first PD request. + - patch: deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.py + record: deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.upstream.status.yaml + component: model-definition + engines: [atom] + status: internal-engine + alive_because: internal engine — a base image whose model file and rope API disagree + drop_signal: anchor-drift-fails-loudly + ours_upstream_pr: null + summary: >- + The stock minimax_m2.py does not load — it passes dtype= to a get_rope that does not + accept it — and the TP fused QK-norm kernel needs a batch guard. Makes MiniMax-M2 START; + correct OUTPUT additionally needs ATOM_USE_UNIFIED_ATTN=1 and --block-size 64 at launch. + + hipfile: + repo: ROCm/rocm-systems + note: >- + Built by Dockerfile.vllm from an UNPINNED clone of the default branch, so the target moves on + every build. + patches: + - patch: deploy/docker/patches/hipfile_async/hipfile_async.patch + record: deploy/docker/patches/hipfile_async/hipfile_async.upstream.status.yaml + component: python-bindings + engines: [vllm] + status: drop-candidate + alive_because: >- + ours, MERGED upstream 2026-07-16, and the target is unpinned — so the fix arrives on its + own and this is "not yet deleted" rather than alive + drop_signal: self-guard-marker-skips + ours_upstream_pr: ROCm/rocm-systems#7386 + summary: >- + Adds async stream I/O to the ROCm hipFile Python binding, keeping the driver's + out-parameters on the heap — Cython stack locals passed as &local die before the driver + writes to them, giving bytes_done == 0 and intermittent HipFileException 5022. Upstream + #7386 is the same feature on the same four files by the same author, with a better + mechanism (a Cython AsyncIOHandle owning the slots). Delete ours. + +archived_record: deploy/docker/patches/archived/patch.archived.yaml + +totals: + active_patches: 25 + records: 25 + archived_patches: 6 + by_status: + carry-no-upstream-fix: 8 + carry-upstream-pr-open: 8 + drop-candidate: 6 + internal-engine: 3 + +not_patches: + - path: deploy/docker/patches/mooncake_cpp/apply_mooncake_cpp_patches.sh + role: >- + Driver that applies the three Mooncake C++ diffs, with a `git apply --reverse --check` + so an already-fixed pin is an idempotent skip. + - path: deploy/docker/patches/sglang_dsa/README.md + role: Directory notes for the sglang DSA patch set. + - path: deploy/docker/patches/sglang_disagg/README.md + role: Directory notes for the sglang disaggregation patch. + - path: deploy/docker/patches/archived/vllm-dsv4/README.md + role: Original notes for the archived legacy DSv4 patches, kept as provenance. + +maintenance: + drop_rule: >- + A patch is ready to delete when its upstream fix is merged AND present in the pinned base. + Merged-but-not-in-base still needs the local patch — sglang#30265, Mooncake#2644/#2543 and + rocm-systems#7386 are all that case. Two rows do not fit the rule's shape at all: the staged + write-back drops on sglang#30350 because its MERGED sibling #28534 is what INTRODUCED the + defect, and the Mooncake diffs drop on the pin advancing rather than on any merge. + add_rule: >- + A new patch needs a .upstream.status.yaml beside it and an entry here, or + scripts/validate-patch-status.py fails the lint gate. If a file under patches/ carries no fix + of its own, list it under not_patches instead. + warnings: + - >- + Six patches cannot announce their own obsolescence. Anything with drop_signal + base-version-only or pin-advance-only keeps applying cleanly after upstream has fixed the + problem, so only the pinned version tells you. Review those on every base bump. + - >- + A self-guard marker that keys on text UPSTREAM added — not on text we add — prints + "already patched" on a fixed base. That is safe but indistinguishable from success in the + build log. patch_vllm_mooncake_blocksize.py and patch_moriio_write.py are both this shape. + - >- + Dockerfile.vllm's patch loop swallows sys.exit(1) into "[vllm-patch] skipped", so a genuine + failure and a benign no-op look the same. Read the per-layer build log when bumping the base. + - >- + Four patches carry silent correctness bugs — wrong output, no crash, no log line: + patch_mooncake_early_send_wait_event.py, patch_moriio_pagelen.py, + patch_gdn_pd_state_transfer.py, and patch_vllm_mooncake_blocksize.py. Never assume a smoke + test covers these; three of them pass one. + - >- + HIPFILE_GIT_REF is unpinned, so images from the same infera commit can contain different + hipfile code depending on the build date. That is a reproducibility problem in its own right. diff --git a/deploy/docker/patches/_schema/patch.archived.schema.json b/deploy/docker/patches/_schema/patch.archived.schema.json new file mode 100644 index 00000000..f7a142bd --- /dev/null +++ b/deploy/docker/patches/_schema/patch.archived.schema.json @@ -0,0 +1,199 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/AMD-AGI/infera/deploy/docker/patches/_schema/patch.archived.schema.json", + "title": "Infera retired patches", + "description": "deploy/docker/patches/archived/patch.archived.yaml — one list entry per patch we no longer need. Retired patches get no per-patch record and are not applied by any surface; this file is the whole memory of them, so each entry has to stand alone.", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "status_updated", "patches"], + "properties": { + "schema_version": { "const": 1 }, + "status_updated": { "$ref": "#/$defs/date" }, + "note": { "type": "string", "minLength": 1 }, + + "patches": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "name", + "source", + "library", + "engines_affected", + "retired_on", + "retired_reason", + "problem", + "resolution" + ], + "properties": { + "name": { + "type": "string", + "minLength": 1, + "description": "Original patch file name, without directory." + }, + "source": { + "type": "object", + "additionalProperties": false, + "required": ["original_path", "current_path"], + "properties": { + "original_path": { + "$ref": "#/$defs/repo_path", + "description": "Where it lived while it was applied." + }, + "current_path": { + "anyOf": [ + { "$ref": "#/$defs/repo_path" }, + { "const": "deleted" } + ], + "description": "Where the file sits now, or 'deleted' when it was removed before this record existed — in which case last_commit_with_file must be set so it can still be recovered." + }, + "extra_files": { + "type": "array", + "items": { "$ref": "#/$defs/repo_path" } + }, + "last_commit_with_file": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "type": "null" }], + "description": "Required in spirit when current_path is 'deleted': the commit to recover the file from." + } + } + }, + "library": { "$ref": "#/$defs/library" }, + "component": { "type": ["string", "null"] }, + "engines_affected": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { "$ref": "#/$defs/engine" }, + "description": "Which engines were failing before this patch existed." + }, + "born": { + "type": ["object", "null"], + "additionalProperties": false, + "required": ["date", "commit"], + "properties": { + "date": { "$ref": "#/$defs/date" }, + "commit": { "$ref": "#/$defs/commit" }, + "subject": { "type": "string", "minLength": 1 } + } + }, + "retired_on": { + "type": "object", + "additionalProperties": false, + "required": ["date", "infera_commit"], + "properties": { + "date": { "$ref": "#/$defs/date" }, + "infera_commit": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "const": "pending" }], + "description": "The infera commit that stopped applying it. 'pending' while the retiring change is still in review." + }, + "subject": { "type": "string", "minLength": 1 } + } + }, + "retired_reason": { + "enum": [ + "upstream-fixed-and-in-base", + "upstream-fixed-differently", + "base-refactor-made-anchor-obsolete", + "superseded-by-local-patch", + "never-reproduced-again", + "wrong-diagnosis" + ] + }, + "upstream_fix": { + "type": ["object", "null"], + "additionalProperties": false, + "required": ["ref", "url", "state"], + "properties": { + "ref": { "$ref": "#/$defs/gh_ref" }, + "url": { "$ref": "#/$defs/url" }, + "title": { "type": "string", "minLength": 1 }, + "state": { "enum": ["MERGED", "CLOSED", "OPEN"] }, + "merged_at": { + "anyOf": [{ "$ref": "#/$defs/date" }, { "type": "null" }] + }, + "commit": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "type": "null" }], + "description": "Upstream commit that carries the fix, when known." + }, + "author": { "type": ["string", "null"] }, + "same_approach": { "type": ["boolean", "null"] } + }, + "description": "The upstream change that made this patch unnecessary. null when it was retired for a local reason — a base refactor, or a diagnosis that turned out wrong." + }, + "integrated_by": { + "type": ["object", "null"], + "additionalProperties": false, + "required": ["infera_commit", "how"], + "properties": { + "infera_commit": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "const": "pending" }] + }, + "how": { + "type": "string", + "minLength": 1, + "description": "How the upstream fix arrived here — base image bump, pin advance, engine version bump." + }, + "base_before": { "type": ["string", "null"] }, + "base_after": { "type": ["string", "null"] } + }, + "description": "How and when infera picked up the upstream fix." + }, + "problem": { + "type": "object", + "additionalProperties": false, + "required": ["what", "why", "how", "before_fix", "after_fix", "context"], + "properties": { + "what": { "type": "string", "minLength": 1 }, + "why": { "type": "string", "minLength": 1 }, + "how": { "type": "string", "minLength": 1 }, + "before_fix": { "type": "string", "minLength": 1 }, + "after_fix": { "type": "string", "minLength": 1 }, + "context": { + "type": "string", + "minLength": 1, + "description": "The original failing scenario: model, arch, topology, engine and flags." + }, + "call_chain": { + "type": "array", + "items": { "type": "string", "minLength": 1 } + }, + "symptom_signature": { "type": ["string", "null"] } + } + }, + "resolution": { + "type": "string", + "minLength": 1, + "description": "The one-paragraph account of how this ended, written so it makes sense without the surrounding table." + }, + "reusable_on": { + "type": ["string", "null"], + "description": "Bases where this patch would still apply, for anyone reproducing on an older stack." + }, + "lesson": { + "type": ["string", "null"], + "description": "What this cost us and what would have caught it sooner. Optional, but this is where the value of an archive actually is." + } + } + } + } + }, + + "$defs": { + "date": { + "type": "string", + "format": "date", + "pattern": "^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])$" + }, + "commit": { "type": "string", "pattern": "^[0-9a-f]{7,40}$" }, + "repo_path": { "type": "string", "pattern": "^[A-Za-z0-9._/-]+$" }, + "url": { "type": "string", "format": "uri", "pattern": "^https://" }, + "gh_ref": { + "type": "string", + "pattern": "^([A-Za-z0-9._-]+/[A-Za-z0-9._-]+#[0-9]+|internal#[0-9]+)$" + }, + "engine": { "enum": ["sglang", "vllm", "atom"] }, + "library": { "enum": ["sglang", "vllm", "atom", "mooncake", "mori", "aiter", "hipfile"] } + } +} diff --git a/deploy/docker/patches/_schema/patch.upstream.index.schema.json b/deploy/docker/patches/_schema/patch.upstream.index.schema.json new file mode 100644 index 00000000..157d1b3c --- /dev/null +++ b/deploy/docker/patches/_schema/patch.upstream.index.schema.json @@ -0,0 +1,205 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/AMD-AGI/infera/deploy/docker/patches/_schema/patch.upstream.index.schema.json", + "title": "Infera patch index", + "description": "deploy/docker/patch.upstream.status.yaml — the one-screen view of every patch this repo carries, grouped by the library it patches. Each entry references the per-patch record that holds the argument; this file carries only what you need to decide whether to open it.", + "type": "object", + "additionalProperties": false, + "required": ["schema_version", "status_updated", "libraries", "archived_record", "totals"], + "properties": { + "schema_version": { "const": 1 }, + + "status_updated": { + "$ref": "#/$defs/date", + "description": "Date of the last full sweep. Individual entries carry their own date when they were checked separately." + }, + + "verification_note": { + "type": "string", + "minLength": 1, + "description": "What 'verified' means for this sweep, and what it does not cover." + }, + + "pinned_bases": { + "type": "array", + "description": "What the images actually build against. A merged upstream fix that is not in one of these still needs its local patch.", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["surface", "library", "ref", "commit", "pinned_ref_on_main"], + "properties": { + "surface": { "type": "string", "minLength": 1 }, + "library": { "$ref": "#/$defs/library" }, + "ref": { "type": "string", "minLength": 1 }, + "commit": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "type": "null" }] + }, + "pinned_ref_on_main": { "type": ["boolean", "null"] }, + "image": { + "type": ["object", "null"], + "additionalProperties": false, + "required": ["tag", "digest", "digest_source"], + "properties": { + "tag": { "type": "string", "minLength": 1 }, + "digest": { + "anyOf": [{ "$ref": "#/$defs/sha256" }, { "type": "null" }] + }, + "digest_source": { "enum": ["dockerfile-pin", "unresolved"] } + } + }, + "note": { "type": ["string", "null"] } + } + } + }, + + "libraries": { + "type": "object", + "minProperties": 1, + "additionalProperties": false, + "description": "Keyed by the library being patched. Only libraries this repo actually patches appear; add a key when a new dependency needs one.", + "propertyNames": { "$ref": "#/$defs/library" }, + "patternProperties": { + "^(sglang|vllm|atom|mooncake|mori|aiter|hipfile)$": { + "type": "object", + "additionalProperties": false, + "required": ["repo", "patches"], + "properties": { + "repo": { + "anyOf": [{ "$ref": "#/$defs/gh_repo" }, { "const": "internal" }] + }, + "note": { "type": ["string", "null"] }, + "patches": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "patch", + "record", + "component", + "engines", + "status", + "alive_because", + "drop_signal", + "summary" + ], + "properties": { + "patch": { "$ref": "#/$defs/repo_path" }, + "record": { + "$ref": "#/$defs/repo_path", + "description": "The per-patch .upstream.status.yaml. Must exist and must validate." + }, + "component": { "type": ["string", "null"] }, + "engines": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { "$ref": "#/$defs/engine" } + }, + "status": { + "enum": [ + "carry-no-upstream-fix", + "carry-upstream-pr-open", + "drop-candidate", + "internal-engine", + "archival" + ], + "description": "drop-candidate means an upstream fix exists and merged; it is waiting only on a base or pin bump here." + }, + "alive_because": { + "type": "string", + "minLength": 1, + "description": "One clause. The record carries the argument." + }, + "drop_signal": { + "enum": [ + "anchor-drift-fails-loudly", + "self-guard-marker-skips", + "precondition-check-refuses", + "base-version-only", + "pin-advance-only", + "manual-review" + ] + }, + "ours_upstream_pr": { + "anyOf": [{ "$ref": "#/$defs/gh_ref" }, { "type": "null" }], + "description": "Our own upstream PR for this defect, when one exists. null here on a crash-class fix is a gap worth acting on." + }, + "summary": { "type": "string", "minLength": 1 }, + "status_updated": { "$ref": "#/$defs/date" } + } + } + } + } + } + } + }, + + "archived_record": { + "$ref": "#/$defs/repo_path", + "description": "The single file recording retired patches." + }, + + "totals": { + "type": "object", + "additionalProperties": false, + "required": ["active_patches", "records", "archived_patches"], + "properties": { + "active_patches": { "type": "integer", "minimum": 0 }, + "records": { "type": "integer", "minimum": 0 }, + "archived_patches": { "type": "integer", "minimum": 0 }, + "by_status": { + "type": "object", + "additionalProperties": { "type": "integer", "minimum": 0 } + } + }, + "description": "Cross-checked by the validator against what is on disk, so a patch added without a record fails CI." + }, + + "not_patches": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["path", "role"], + "properties": { + "path": { "$ref": "#/$defs/repo_path" }, + "role": { "type": "string", "minLength": 1 } + } + }, + "description": "Files under patches/ that carry no fix of their own — drivers, READMEs, schemas. The validator uses this list to decide what is allowed to have no record." + }, + + "maintenance": { + "type": "object", + "additionalProperties": false, + "properties": { + "drop_rule": { "type": "string", "minLength": 1 }, + "add_rule": { "type": "string", "minLength": 1 }, + "warnings": { + "type": "array", + "items": { "type": "string", "minLength": 1 } + } + } + } + }, + + "$defs": { + "date": { + "type": "string", + "format": "date", + "pattern": "^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])$" + }, + "commit": { "type": "string", "pattern": "^[0-9a-f]{7,40}$" }, + "sha256": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" }, + "repo_path": { "type": "string", "pattern": "^[A-Za-z0-9._/-]+$" }, + "gh_repo": { "type": "string", "pattern": "^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$" }, + "gh_ref": { + "type": "string", + "pattern": "^([A-Za-z0-9._-]+/[A-Za-z0-9._-]+#[0-9]+|internal#[0-9]+)$" + }, + "engine": { "enum": ["sglang", "vllm", "atom"] }, + "library": { "enum": ["sglang", "vllm", "atom", "mooncake", "mori", "aiter", "hipfile"] } + } +} diff --git a/deploy/docker/patches/_schema/patch.upstream.status.schema.json b/deploy/docker/patches/_schema/patch.upstream.status.schema.json new file mode 100644 index 00000000..4bd2563f --- /dev/null +++ b/deploy/docker/patches/_schema/patch.upstream.status.schema.json @@ -0,0 +1,585 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/AMD-AGI/infera/deploy/docker/patches/_schema/patch.upstream.status.schema.json", + "title": "Infera per-patch upstream status", + "description": "One record per patch under deploy/docker/patches/. Lives beside the patch it describes, named .upstream.status.yaml. The patch's own header carries the full argument; this file is the machine-checkable index of where that patch stands against the project it patches.", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "status_updated", + "verified_by", + "patch", + "target", + "upstream_main_affected", + "applies_to", + "alive_because", + "history", + "upstream_issues", + "upstream_prs", + "related_refs", + "problem" + ], + "properties": { + "schema_version": { + "description": "Bumped when this schema makes a breaking change.", + "const": 1 + }, + + "status_updated": { + "$ref": "#/$defs/date", + "description": "Date every upstream claim in this file was last checked. Upstream moves and this file does not, so a stale date is the signal to re-check rather than to trust." + }, + + "verified_by": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "enum": [ + "gh-pr-issue-state", + "gh-search", + "upstream-source-read", + "git-compare", + "local-repro", + "web-ui", + "patch-header-only" + ] + }, + "description": "How the upstream claims below were established. 'gh-search' proves only that a title/body search missed — it does not prove absence. 'upstream-source-read' is stronger: the defect was confirmed present or absent by reading the upstream file." + }, + + "patch": { + "type": "object", + "additionalProperties": false, + "required": ["path", "kind", "applied_by"], + "properties": { + "path": { + "$ref": "#/$defs/repo_path", + "description": "Primary patch file, relative to the git root." + }, + "extra_files": { + "type": "array", + "items": { "$ref": "#/$defs/repo_path" }, + "description": "Further files that are part of the same patch (a fragment appended by a script, a companion .diff)." + }, + "kind": { + "enum": ["python-anchor-script", "shell-script", "context-diff", "git-diff"], + "description": "python-anchor-script and shell-script locate their target at run time and no-op on a miss; context-diff and git-diff fail loudly on ref drift." + }, + "idempotent": { + "type": "boolean", + "description": "Re-running is a no-op. Absent means unknown, which for a patch that runs on every container start is a defect." + }, + "applied_by": { + "type": "array", + "minItems": 1, + "items": { + "enum": [ + "Dockerfile.sglang", + "Dockerfile.sglang.gfx942", + "Dockerfile.vllm", + "Dockerfile.atom", + "deploy/overlay/Dockerfile.payload", + "deploy/overlay/infera-exec", + "deploy/docker/scripts/apply_sglang_dsa_patches.sh", + "deploy/docker/scripts/build_mooncake_rocm.sh", + "deploy/docker/scripts/build_mooncake_sglang.sh", + "deploy/docker/patches/mooncake_cpp/apply_mooncake_cpp_patches.sh", + "none" + ] + }, + "description": "Every surface that applies this patch. 'none' means carried but not wired up, which needs a reason in alive_because.detail." + }, + "opt_in_flag": { + "type": ["string", "null"], + "description": "Build ARG or env var that gates application, when it is not applied unconditionally." + } + } + }, + + "target": { + "type": "object", + "additionalProperties": false, + "required": ["library", "repo", "files", "versions"], + "properties": { + "library": { + "$ref": "#/$defs/library", + "description": "The project whose source this patch edits — not the engine that runs it. A patch to vLLM's bundled MoRIIO connector has library 'vllm', component 'moriio'." + }, + "component": { + "type": ["string", "null"], + "description": "Subsystem inside that library: mooncake, moriio, hicache, dsa, disaggregation, hipfile, flydsl-moe, scheduler." + }, + "repo": { + "anyOf": [ + { "$ref": "#/$defs/gh_repo" }, + { "const": "internal" } + ], + "description": "owner/name on GitHub, or 'internal' for a closed-source engine." + }, + "files": { + "type": "array", + "minItems": 1, + "items": { "type": "string", "minLength": 1 }, + "description": "Paths edited inside the target project, as the patch addresses them." + }, + "versions": { + "type": "array", + "minItems": 1, + "description": "Every pinned version of the target this patch is carried against. Two entries when a patch spans two engine bases — they can disagree about whether the defect is even present.", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["ref", "commit", "pinned_ref_on_main"], + "properties": { + "surface": { + "type": ["string", "null"], + "description": "Which build this version belongs to, when the patch spans more than one." + }, + "ref": { + "type": "string", + "minLength": 1, + "description": "Tag, branch, version string, or 'unpinned-default-branch'." + }, + "commit": { + "anyOf": [{ "$ref": "#/$defs/commit" }, { "type": "null" }], + "description": "Commit the ref resolves to. null only when the ref is genuinely unpinned, in which case unpinned_risk must say what that costs." + }, + "pinned_ref_on_main": { + "type": ["boolean", "null"], + "description": "Is the pinned commit an ancestor of the upstream default branch? false means a release branch, so upstream main fixes do not arrive by bumping a patch level. null for internal or unpinned targets." + }, + "release_branch": { + "type": ["string", "null"], + "description": "Upstream branch the pinned ref sits on, when it is not main." + }, + "unpinned_risk": { + "type": ["string", "null"], + "description": "Required in spirit when commit is null: what an unpinned target means for this patch." + }, + "images": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["tag", "digest", "digest_source"], + "properties": { + "tag": { "type": "string", "minLength": 1 }, + "digest": { + "anyOf": [{ "$ref": "#/$defs/sha256" }, { "type": "null" }] + }, + "digest_source": { + "enum": ["dockerfile-pin", "unresolved"], + "description": "'unresolved' means the Dockerfile pins the tag only; the digest is whatever the registry serves at build time." + }, + "note": { "type": ["string", "null"] } + } + } + } + } + } + } + } + }, + + "upstream_main_affected": { + "type": "object", + "additionalProperties": false, + "required": ["value", "evidence"], + "properties": { + "value": { + "type": ["boolean", "null"], + "description": "Does the upstream default branch still carry the defect this patch fixes? null when there is no public upstream to check." + }, + "evidence": { + "type": "string", + "minLength": 1, + "description": "What settles it. Naming the file and line read from main is the standard here; 'no search hit' is not." + } + } + }, + + "applies_to": { + "type": "array", + "minItems": 1, + "description": "Field 4a — what this patch actually does in each engine image it reaches.", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["engine", "surface", "effect", "evidence"], + "properties": { + "engine": { "$ref": "#/$defs/engine" }, + "surface": { + "type": "string", + "minLength": 1, + "description": "Dockerfile or runtime path this row is about." + }, + "effect": { + "enum": ["op", "no-op", "not-applied", "unverified"], + "description": "op = changes bytes and is load-bearing. no-op = reached but changed nothing (target absent, already fixed, or anchor gone). not-applied = never traversed on this surface. unverified = never checked on this surface, which is a gap, not a state." + }, + "required_to_run": { + "type": "boolean", + "description": "true when the engine cannot serve the target workload without this patch." + }, + "evidence": { "type": "string", "minLength": 1 } + } + } + }, + + "alive_because": { + "type": "object", + "additionalProperties": false, + "required": ["reason", "detail", "consumers", "drop_when", "drop_signal"], + "description": "Field 4b — why we still carry this.", + "properties": { + "reason": { + "enum": [ + "no-upstream-pr", + "upstream-pr-open", + "upstream-pr-closed-unfixed", + "merged-not-in-pinned-base", + "merged-not-released", + "released-but-infera-not-bumped", + "upstream-fixed-differently", + "internal-engine-no-upstream", + "local-only-by-design", + "archival" + ] + }, + "detail": { "type": "string", "minLength": 1 }, + "consumers": { + "type": "array", + "minItems": 1, + "items": { "type": "string", "minLength": 1 }, + "description": "Which infera engines, images or workloads still need it. 'none' as the sole entry means the patch is a drop candidate on every surface." + }, + "drop_when": { + "type": "string", + "minLength": 1, + "description": "The concrete condition that retires this patch." + }, + "drop_signal": { + "enum": [ + "anchor-drift-fails-loudly", + "self-guard-marker-skips", + "precondition-check-refuses", + "base-version-only", + "pin-advance-only", + "manual-review" + ], + "description": "How we find out. 'base-version-only' and 'pin-advance-only' mean the patch keeps applying cleanly on a fixed base, so nothing fails and a human has to decide." + }, + "silent_misapply_risk": { + "type": ["string", "null"], + "description": "What goes wrong if this patch outlives its reason and nobody notices. Non-null whenever drop_signal cannot self-announce." + } + } + }, + + "history": { + "type": "object", + "additionalProperties": false, + "required": ["born", "last_modified"], + "properties": { + "born": { "$ref": "#/$defs/commit_event" }, + "last_modified": { + "allOf": [ + { "$ref": "#/$defs/commit_event" }, + { + "type": "object", + "required": ["reason"], + "properties": { + "reason": { + "type": "string", + "minLength": 1, + "description": "Why it changed — not a restatement of the commit subject." + } + } + } + ] + }, + "supersedes": { + "type": "array", + "items": { "type": "string", "minLength": 1 }, + "description": "Earlier patch files this one replaced." + } + } + }, + + "upstream_issues": { + "description": "Field 7 — upstream issues for this defect. null is a legitimate value and must be written explicitly.", + "anyOf": [ + { "type": "null" }, + { + "type": "array", + "minItems": 1, + "items": { "$ref": "#/$defs/upstream_ref" } + } + ] + }, + + "upstream_prs": { + "description": "Field 8 — PRs that fix, or claim to fix, this defect.", + "anyOf": [ + { "type": "null" }, + { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["ref", "url", "title", "state", "author", "ours", "same_approach", "approach_note"], + "properties": { + "ref": { "$ref": "#/$defs/gh_ref" }, + "url": { "$ref": "#/$defs/url" }, + "title": { "type": "string", "minLength": 1 }, + "state": { "$ref": "#/$defs/pr_state" }, + "review_decision": { + "anyOf": [ + { "enum": ["APPROVED", "CHANGES_REQUESTED", "REVIEW_REQUIRED"] }, + { "type": "null" } + ] + }, + "merged_at": { + "anyOf": [{ "$ref": "#/$defs/date" }, { "type": "null" }] + }, + "author": { + "type": ["string", "null"], + "description": "GitHub handle. null means not recorded — better than a placeholder, and note that `ours` can still be answered without it." + }, + "ours": { + "type": "boolean", + "description": "Opened by a contributor of this repo." + }, + "same_approach": { + "type": ["boolean", "null"], + "description": "Does it fix the defect the way our patch does? null when it does not fix this defect at all and is listed for collision or context." + }, + "approach_note": { + "type": "string", + "minLength": 1, + "description": "Why ours differs, or why it does not. 'we needed a local narrow fix and are not carrying the general case' is a valid and useful answer." + }, + "requested_action": { + "type": ["string", "null"], + "description": "What an upstream maintainer explicitly asked for, and who asked." + }, + "in_pinned_base": { + "type": ["boolean", "null"], + "description": "Is this PR present in the version we actually build against? A merged PR that is not in the pinned base still needs the local patch." + } + } + } + } + ] + }, + + "related_refs": { + "description": "Field 9 — PRs and issues that do not decide whether we keep this patch but inform how it should be fixed: the same defect on another platform, a refactor that will move our anchor, a test that proves no CI covers this.", + "anyOf": [ + { "type": "null" }, + { + "type": "array", + "minItems": 1, + "items": { + "allOf": [ + { "$ref": "#/$defs/upstream_ref" }, + { + "type": "object", + "required": ["relevance"], + "properties": { + "relevance": { + "enum": [ + "same-defect-other-platform", + "anchor-collision", + "alternative-repair", + "introduced-the-defect", + "adjacent-site", + "missing-test-coverage", + "shape-we-copied", + "background" + ] + } + } + } + ] + } + } + ] + }, + + "problem": { + "type": "object", + "additionalProperties": false, + "required": ["what", "why", "how", "before_fix", "after_fix", "context"], + "properties": { + "what": { + "type": "string", + "minLength": 1, + "description": "The defect, in one or two sentences." + }, + "why": { + "type": "string", + "minLength": 1, + "description": "Root cause — why the upstream code is wrong, not just what it does." + }, + "how": { + "type": "string", + "minLength": 1, + "description": "What the patch changes, and why that is the minimal correct change." + }, + "before_fix": { + "type": "string", + "minLength": 1, + "description": "Observed failure. Quote the error or the wrong output." + }, + "after_fix": { + "type": "string", + "minLength": 1, + "description": "Observed behaviour with the patch, measured where possible." + }, + "context": { + "type": "string", + "minLength": 1, + "description": "Which model, topology, arch and flags this reproduces under, and where it does not." + }, + "call_chain": { + "type": "array", + "items": { "type": "string", "minLength": 1 }, + "description": "Ordered path from entry point to failure site." + }, + "symptom_signature": { + "type": ["string", "null"], + "description": "The grep-able fingerprint — the exact assert, error string or fault address shape." + }, + "silent": { + "type": "boolean", + "description": "true when the defect produces wrong results rather than a crash. These are the expensive ones." + } + } + }, + + "verification": { + "type": "object", + "additionalProperties": false, + "description": "How the fix was established locally.", + "properties": { + "date": { "$ref": "#/$defs/date" }, + "hardware": { "type": "string", "minLength": 1 }, + "software": { "type": "string", "minLength": 1 }, + "workload": { "type": "string", "minLength": 1 }, + "result": { "type": "string", "minLength": 1 }, + "notes": { "type": ["string", "null"] } + } + }, + + "scope_limits": { + "type": ["string", "null"], + "description": "What this patch deliberately does not cover, so the next person does not assume it does." + }, + + "open_actions": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["action", "owner"], + "properties": { + "action": { "type": "string", "minLength": 1 }, + "owner": { "type": "string", "minLength": 1 }, + "blocked_on": { "type": ["string", "null"] } + } + }, + "description": "What still has to happen upstream or here. An empty list is fine; a missing PR of ours is not." + } + }, + + "$defs": { + "date": { + "type": "string", + "format": "date", + "pattern": "^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])$" + }, + "commit": { + "type": "string", + "pattern": "^[0-9a-f]{7,40}$", + "description": "Lowercase hex, 7 to 40 characters." + }, + "sha256": { + "type": "string", + "pattern": "^sha256:[0-9a-f]{64}$" + }, + "url": { + "type": "string", + "format": "uri", + "pattern": "^https://" + }, + "repo_path": { + "type": "string", + "pattern": "^[A-Za-z0-9._/-]+$", + "description": "Path relative to the git root, no leading slash and no '..'." + }, + "gh_repo": { + "type": "string", + "pattern": "^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$" + }, + "gh_ref": { + "type": "string", + "pattern": "^([A-Za-z0-9._-]+/[A-Za-z0-9._-]+#[0-9]+|internal#[0-9]+)$", + "description": "owner/name#123, or internal#123 for the private tracker." + }, + "engine": { + "enum": ["sglang", "vllm", "atom"] + }, + "library": { + "enum": ["sglang", "vllm", "atom", "mooncake", "mori", "aiter", "hipfile"] + }, + "pr_state": { + "enum": ["OPEN", "MERGED", "CLOSED", "DRAFT", "not-submitted", "source-only"] + }, + "issue_state": { + "enum": ["OPEN", "CLOSED"] + }, + "commit_event": { + "type": "object", + "required": ["date", "commit", "subject"], + "properties": { + "date": { "$ref": "#/$defs/date" }, + "commit": { "$ref": "#/$defs/commit" }, + "subject": { "type": "string", "minLength": 1 }, + "reason": { "type": "string", "minLength": 1 } + }, + "additionalProperties": false + }, + "upstream_ref": { + "type": "object", + "required": ["ref", "url", "kind", "state", "note"], + "properties": { + "ref": { "$ref": "#/$defs/gh_ref" }, + "url": { "$ref": "#/$defs/url" }, + "kind": { "enum": ["pr", "issue"] }, + "title": { "type": "string", "minLength": 1 }, + "state": { + "anyOf": [ + { "$ref": "#/$defs/pr_state" }, + { "$ref": "#/$defs/issue_state" } + ] + }, + "state_reason": { + "type": ["string", "null"], + "description": "Why it closed — 'inactive', 'superseded by #X', 'fixed'." + }, + "author": { "type": ["string", "null"] }, + "relevance": { "type": "string" }, + "note": { + "type": "string", + "minLength": 1, + "description": "What this reference tells us that the title does not." + } + }, + "additionalProperties": false + } + } +} diff --git a/deploy/docker/patches/archived/patch.archived.yaml b/deploy/docker/patches/archived/patch.archived.yaml new file mode 100644 index 00000000..f9f1a687 --- /dev/null +++ b/deploy/docker/patches/archived/patch.archived.yaml @@ -0,0 +1,360 @@ +# yaml-language-server: $schema=../_schema/patch.archived.schema.json +# +# Retired patches. These are applied by nothing and have no per-patch record — this +# file is their entire memory, so each entry is written to stand on its own. +# +# Two kinds of entry: +# * source still present, moved under archived/ — recoverable by reading the file +# * source deleted before this file existed — recoverable from the named commit +# +# Nothing here should be deleted to tidy up. The value of an archive is answering +# "did we already try this, and what happened" two years later. +schema_version: 1 +status_updated: 2026-08-05 +note: >- + Added in the change that introduced the per-patch record system. The four legacy / + already-deleted patches below were previously described in prose in + patch.upstream.status.md, or in the case of the three MoRIIO patches, only in a + Dockerfile comment. Source files were MOVED rather than deleted so this can be + reviewed in a PR without losing anything. + +patches: + - name: patch_dsv4_aiter_moe.py + source: + original_path: deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_aiter_moe.py + current_path: deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_aiter_moe.py + last_commit_with_file: null + library: aiter + component: moe + engines_affected: + - vllm + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + retired_on: + date: 2026-08-05 + infera_commit: pending + subject: "move the legacy DSv4 patches under archived/ and record them" + retired_reason: upstream-fixed-and-in-base + upstream_fix: + ref: ROCm/aiter#3123 + url: https://github.com/ROCm/aiter/pull/3123 + title: "[MoE] Align Swiglu MXFP4 fused quant paths" + state: MERGED + merged_at: 2026-05-12 + commit: null + author: XiaobingSuper + same_approach: true + integrated_by: + infera_commit: pending + how: >- + Arrived by rebuilding from the open-source vllm/vllm-openai-rocm base instead of the + private pre-baked image early bring-up used. No deliberate bump — the fix was already + in the public base. + base_before: private pre-baked DSv4 image (older vLLM / aiter) + base_after: vLLM 0.23.1rc1.dev748+g2dfaae752 with amd-aiter 0.1.16.post2 + problem: + what: >- + DSv4 MXFP4 MoE needed gate-mode plumbing (SWIGLUOAI / activation_interleave) that aiter + did not yet carry, so the fused quant path could not be selected correctly. + why: >- + aiter's rocm_aiter_moe.py had no notion of the interleaved-activation SwiGLU variant DSv4 + uses, so there was no way to ask for it from vLLM. + how: >- + Monkey-patched rocm_aiter_moe.py to add the gate-mode plumbing, with a self-detect that + skips if upstream already has it. + before_fix: DSv4 MXFP4 MoE could not select the correct fused quant path. + after_fix: >- + Upstream aiter carries SWIGLUOAI / activation_interleave and defaults gate_mode="" for + Silu, so the patch self-detects and skips. + context: >- + DeepSeek-V4 MXFP4 MoE on vLLM ROCm during early DSv4 bring-up, debugged against a private + pre-baked image with an older aiter. + call_chain: + - "vLLM MoE layer -> aiter rocm_aiter_moe.py fused quant path selection" + symptom_signature: null + resolution: >- + Obsolete by the time it was first published. Early DSv4 bring-up was debugged against a + private pre-baked image with an older aiter; once the image was rebuilt from the + open-source vllm/vllm-openai-rocm base and markers were checked on real hardware, aiter + already carried the feature via #3123 (merged 2026-05-12). The patch self-detects upstream + and skips, so it had been a no-op on every published image. + reusable_on: >- + An aiter older than #3123 — e.g. the earlier vLLM dev301/dev424 bases. Reference the file + under archived/ to reproduce on such a stack. + lesson: >- + Patches written against a private pre-baked image should be re-verified against the public + base before being carried, not after. Three of the four entries in this file are the same + story. + + - name: patch_dsv4_mhc_aiter.py + source: + original_path: deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_mhc_aiter.py + current_path: deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.py + extra_files: + - deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.diff + last_commit_with_file: null + library: aiter + component: mhc + engines_affected: + - vllm + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + retired_on: + date: 2026-08-05 + infera_commit: pending + subject: "move the legacy DSv4 patches under archived/ and record them" + retired_reason: upstream-fixed-differently + upstream_fix: + ref: ROCm/aiter#3033 + url: https://github.com/ROCm/aiter/pull/3033 + title: Fix sqrsum store race condition + state: MERGED + merged_at: 2026-05-06 + commit: null + author: kkHuang-amd + same_approach: false + integrated_by: + infera_commit: pending + how: >- + Same as the sibling entry — arrived with the move to the open-source base, not by a + deliberate bump. + base_before: private pre-baked DSv4 image (older vLLM / aiter) + base_after: vLLM 0.23.1rc1.dev748+g2dfaae752 with amd-aiter 0.1.16.post2 + problem: + what: >- + A store race in aiter's mhc_pre_gemm_sqrsum_kernel killed the vLLM EngineCore. + why: >- + The kernel's sqrsum store was not correctly synchronized, so concurrent stores raced. + how: >- + Locally, an env-gated backend selection (VLLM_MHC_BACKEND) that routed around the racing + path, hooked on a _mhc_backend() anchor. + before_fix: EngineCore dies during DSv4 MHC execution. + after_fix: >- + Upstream fixed the race itself and made aiter the MHC default — mhc.py now reads + `if HAS_AITER_MHC and hidden % 256 == 0: mhc_pre_aiter` with no env gate at all — so the + local anchor _mhc_backend() no longer exists and the patch auto-skips. + context: >- + DeepSeek-V4 MHC on vLLM ROCm during early bring-up against an older aiter. + call_chain: + - "vLLM DSv4 MHC path -> aiter mhc.py backend selection -> mhc_pre_gemm_sqrsum_kernel" + symptom_signature: EngineCore death during DSv4 MHC execution + resolution: >- + Functionally upstream in a DIFFERENT form, which is why it retired quietly rather than by + the anchor drifting into a conflict. We routed around the race with an env-gated backend + choice; upstream fixed the race (#3033, merged 2026-05-06) and then made aiter the MHC + default outright, so the env gate our anchor keyed on does not exist. The .diff is the + unified-diff companion, never executed, kept as reference. + reusable_on: "An aiter older than #3033, e.g. the earlier vLLM dev301/dev424 bases." + lesson: >- + "Functionally upstream, different form" is the failure mode worth naming: the local patch + does not conflict, it simply stops matching, so nothing announces that it has become dead + weight. This is the same class as the still-active drop candidates whose drop_signal is + base-version-only. + + - name: patch_moriio_dsa_write.py + source: + original_path: deploy/docker/patches/vllm/patch_moriio_dsa_write.py + current_path: deleted + last_commit_with_file: 89c86fb + library: vllm + component: moriio + engines_affected: + - vllm + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + retired_on: + date: 2026-07-23 + infera_commit: "3375773" + subject: Remove three no-op MoRIIO patches dead on the v0.25.1 base + retired_reason: base-refactor-made-anchor-obsolete + upstream_fix: null + integrated_by: + infera_commit: "3375773" + how: >- + vLLM base bump to v0.25.1, which refactored the MoRIIO connector into moriio_layout.py. + base_before: vLLM 0.23.x ROCm + base_after: vllm/vllm-openai-rocm:v0.25.1 + problem: + what: >- + GLM-5.1 over MoRIIO PD needed a full-layer wait_for_save so the KV write completed before + the transfer was considered done. + why: >- + The pre-v0.25.1 MoRIIO connector did not wait for the whole layer to be saved on the DSA + write path, so a transfer could observe a partially written layer. + how: Monkey-patched the connector to wait for the full layer on save. + before_fix: GLM-5.1 MoRIIO PD could transfer a partially-written layer. + after_fix: >- + v0.25.1's moriio_layout.py handles full-layer wait_for_save natively, so the patch's anchor + no longer matches and it contributed nothing. + context: GLM-5.1 over MoRIIO PD on vLLM ROCm 0.23.x. + call_chain: + - "moriio connector save path -> wait_for_save on the DSA write" + symptom_signature: null + resolution: >- + Retired as part of a group of three. v0.25.1 refactored the MoRIIO connector into + moriio_layout.py, which natively covers all three concerns these patches addressed, so their + anchors stopped matching. They were deleted rather than archived at the time — specifically + to stop the build log printing misleading "skipped" lines — which is why this entry has no + file and cites commit 89c86fb for recovery instead. + reusable_on: vLLM 0.23.x and earlier ROCm bases, before the moriio_layout.py refactor. + lesson: >- + Deleting them was right, but the reasoning lived only in a Dockerfile comment. That is the + gap this file closes. + + - name: patch_moriio_hetero.py + source: + original_path: deploy/docker/patches/vllm/patch_moriio_hetero.py + current_path: deleted + last_commit_with_file: 89c86fb + library: vllm + component: moriio + engines_affected: + - vllm + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + retired_on: + date: 2026-07-23 + infera_commit: "3375773" + subject: Remove three no-op MoRIIO patches dead on the v0.25.1 base + retired_reason: base-refactor-made-anchor-obsolete + upstream_fix: null + integrated_by: + infera_commit: "3375773" + how: vLLM base bump to v0.25.1 and its moriio_layout.py refactor. + base_before: vLLM 0.23.x ROCm + base_after: vllm/vllm-openai-rocm:v0.25.1 + problem: + what: >- + Per-layer transfer geometry was not handled for heterogeneous layer layouts, so GLM-5.1 + MoRIIO PD computed transfer offsets from a single global geometry. + why: >- + The pre-v0.25.1 connector assumed a uniform layer geometry across the model. + how: Monkey-patched the connector to compute transfer geometry per layer. + before_fix: Wrong transfer geometry on models with heterogeneous layer layouts. + after_fix: >- + v0.25.1's moriio_layout.py derives per-layer transfer geometry natively, so the anchor + no longer matched. + context: GLM-5.1 over MoRIIO PD on vLLM ROCm 0.23.x. + call_chain: + - "moriio connector -> per-layer transfer geometry computation" + symptom_signature: null + resolution: >- + One of the three retired together on the v0.25.1 move. Note what did NOT retire with them: + patch_moriio_pagelen.py addresses the same file and the same subject — MLA per-block transfer + geometry — and is still load-bearing, because the refactor changed WHERE the geometry is + computed without fixing that it is derived from tensor shape instead of spec.page_size_bytes. + reusable_on: vLLM 0.23.x and earlier ROCm bases. + lesson: >- + A refactor that makes three patches obsolete does not necessarily make the fourth one + obsolete. Each anchor has to be checked individually against the new base. + + - name: patch_vllm_moriio_blocksize.py + source: + original_path: deploy/docker/patches/vllm/patch_vllm_moriio_blocksize.py + current_path: deleted + last_commit_with_file: 89c86fb + library: vllm + component: moriio + engines_affected: + - vllm + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + retired_on: + date: 2026-07-23 + infera_commit: "3375773" + subject: Remove three no-op MoRIIO patches dead on the v0.25.1 base + retired_reason: base-refactor-made-anchor-obsolete + upstream_fix: null + integrated_by: + infera_commit: "3375773" + how: vLLM base bump to v0.25.1 and its moriio_layout.py refactor. + base_before: vLLM 0.23.x ROCm + base_after: vllm/vllm-openai-rocm:v0.25.1 + problem: + what: >- + The MoRIIO connector ignored the kernel/logical block-size ratio, so it indexed logical + pages at kernel granularity. + why: >- + Same defect class as the Mooncake connector's: backends that force a kernel block size of + 1 make cache.shape[0] a multiple of the logical page count, and the connector registered + at the wrong granularity. + how: Monkey-patched the connector to account for the ratio. + before_fix: >- + Transfers addressed the wrong offsets for models whose kernel block size differs from the + scheduler's. + after_fix: >- + v0.25.1's moriio_layout.py handles the kernel/logical ratio natively for the K/V branches + (via kernel_blocks_per_block), so the anchor no longer matched. + context: GLM-5.1 over MoRIIO PD on vLLM ROCm 0.23.x. + call_chain: + - "moriio connector registration -> block_len / num_blocks at kernel rather than logical granularity" + symptom_signature: null + resolution: >- + The third of the group retired on the v0.25.1 move. Worth pairing with its Mooncake + counterpart: patch_vllm_mooncake_blocksize.py is the same defect in the Mooncake connector, + and that one was fixed by UPSTREAM (vllm#46807, merged 2026-06-30) rather than by a refactor. + Same bug, two connectors, two entirely different retirement routes. + reusable_on: vLLM 0.23.x and earlier ROCm bases. + lesson: >- + The kernel-versus-logical block-size confusion has now appeared in three connectors + (Mooncake, MoRIIO, and MoRIIO MLA geometry). It is a design-level trap in the connector + interface, not three unrelated bugs. + + - name: dsa_indexer_hip_dp_padded_rows.diff + source: + original_path: deploy/docker/patches/sglang_dsa/dsa_indexer_hip_dp_padded_rows.diff + current_path: deleted + last_commit_with_file: c91db76 + library: sglang + component: dsa + engines_affected: + - sglang + born: + date: 2026-08-01 + commit: c91db76 + subject: "sglang DSA: enable PD + DP-attention + EAGLE MTP for GLM-5.2 on gfx950" + retired_on: + date: 2026-08-03 + infera_commit: "1380228" + subject: "fix(image): carry the DSA indexer row fix onto the gfx942 v0.5.16 base" + retired_reason: superseded-by-local-patch + upstream_fix: null + integrated_by: null + problem: + what: >- + The DSA indexer's aiter (HIP) paged-MQA branch sizes its logits from the padded row count + while `lengths` is sized from the real one, so fast_topk_v2 asserts under DP-attention. + why: >- + Same root cause as its replacement — see + sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.upstream.status.yaml for the full account. + how: >- + A `patch -p1 --fuzz=0` context diff cut against sglang v0.5.15.post1. + before_fix: >- + "RuntimeError: Expected lengths.size(0) == B to be true, but got false" on gfx950. + after_fix: Fixed on gfx950 — but only on the base the diff was cut against. + context: >- + gfx950 / MI355X, sglang v0.5.15.post1, GLM-5.2 DSA with DP-attention and EAGLE MTP. + call_chain: + - "dsa_indexer.py :: forward (aiter/HIP branch) -> fast_topk_v2 asserts lengths.size(0) == B" + symptom_signature: "Expected lengths.size(0) == B to be true, but got false" + resolution: >- + Not obsolete — REPLACED, by a Python anchor script doing the same repair. A --fuzz=0 diff + only applies to the exact base it was cut against, and the fix was needed on the gfx942 + v0.5.16 base too. The script form also made room for the second direction of the bug (the + DP-idle rank under MTP draft-extend, where there are FEWER query rows than lengths entries), + which was added the same day as GLM52_P1V3 and which a context diff would have made awkward. + reusable_on: sglang v0.5.15.post1 only, by construction. + lesson: >- + Prefer an anchor script to a context diff for anything that has to span two engine bases. + The three sglang_dsa patches now split exactly along that line: the one that became a script + runs on both images, and the two that stayed diffs run on one. diff --git a/deploy/docker/patches/vllm-dsv4/legacy/README.md b/deploy/docker/patches/archived/vllm-dsv4/README.md similarity index 79% rename from deploy/docker/patches/vllm-dsv4/legacy/README.md rename to deploy/docker/patches/archived/vllm-dsv4/README.md index e2ceb32e..c0ea06bf 100644 --- a/deploy/docker/patches/vllm-dsv4/legacy/README.md +++ b/deploy/docker/patches/archived/vllm-dsv4/README.md @@ -1,4 +1,8 @@ -# vllm-dsv4/legacy — deprecated / upstreamed DSv4 patches (not baked into any image) +# archived/vllm-dsv4 — deprecated / upstreamed DSv4 patches (not baked into any image) + +> Moved here from `patches/vllm-dsv4/legacy/` on 2026-08-05. The authoritative record for +> these is now [`../patch.archived.yaml`](../patch.archived.yaml); this file is kept as the +> original provenance note. Nothing under `archived/` is copied into any image. ## Purpose @@ -30,4 +34,5 @@ archival and provenance. - `patch_dsv4_mhc_aiter`'s functionality is upstream via a **different implementation** (aiter as the MHC default rather than an env gate) — "functionally upstream, different form" — so the monkey-patch anchor mismatches on dev748 and auto-skips. -- Active DSv4 patches live in the parent directory `../` (the moriio_dsv4 trio). +- Active DSv4 patches live in `deploy/docker/patches/vllm-dsv4/` (the moriio_dsv4 trio), + each with its own `.upstream.status.yaml` record. diff --git a/deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_aiter_moe.py b/deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_aiter_moe.py similarity index 100% rename from deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_aiter_moe.py rename to deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_aiter_moe.py diff --git a/deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_mhc_aiter.diff b/deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.diff similarity index 100% rename from deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_mhc_aiter.diff rename to deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.diff diff --git a/deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_mhc_aiter.py b/deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.py similarity index 100% rename from deploy/docker/patches/vllm-dsv4/legacy/patch_dsv4_mhc_aiter.py rename to deploy/docker/patches/archived/vllm-dsv4/patch_dsv4_mhc_aiter.py diff --git a/deploy/docker/patches/atom/patch_gdn_pd_state_transfer.upstream.status.yaml b/deploy/docker/patches/atom/patch_gdn_pd_state_transfer.upstream.status.yaml new file mode 100644 index 00000000..84f5cf5e --- /dev/null +++ b/deploy/docker/patches/atom/patch_gdn_pd_state_transfer.upstream.status.yaml @@ -0,0 +1,146 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - local-repro + - patch-header-only + +patch: + path: deploy/docker/patches/atom/patch_gdn_pd_state_transfer.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.atom + opt_in_flag: null + +target: + library: atom + component: attention-metadata + repo: internal + files: + - atom/attention/backends/gdn_attn.py + versions: + - surface: Dockerfile.atom + ref: atom0.1.4 + commit: null + pinned_ref_on_main: null + release_branch: null + unpinned_risk: >- + ATOM is an internal engine consumed only as a base image, so there is no public commit + to record and no way to express "on main". The image tag is the entire pin. + images: + - tag: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612 + digest: null + digest_source: unresolved + note: >- + ROCm 7.2.4, Ubuntu 24.04, Python 3.12, PyTorch 2.10.0, ATOM 0.1.4, built 2026-06-12. + Dockerfile.atom pins the tag only. + +upstream_main_affected: + value: null + evidence: >- + No public upstream exists. ATOM is internal, so there is no repository to check and this + field is null by construction rather than by omission. + +applies_to: + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: true + evidence: >- + Without it, hybrid GDN models cannot recall prompt context in PD at all: memory probes + measured single-node MIXED 5/5 recall against 2-node PD 0/5. + +alive_because: + reason: internal-engine-no-upstream + detail: >- + ATOM has no public upstream to file against, so this cannot be upstreamed in the sense the + other records use. The equivalent action is landing it in ATOM itself; until then the image + patch is the only delivery mechanism. + consumers: + - Dockerfile.atom, for hybrid GDN models (e.g. Qwen3.5-397B-A17B) in PD + drop_when: >- + ATOM's GDNAttentionMetadataBuilder publishes the mamba per-request state itself, at which + point the anchor is gone and the hunk is skipped with a warning. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + Hybrid GDN models produce garbage in PD disaggregation — they cannot recall anything from + the prompt — because the GatedDeltaNet recurrent state is never transferred from the + prefill worker to the decode worker. + why: >- + GDNAttentionMetadataBuilder inherits get_kv_transfer_tensors from + AiterAttentionMetadataBuilder, which registers only the full-attention paged KV cache as + block_regions and returns slot_regions=[]. The linear-attention layers' recurrent state — + conv_state plus temporal/ssm state, held per request in mamba_k_cache / mamba_v_cache — is + therefore not in the transfer set at all, so the decode worker decodes from a ZERO mamba + state. + how: >- + Override get_kv_transfer_tensors on GDNAttentionMetadataBuilder to also publish the mamba + per-request state as direct slot_regions, one region per GDN layer for K and V. The Mooncake + connector already supports exactly this path — register_kv_caches sets + _has_slot_regions=True and _execute_block_slot_transfer Phase 2a does a direct per-slot RDMA + write (src_base + src_slot*unit_bytes -> dst_base + dst_slot*unit_bytes) with no + staging, gather or scatter, because one request's state is contiguous within each layer + buffer of shape [n_gdn, num_slots, *state_shape]. unit_bytes is sized per CACHE GROUP + (= slots_per_req contiguous tensor slots) because the local_slot_index carried in + kv_transfer_params is the per-request cache-group index; with speculative decoding off, + slots_per_req == 1 and a group is a single slot. + before_fix: >- + Anything depending on prompt context comes out as token-salad or off-topic. Memory probes: + single-node MIXED 5/5 recall, 2-node PD 0/5. + after_fix: PD recall matches single-node mixed serving. + context: >- + Hybrid GDN models such as Qwen3.5-397B-A17B on ATOM in 2-node PD over Mooncake. What makes + this dangerous is the masking: world-knowledge facts stored in the WEIGHTS (Paris, Tokyo, + 2+2) still look correct, so a smoke test passes while in-context recall is entirely broken. + call_chain: + - "GDNAttentionMetadataBuilder inherits get_kv_transfer_tensors from AiterAttentionMetadataBuilder" + - "returns block_regions = full-attention paged KV only, slot_regions = []" + - "mamba_k_cache / mamba_v_cache recurrent state is never transferred" + - "decode worker decodes from a zero mamba state" + symptom_signature: >- + in-context recall fails while weight-stored world knowledge answers correctly — 5/5 mixed + versus 0/5 PD + silent: true + +verification: + date: 2026-07-20 + hardware: 2-node AMD ROCm PD + software: ATOM 0.1.4 base image + workload: Qwen3.5-397B-A17B hybrid GDN, memory-probe recall test + result: 0/5 PD recall becomes parity with the 5/5 single-node mixed baseline. + notes: >- + The probe design is the transferable part: testing only world knowledge would have passed. + Any PD correctness check for a hybrid model needs an in-context recall probe. + +scope_limits: >- + Assumes slots_per_req == 1 in the common case (speculative decoding off). With speculative + decoding on, a cache group spans several slots and the unit_bytes sizing is what makes that + correct — worth re-reading before enabling both together. + +open_actions: + - action: >- + Land the equivalent inside ATOM so the image patch can be dropped. There is no public + upstream, so this is the only path to retiring it. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.upstream.status.yaml b/deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.upstream.status.yaml new file mode 100644 index 00000000..b11ba5e4 --- /dev/null +++ b/deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.upstream.status.yaml @@ -0,0 +1,140 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - local-repro + - patch-header-only + +patch: + path: deploy/docker/patches/atom/patch_minimax_m2_qknorm_rope.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.atom + opt_in_flag: null + +target: + library: atom + component: model-definition + repo: internal + files: + - atom/models/minimax_m2.py + versions: + - surface: Dockerfile.atom + ref: atom0.1.4 + commit: null + pinned_ref_on_main: null + release_branch: null + unpinned_risk: >- + Internal engine consumed as a base image; the tag is the entire pin. + images: + - tag: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612 + digest: null + digest_source: unresolved + note: ATOM 0.1.4, built 2026-06-12. Tag-pinned only. + +upstream_main_affected: + value: null + evidence: No public upstream exists — ATOM is internal. + +applies_to: + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: true + evidence: >- + The stock model file does not load at all on this base — get_rope() raises a TypeError on + import. Required to START MiniMax-M2; see scope_limits for what it does not give you. + +alive_because: + reason: internal-engine-no-upstream + detail: >- + No public upstream. This is a base-image inconsistency rather than a design defect: the + shipped minimax_m2.py calls a get_rope API the shipped rope module does not have, so the + two halves of the same image disagree. + consumers: + - Dockerfile.atom, for MiniMax-M2 + drop_when: >- + An ATOM base image ships a minimax_m2.py consistent with its own rope API, and a QK-norm + kernel that does not need the batch guard. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: >- + Hunk B caps the fused QK-norm kernel at qkv.shape[0] <= 256 and falls back to a manual fp32 + path above that. If a future ATOM raises the kernel's real limit, the cap silently keeps the + slower fallback for large batches — correct output, quietly worse throughput, and nothing + fails. The 256 threshold is the number to re-check on a base bump. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + The stock atom/models/minimax_m2.py in the base image fails to load, and its TP fused + QK-norm all-reduce kernel is unsafe for large batches. + why: >- + Two independent problems. (A) The model file passes dtype= to get_rope(), but the installed + rope API does not accept it, so import raises a TypeError — the model file and the rope + module in the same image are out of sync. (B) The TP fused QK-norm all-reduce kernel does not + hold for large batches, so it needs a size guard rather than being taken unconditionally. + how: >- + Two minimal, independently guarded hunks: (A) drop the dtype= kwarg from the get_rope call; + (B) guard the fused kernel with qkv.shape[0] <= 256 so larger batches fall back to the manual + fp32 QK-norm path. Each hunk is applied only if its exact, unique pre-image is present, and + a missing or non-unique pre-image skips that hunk with a loud warning instead of editing code + the patch does not recognise. + before_fix: TypeError from get_rope() at model load — MiniMax-M2 cannot start. + after_fix: >- + MiniMax-M2 loads and runs. Correct OUTPUT additionally requires the triton/unified attention + backend, which is selected at launch (ATOM_USE_UNIFIED_ATTN=1 plus --block-size 64 in + models/minimax_m27.conf) and is not this patch's job. + context: >- + MiniMax-M2 on the rocm/atom 0.1.4 base image. Nothing model-agnostic here — both hunks are + specific to minimax_m2.py. + call_chain: + - "atom/models/minimax_m2.py imported at model load" + - "get_rope(..., dtype=getattr(quant_config, 'torch_dtype', None)) -> TypeError" + - "(separately) TP fused QK-norm all-reduce kernel taken unconditionally for any batch size" + symptom_signature: "TypeError from get_rope() during MiniMax-M2 model load" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm + software: ATOM 0.1.4 base image + workload: MiniMax-M2 model load and serve + result: >- + Model loads and runs. Coherent output needs the unified attention backend selected at launch, + which is configured separately. + notes: >- + Worth keeping the two concerns distinct: this patch makes the model START; the launch config + makes it CORRECT. Conflating them has previously led to blaming the patch for output quality. + +scope_limits: >- + Makes MiniMax-M2 start. It does not make its output correct — that needs + ATOM_USE_UNIFIED_ATTN=1 and --block-size 64 from models/minimax_m27.conf. + +open_actions: + - action: >- + Get a base image whose minimax_m2.py matches its own rope API. Hunk A is fixing an + inconsistency inside the vendor image, which is the vendor's to resolve. + owner: unassigned + blocked_on: null + - action: >- + Establish the fused QK-norm kernel's real batch limit so the 256 guard is a measured + threshold rather than a safe-looking round number. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/atom/patch_mooncake_consumer_slot.upstream.status.yaml b/deploy/docker/patches/atom/patch_mooncake_consumer_slot.upstream.status.yaml new file mode 100644 index 00000000..ac445fb4 --- /dev/null +++ b/deploy/docker/patches/atom/patch_mooncake_consumer_slot.upstream.status.yaml @@ -0,0 +1,123 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - local-repro + - patch-header-only + +patch: + path: deploy/docker/patches/atom/patch_mooncake_consumer_slot.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.atom + opt_in_flag: null + +target: + library: atom + component: mooncake-connector + repo: internal + files: + - atom/kv_transfer/disaggregation/mooncake/mooncake_connector.py + versions: + - surface: Dockerfile.atom + ref: atom0.1.4 + commit: null + pinned_ref_on_main: null + release_branch: null + unpinned_risk: >- + Internal engine consumed as a base image; the tag is the entire pin. + images: + - tag: rocm/atom:rocm7.2.4_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom0.1.4_20260612 + digest: null + digest_source: unresolved + note: ATOM 0.1.4, built 2026-06-12. Tag-pinned only. + +upstream_main_affected: + value: null + evidence: No public upstream exists — ATOM is internal. + +applies_to: + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: true + evidence: >- + Without it the decode worker crashes on the FIRST PD request for any model with + cache-group slot state but no registered slot_regions. + +alive_because: + reason: internal-engine-no-upstream + detail: >- + No public upstream. Note the coupling worth knowing: this defect is reached by exactly the + model class that the GDN state-transfer patch also serves — Qwen3.5's GDN layout, where + local_slot_index >= 0 while _has_slot_regions is False. + consumers: + - Dockerfile.atom, for hybrid models with slot state but no slot_regions (e.g. Qwen3.5 GDN) + drop_when: ATOM initialises consumer_staging_pool_idx unconditionally. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + In the Mooncake PD KV-transfer consumer, start_load_kv reads consumer_staging_pool_idx on a + path where it was never bound, so the decode worker dies with an UnboundLocalError on the + first PD request. + why: >- + The variable is bound only inside the `if self._has_slot_regions:` branch but referenced + unconditionally when recording a pending recv slot. Hybrid models that use cache-group slot + state but do NOT register slot_regions — Qwen3.5's GDN linear-attention layout, where + local_slot_index >= 0 while _has_slot_regions is False — take the else path and hit the + unbound read. + how: >- + Initialise consumer_staging_pool_idx = -1 at the top of the per-request loop. The WRITE_DONE + release path is already guarded by `if pool_idx >= 0:`, so -1 is safe and simply skips the + staging scatter and release for the block-only path. + before_fix: >- + "UnboundLocalError: cannot access local variable 'consumer_staging_pool_idx'" and the decode + worker crashes on the first PD request. + after_fix: The decode worker survives and the block-only path skips staging cleanly. + context: >- + ATOM Mooncake PD with a hybrid model that has cache-group slot state but no registered + slot_regions. Models that do register slot_regions never take the failing path. + call_chain: + - "atom/kv_transfer/disaggregation/mooncake/mooncake_connector.py :: start_load_kv" + - "consumer_staging_pool_idx bound only inside `if self._has_slot_regions:`" + - "else path records a pending recv slot and reads it anyway" + - "UnboundLocalError kills the decode worker on the first PD request" + symptom_signature: "UnboundLocalError: cannot access local variable 'consumer_staging_pool_idx'" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm PD + software: ATOM 0.1.4 base image + workload: Qwen3.5 GDN PD, first request + result: Crash cleared. + notes: >- + The -1 sentinel is safe only because the release path already tests pool_idx >= 0 — that + pre-existing guard is what makes a one-line fix correct here. + +scope_limits: null + +open_actions: + - action: Land the equivalent inside ATOM so the image patch can be dropped. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/hipfile_async/hipfile_async.upstream.status.yaml b/deploy/docker/patches/hipfile_async/hipfile_async.upstream.status.yaml new file mode 100644 index 00000000..42b0fbc4 --- /dev/null +++ b/deploy/docker/patches/hipfile_async/hipfile_async.upstream.status.yaml @@ -0,0 +1,180 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/hipfile_async/hipfile_async.patch + extra_files: + - deploy/docker/patches/hipfile_async/file_async_fragment.py + - deploy/docker/patches/hipfile_async/patch_hipfile_async.sh + kind: git-diff + idempotent: true + applied_by: + - Dockerfile.vllm + opt_in_flag: BUILD_HIPFILE=1 (build ARG; default 1) + +target: + library: hipfile + component: python-bindings + repo: ROCm/rocm-systems + files: + - projects/hipfile/python/hipfile/__init__.py + - projects/hipfile/python/hipfile/_chipfile.pxd + - projects/hipfile/python/hipfile/_hipfile.pyx + - projects/hipfile/python/hipfile/file.py + versions: + - surface: Dockerfile.vllm (build_hipfile.sh) + ref: unpinned-default-branch + commit: null + pinned_ref_on_main: true + release_branch: null + unpinned_risk: >- + This is the important fact about this patch. build_hipfile.sh defaults + HIPFILE_GIT_REF to empty, i.e. HEAD of the ROCm/rocm-systems default branch, so the + target moves on every build with no record of what was built. Since #7386 merged on + 2026-07-16, any clone taken after that date already carries the async API — which + means this patch is expected to be a no-op on any fresh build today, and the image + content depends on the build DATE rather than on anything in this repo. + images: [] + +upstream_main_affected: + value: false + evidence: >- + ROCm/rocm-systems#7386 is MERGED (2026-07-16) and lands the same feature on the same + four files this patch edits, with the same root-cause account of the pointer-lifetime + bug. The default branch therefore carries the API. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (BUILD_HIPFILE=1, step 4) + effect: unverified + required_to_run: false + evidence: >- + Effect depends on the build date, not on the repo: the script greps the cloned tree + for the hipFileStreamRegister marker and skips if present. Any build after 2026-07-16 + clones a tree that has #7386, so this is very likely a no-op now — but nothing in the + repo records which it was for a given image, so it cannot be asserted either way. + +alive_because: + reason: released-but-infera-not-bumped + detail: >- + Ours, merged upstream on 2026-07-16, and the target is unpinned — so upstream's fix + arrives on its own the next time anyone builds. This is not really "alive" so much as + "not yet deleted". Note the two implementations differ in HOW they keep the driver's + out-parameters alive, so this is merged-but-not-equivalent: upstream owns the slots in a + Cython AsyncIOHandle cdef class, while this patch sidesteps Cython and calls + libhipfile.so through ctypes with heap-allocated c_size_t / c_int64 / c_ssize_t slots. + Upstream's is the better shape. + consumers: + - >- + none, most likely — only a build pinned to a rocm-systems tree older than #7386 would + still need it + drop_when: >- + Now, in practice: the base image's /opt/rocm-systems-src/projects/hipfile carries #7386 + on any fresh clone. Confirm with one grep for hipFileStreamRegister and delete. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + The marker grep makes the skip safe, so there is no misapply risk. The real risk is the + opposite and worth stating plainly: because HIPFILE_GIT_REF is unpinned, two images built + from the same infera commit on either side of 2026-07-16 contain DIFFERENT hipfile code — + one with our ctypes implementation, one with upstream's Cython one. That is an + unreproducible build, independent of whether the patch is kept. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged. The upstream PR was ours and merged four days before this repo's v0.1.0 + commit landed the local copy, and nobody revisited it afterwards. + +upstream_issues: null + +upstream_prs: + - ref: ROCm/rocm-systems#7386 + url: https://github.com/ROCm/rocm-systems/pull/7386 + title: "hipfile: feat(hipfile/python): add async stream I/O bindings" + state: MERGED + review_decision: null + merged_at: 2026-07-16 + author: jiejingzhangamd + ours: true + same_approach: false + approach_note: >- + Same feature, same four files, same root-cause account — but a different mechanism for + keeping the driver's out-parameters alive. Upstream owns the slots in a Cython + AsyncIOHandle cdef class; the local patch avoids Cython entirely and calls libhipfile.so + through ctypes with heap-allocated slots. Ours was the fast local implementation; the + upstream one is the general fix and is better. Do not carry the local one forward. + requested_action: null + in_pinned_base: null + +related_refs: null + +problem: + what: >- + The ROCm hipFile Python binding exposes no async API, and the obvious Cython + implementation of one is unsafe: out-parameters passed as `&local` die before the driver + dereferences them. + why: >- + Cython stack locals passed by address to an asynchronous driver call go out of scope when + the Python frame returns, while the driver writes to them later. The write then lands on + reclaimed stack, so bytes_done reads 0 and HipFileException 5022 appears + intermittently — a lifetime bug, so it is timing-dependent and looks flaky. + how: >- + Add write_async / read_async, Stream and supports_async() to the binding, and keep the + driver's out-parameter slots on the HEAP: the wrapper calls libhipfile.so via ctypes with + heap-allocated c_size_t / c_int64 / c_ssize_t slots that outlive the call. + before_fix: >- + No async API at all; a naive Cython binding gives bytes_done == 0 and intermittent + HipFileException 5022. + after_fix: >- + Async stream I/O works and bytes_done is correct, which is what kvd's gpu-direct L3 load + path needs. + context: >- + ROCm hipFile Python bindings, used by kvd's gpu-direct L3. Built into Dockerfile.vllm + from a clone of ROCm/rocm-systems, so the effective target is whatever that default + branch holds at build time. + call_chain: + - "hipfile/file.py :: write_async / read_async" + - "_hipfile.pyx passes &local out-parameters to the async driver call" + - "the Python frame returns; the driver writes to reclaimed stack" + - "bytes_done reads 0 / HipFileException 5022" + symptom_signature: "HipFileException 5022 with bytes_done == 0, intermittently" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm with hipFile / AIS + software: ROCm hipfile python bindings from ROCm/rocm-systems + workload: kvd gpu-direct L3 async read/write + result: Async I/O completes with correct bytes_done; the intermittent 5022 is gone. + notes: >- + Also note the neighbouring trap in the same build step: a missing ais-check probe + SILENTLY downgrades the kvd load path to CPU-bounce, which is why Dockerfile.vllm asserts + `test -x /opt/rocm/bin/ais-check`. + +scope_limits: >- + A local implementation, superseded upstream by a better one. It should not be carried + forward on its own merits. + +open_actions: + - action: >- + Delete this patch. It is ours, merged upstream on 2026-07-16, the target is unpinned so + the fix arrives by itself, and upstream's implementation is the better of the two. + owner: unassigned + blocked_on: null + - action: >- + Separately, pin HIPFILE_GIT_REF. An unpinned default-branch clone makes image contents a + function of the build date; that is a reproducibility problem regardless of this patch. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.upstream.status.yaml b/deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.upstream.status.yaml new file mode 100644 index 00000000..cccbd185 --- /dev/null +++ b/deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.upstream.status.yaml @@ -0,0 +1,177 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - git-compare + - local-repro + +patch: + path: deploy/docker/patches/mooncake_cpp/rdma_auto_chunk_mr_2017.diff + kind: git-diff + idempotent: true + applied_by: + - deploy/docker/patches/mooncake_cpp/apply_mooncake_cpp_patches.sh + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + - Dockerfile.vllm + - Dockerfile.atom + - deploy/overlay/Dockerfile.payload + opt_in_flag: null + +target: + library: mooncake + component: transfer-engine-rdma + repo: kvcache-ai/Mooncake + files: + - mooncake-transfer-engine/include/transport/rdma_transport/rdma_transport.h + - mooncake-transfer-engine/src/transport/rdma_transport/rdma_context.cpp + - mooncake-transfer-engine/src/transport/rdma_transport/rdma_transport.cpp + versions: + - surface: all images (MOONCAKE_GIT_REF) + ref: main @ 747003c + commit: 747003c058015c4077a266e7ccd7549bbc9baede + pinned_ref_on_main: true + release_branch: null + images: [] + +upstream_main_affected: + value: false + evidence: >- + Mooncake#2644 is MERGED (2026-07-28) and the pin is an ancestor of main, so main + carries the fix. Established with `git compare` between #2644's merge commit and + 747003c: the pin is BEHIND the merge, so main has it and we do not. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang / Dockerfile.sglang.gfx942 (Mooncake rebuilt in place) + effect: op + required_to_run: false + evidence: >- + Load-bearing only above the device max_mr_size — on ionic that is ~2 GiB, which a + real KV pool exceeds. `git apply --reverse --check` in the apply script makes the + already-applied case an idempotent skip. + - engine: vllm + surface: Dockerfile.vllm (build_mooncake_rocm.sh, MOONCAKE_DMABUF=1) + effect: op + required_to_run: false + evidence: Same diff, same pin, same reasoning. + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: false + evidence: Same diff, same pin. + +alive_because: + reason: merged-not-in-pinned-base + detail: >- + Ours, and merged upstream on 2026-07-28 — but MOONCAKE_GIT_REF is still 747003c + (2026-06-26), which is behind that merge. So the local diff is still doing the work. + consumers: + - all four engine images plus the overlay payload, on any fabric whose max_mr_size is + smaller than the KV pool + drop_when: "MOONCAKE_GIT_REF advances past #2644's merge commit." + drop_signal: pin-advance-only + silent_misapply_risk: >- + Low but not zero: `git apply --reverse --check` detects the already-applied case and + skips, so a pin advance past #2644 produces a clean skip rather than a failure. That + is the desired behaviour, but it means nothing announces that the diff has become + dead weight — only the pin tells you. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged since it was written — the upstream review of #2644 did not require a + change to the local diff. + +upstream_issues: + - ref: kvcache-ai/Mooncake#2017 + url: https://github.com/kvcache-ai/Mooncake/issues/2017 + kind: issue + title: Buffers larger than device max_mr_size are silently truncated + state: CLOSED + state_reason: "fixed by #2644" + author: null + note: >- + Our own report, and the number the diff and its chunk_map_ comments cite + throughout. + +upstream_prs: + - ref: kvcache-ai/Mooncake#2644 + url: https://github.com/kvcache-ai/Mooncake/pull/2644 + title: "fix(rdma): auto-chunk MRs larger than device max_mr_size (#2017)" + state: MERGED + review_decision: null + merged_at: 2026-07-28 + author: jiejingzhangamd + ours: true + same_approach: true + approach_note: >- + Same change, upstreamed by us and merged. The local diff is the same content + carried against the older pin. + requested_action: null + in_pinned_base: false + +related_refs: null + +problem: + what: >- + Buffers larger than the device max_mr_size are silently truncated by ibv_reg_mr while + BufferDesc.length keeps advertising the full size, so RDMA operations whose target + lands past the boundary fail. + why: >- + registerMemoryRegionInternal saw `length > max_mr_size`, logged a warning and simply + shortened `length`. The registration then succeeded for a shorter region than the + descriptor advertises, so a remote peer computes a valid-looking address inside a + region that was never registered. + how: >- + Chunk at registration time: registerLocalMemory splits buffers into <= max_mr_size + MRs, one BufferDesc per chunk. Because unregisterLocalMemory() only receives the base + address, the transport remembers each base buffer's chunk start addresses in a + chunk_map_ guarded by chunk_map_mutex_ so cleanup can find them. The truncation site + itself becomes a hard error rather than a warning — after chunking, no larger buffer + should ever reach it, so failing loudly is correct. + before_fix: >- + IBV_WC_REM_ACCESS_ERR on RDMA operations whose target lands past the truncation + boundary, with only a PLOG(WARNING) at registration to hint at why. + after_fix: >- + Large KV pools register as multiple MRs and cross-node transfers complete. A buffer + that somehow still exceeds max_mr_size now returns ERR_INVALID_ARGUMENT instead of + registering short. + context: >- + AMD MI355X + ionic RoCE, where max_mr_size is about 2 GiB — comfortably smaller than + a production KV pool. Any fabric with a max_mr_size below the pool size hits this; + it is not ROCm-specific, which is why upstream took it. + call_chain: + - "RdmaTransport::registerLocalMemory(addr, length, ...)" + - "RdmaContext::registerMemoryRegionInternal — compared length > globalConfig().max_mr_size and shrank length" + - "ibv_reg_mr registers a shorter region than BufferDesc.length advertises" + - "remote RDMA op targeting past the boundary -> IBV_WC_REM_ACCESS_ERR" + symptom_signature: IBV_WC_REM_ACCESS_ERR past the max_mr_size boundary + silent: false + +verification: + date: 2026-07-20 + hardware: AMD MI355X, ionic RoCE (max_mr_size ~2 GiB) + software: Mooncake main @ 747003c + workload: cross-node PD KV transfer with a KV pool larger than max_mr_size + result: "REM_ACCESS_ERR cleared; upstream accepted the same change as #2644." + notes: null + +scope_limits: null + +open_actions: + - action: >- + Advance MOONCAKE_GIT_REF past #2644 and drop this diff. It is the cleanest + retirement available — our own merged fix, only the pin holding it. + owner: unassigned + blocked_on: >- + A pin advance also pulls #2682/#2725, which changes the hip-transport picture; do + the two together, not separately. diff --git a/deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.upstream.status.yaml b/deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.upstream.status.yaml new file mode 100644 index 00000000..9707aae4 --- /dev/null +++ b/deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.upstream.status.yaml @@ -0,0 +1,159 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/mooncake_cpp/rdma_transport_dmabuf_cmake.diff + kind: git-diff + idempotent: true + applied_by: + - deploy/docker/patches/mooncake_cpp/apply_mooncake_cpp_patches.sh + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + - Dockerfile.vllm + - Dockerfile.atom + - deploy/overlay/Dockerfile.payload + opt_in_flag: MOONCAKE_HIP_DMABUF=1 (build ARG; default 0) + +target: + library: mooncake + component: transfer-engine-rdma + repo: kvcache-ai/Mooncake + files: + - mooncake-transfer-engine/CMakeLists.txt + - mooncake-transfer-engine/src/transport/rdma_transport/rdma_context.cpp + versions: + - surface: all images (MOONCAKE_GIT_REF) + ref: main @ 747003c + commit: 747003c058015c4077a266e7ccd7549bbc9baede + pinned_ref_on_main: true + release_branch: null + images: [] + +upstream_main_affected: + value: false + evidence: >- + Mooncake#2543 is MERGED (2026-07-24) and 747003c predates it, so main carries + dma-buf GPUDirect registration and our pin does not. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang / Dockerfile.sglang.gfx942 + effect: op + required_to_run: false + evidence: >- + Only when MOONCAKE_HIP_DMABUF=1. Default is 0, in which case GPU memory is + registered with bare ibv_reg_mr via host-libionic injection and this diff is not + needed. It IS required on a dma-buf-only RoCE fabric with no ib_peer_mem, e.g. the + crusoe amd-spur cluster, where bare ibv_reg_mr cannot pin VRAM at all. + - engine: vllm + surface: Dockerfile.vllm (MOONCAKE_HIP_DMABUF ARG, default 0) + effect: op + required_to_run: false + evidence: Same ARG, same condition. + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: false + evidence: Same ARG, same condition. + +alive_because: + reason: merged-not-in-pinned-base + detail: >- + Upstream merged the equivalent as #2543 on 2026-07-24; MOONCAKE_GIT_REF is still + 747003c (2026-06-26), which is behind it. Purely a pin lag. + consumers: + - any image built with MOONCAKE_HIP_DMABUF=1 (dma-buf-only fabrics such as crusoe amd-spur) + drop_when: "MOONCAKE_GIT_REF advances past #2543's merge commit." + drop_signal: pin-advance-only + silent_misapply_risk: >- + The apply script's `git apply --reverse --check` makes an already-fixed pin a clean + skip, so advancing the pin will not fail the build — nothing will announce that this + diff has become redundant. Track it by the pin, not by the build log. + +history: + born: + date: 2026-07-29 + commit: a322e14 + subject: add pd utest on crusoe cluster + last_modified: + date: 2026-07-29 + commit: a322e14 + subject: add pd utest on crusoe cluster + reason: >- + Added for the crusoe amd-spur bring-up, where the fabric offers no ib_peer_mem and + dma-buf is the only way to register VRAM. Unchanged since. + +upstream_issues: null + +upstream_prs: + - ref: kvcache-ai/Mooncake#2543 + url: https://github.com/kvcache-ai/Mooncake/pull/2543 + title: "feat(rdma): support dma-buf GPUDirect memory registration" + state: MERGED + review_decision: null + merged_at: 2026-07-24 + author: null + ours: false + same_approach: true + approach_note: >- + Same mechanism — detect ibv_reg_dmabuf_mr at configure time and use it for GPU + buffers. Ours is the same content carried against the older pin; nothing here is + local-only. + requested_action: null + in_pinned_base: false + +related_refs: null + +problem: + what: >- + Mooncake at our pin has no dma-buf GPUDirect registration path, so on a fabric that + exposes only dma-buf it cannot register VRAM for RDMA at all. + why: >- + Registration goes through bare ibv_reg_mr, which needs the legacy ib_peer_mem kernel + module to pin device memory. On a dma-buf-only RoCE fabric that module is absent, so + there is no mechanism by which the VRAM can be pinned. + how: >- + Detect ibv_reg_dmabuf_mr availability in CMakeLists.txt and, when + MOONCAKE_HIP_DMABUF=1, register GPU buffers through it (dma-buf GPUDirect) instead of + bare ibv_reg_mr. + before_fix: >- + On crusoe amd-spur, GPU memory registration fails outright — RDMA cannot be set up, + so cross-node PD never starts. + after_fix: >- + GPU buffers register via ibv_reg_dmabuf_mr and cross-node PD works on the dma-buf-only + fabric. PD unit tests pass there. + context: >- + Fabric-dependent, not model-dependent. Required on crusoe amd-spur (dma-buf only, no + ib_peer_mem). On ionic the default path (bare ibv_reg_mr + host-libionic injection) + works, but must not be used at high utilisation — see HIP-209. + call_chain: + - "mooncake-transfer-engine/CMakeLists.txt — probes for ibv_reg_dmabuf_mr" + - "RdmaContext::registerMemoryRegionInternal — bare ibv_reg_mr for GPU buffers" + - "no ib_peer_mem -> VRAM cannot be pinned -> registration fails" + symptom_signature: GPU memory registration failure on a fabric without ib_peer_mem + silent: false + +verification: + date: 2026-07-29 + hardware: crusoe amd-spur cluster (dma-buf-only RoCE, no ib_peer_mem) + software: Mooncake main @ 747003c with MOONCAKE_HIP_DMABUF=1 + workload: cross-node PD unit tests + result: Registration succeeds and the PD tests pass on that fabric. + notes: null + +scope_limits: >- + Opt-in only. With the default MOONCAKE_HIP_DMABUF=0 this changes nothing, so it is not + a fix for the ionic default path. + +open_actions: + - action: >- + Advance MOONCAKE_GIT_REF past #2543 and drop this diff — same pin bump that retires + rdma_auto_chunk_mr_2017.diff. + owner: unassigned + blocked_on: >- + The same pin advance also pulls #2682/#2725; sequence it with the hip-transport gate. diff --git a/deploy/docker/patches/mooncake_cpp/transfer_engine_impl.upstream.status.yaml b/deploy/docker/patches/mooncake_cpp/transfer_engine_impl.upstream.status.yaml new file mode 100644 index 00000000..bad5aa4c --- /dev/null +++ b/deploy/docker/patches/mooncake_cpp/transfer_engine_impl.upstream.status.yaml @@ -0,0 +1,183 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/mooncake_cpp/transfer_engine_impl.diff + kind: git-diff + idempotent: true + applied_by: + - deploy/docker/patches/mooncake_cpp/apply_mooncake_cpp_patches.sh + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + - Dockerfile.vllm + - Dockerfile.atom + - deploy/overlay/Dockerfile.payload + opt_in_flag: MC_DISABLE_HIP_TRANSPORT / MC_ENABLE_HIP_TRANSPORT (runtime env; the gate itself is always compiled in) + +target: + library: mooncake + component: transfer-engine + repo: kvcache-ai/Mooncake + files: + - mooncake-transfer-engine/src/transfer_engine_impl.cpp + versions: + - surface: all images (MOONCAKE_GIT_REF) + ref: main @ 747003c + commit: 747003c058015c4077a266e7ccd7549bbc9baede + pinned_ref_on_main: true + release_branch: null + images: [] + +upstream_main_affected: + value: true + evidence: >- + Mooncake#2725, which would add the env gate upstream, is still OPEN with no review + activity, so main still installs the HIP transport unconditionally. #2682, the PR + that introduced the unconditional install, is merged and in our pin. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang / Dockerfile.sglang.gfx942 + effect: op + required_to_run: true + evidence: >- + Cross-node PD does not work without it. Both sglang Dockerfiles additionally + ASSERT the gate is present in the .so they will load (REQUIRE_MOONCAKE_HIP_GATE=1 + greps the binary for MC_ENABLE_HIP_TRANSPORT / MC_DISABLE_HIP_TRANSPORT), because + a build that silently skipped the rebuild ships stock #2682 and dies at run time. + - engine: vllm + surface: Dockerfile.vllm + effect: op + required_to_run: true + evidence: Same assertion layer (step 3b of Dockerfile.vllm), same reasoning. + - engine: atom + surface: Dockerfile.atom + effect: op + required_to_run: true + evidence: Same diff and pin. + +alive_because: + reason: upstream-pr-open + detail: >- + Ours as #2725, open and unreviewed. Until it merges AND the pin advances past it, + every ROCm image needs the local gate — and because the failure is a hard runtime + error on cross-node PD, the images assert the gate is compiled in rather than trust + the build. + consumers: + - every ROCm engine image doing cross-node PD over RDMA + drop_when: >- + #2725 merges and MOONCAKE_GIT_REF advances past it. + drop_signal: pin-advance-only + silent_misapply_risk: >- + Inverted here, and worth being clear about: the danger is not the patch outliving its + reason but the patch silently NOT being applied. That is what the + REQUIRE_MOONCAKE_HIP_GATE assertion exists for, plus + deploy/docker/scripts/verify_image_mooncake.sh for images built FROM an already + published image where the build stage never re-runs. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-08-01 + commit: b7df12a + subject: "fix(mooncake): disable the HIP (hipIpc) transport on ROCm" + reason: >- + Turned into a real env-var gate. Before this the diff addressed the same defect + differently; b7df12a made it MC_DISABLE_HIP_TRANSPORT / MC_ENABLE_HIP_TRANSPORT so + the behaviour can be selected at run time and asserted in the shipped .so. + +upstream_issues: + - ref: kvcache-ai/Mooncake#2724 + url: https://github.com/kvcache-ai/Mooncake/issues/2724 + kind: issue + title: HIP transport is installed unconditionally and preferred over RDMA on ROCm + state: OPEN + state_reason: null + author: jiejingzhangamd + note: "Our report; #2725 is the fix for it." + +upstream_prs: + - ref: kvcache-ai/Mooncake#2725 + url: https://github.com/kvcache-ai/Mooncake/pull/2725 + title: "fix(hip): gate the HIP transport behind an env var (#2724)" + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: jiejingzhangamd + ours: true + same_approach: true + approach_note: Same gate, submitted upstream. No review activity yet. + requested_action: null + in_pinned_base: false + +related_refs: + - ref: kvcache-ai/Mooncake#2682 + url: https://github.com/kvcache-ai/Mooncake/pull/2682 + kind: pr + title: Add HIP (hipIpc) transport + state: MERGED + state_reason: fixed + author: null + relevance: introduced-the-defect + note: >- + This is what installs the HIP transport unconditionally and prefers it over RDMA. + It is IN our pin, which is why every image has to rebuild Mooncake rather than use + the one the base ships. + +problem: + what: >- + Mooncake installs the HIP (hipIpc) transport unconditionally and prefers it over + RDMA, so on a multi-node ROCm deployment the engine picks a transport that only works + within a node. + why: >- + hipIpc memory handles are node-local by construction. Upstream #2682 added the + transport and registered it without any topology or reachability test, and the + selection order puts it ahead of RDMA, so a cross-node target is attempted over + hipIpc and fails at handle-open time. + how: >- + Gate installation behind an explicit env var — MC_DISABLE_HIP_TRANSPORT to force it + off, MC_ENABLE_HIP_TRANSPORT to opt in — so ROCm multi-node deployments fall through + to RDMA. Compiling the gate in unconditionally is what lets the images assert on it. + before_fix: >- + Cross-node PD dies with hipIpcOpenMemHandle error 201. + after_fix: >- + Cross-node PD runs over RDMA. The strings MC_ENABLE_HIP_TRANSPORT and + MC_DISABLE_HIP_TRANSPORT are present in the shipped engine .so, which is what the + build-time assertion checks. + context: >- + Any ROCm multi-node deployment on a Mooncake carrying #2682. Not model-specific and + not arch-specific beyond being ROCm. + call_chain: + - "TransferEngineImpl::init -> installTransport(hip) with no reachability test" + - transport selection prefers hip over rdma + - "cross-node target -> hipIpcOpenMemHandle -> error 201" + symptom_signature: "hipIpcOpenMemHandle ... 201" + silent: false + +verification: + date: 2026-08-01 + hardware: multi-node AMD ROCm with RDMA + software: "Mooncake main @ 747003c (i.e. including #2682)" + workload: cross-node PD KV transfer + result: hipIpcOpenMemHandle 201 cleared; RDMA is selected. + notes: >- + The build-time gate assertion is a separate Dockerfile layer on purpose so it runs in + both BUILD_MOONCAKE branches and shows up in `docker history`. + +scope_limits: >- + The gate does not decide anything by topology — it is an explicit switch. A single-node + ROCm deployment that would genuinely benefit from hipIpc has to opt in with + MC_ENABLE_HIP_TRANSPORT. + +open_actions: + - action: "Get #2725 reviewed; it has had no activity since it was opened." + owner: jiejingzhangamd + blocked_on: upstream review diff --git a/deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.upstream.status.yaml b/deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.upstream.status.yaml new file mode 100644 index 00000000..0afdbfb4 --- /dev/null +++ b/deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.upstream.status.yaml @@ -0,0 +1,169 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/sglang/patch_glm52_nextn_quark_exclude.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.sglang + opt_in_flag: null + +target: + library: sglang + component: quantization + repo: sgl-project/sglang + files: + - python/sglang/srt/models/deepseek_nextn.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: >- + This release line was cut without #30265, which is the whole reason the + backport exists. + +upstream_main_affected: + value: false + evidence: >- + #30265 is MERGED and gives GLM-5.2 its own GlmMoeDsaForCausalLMNextN in + glm4_moe.py, so the GLM path on main is fixed. Read from main on 2026-08-05. + Note that the anchor STRING is still on main — see silent_misapply_risk. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + Without it the GLM-5.2 draft weight-load dies and MTP cannot start. + apply_sglang_dsa_patches.sh asserts this patch as a PREREQUISITE of the DSA set + by grepping deepseek_nextn.py for 'num_hidden_layers}.eh_proj'. + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: not-applied + required_to_run: false + evidence: >- + Deliberately not applied — v0.5.16 carries #30265, so the fix is already in that + base. Dockerfile.sglang.gfx942's header says so explicitly. + +alive_because: + reason: merged-not-in-pinned-base + detail: >- + #30265 merged upstream on 2026-07-08, but our mi35x base is the v0.5.15.post1 + release tag and that release line was cut without it. Bumping a patch level does + not help: v0.5.15.post1 sits on release/v0.5.15, not main. + consumers: + - Dockerfile.sglang (GLM-5.2 MTP on the v0.5.15.post1 mi35x base) + drop_when: >- + Dockerfile.sglang moves to a base sglang that carries #30265 — v0.5.16 or later + already does, which is why the gfx942 image does not apply this. + drop_signal: base-version-only + silent_misapply_risk: >- + CORRECTED 2026-08-05. The patch's own header, and the previous status page, both + claimed the anchor would disappear on a fixed base and the script would no-op. It + does not. #30265 touched deepseek_nextn.py (+13/-11) only to lift the quark check + out of __init__ into _resolve_nextn_quant_config(); the literal + 'ckpt_prefix = f"model.layers.{config.num_hidden_layers}"' is still on main at line + 347, and this patch matches that string without leading indentation. On a + post-#30265 base it therefore still applies — and now edits the DeepSeek MTP path, + which GLM-5.2 no longer takes, narrowing that family's quark-exclude probe to + eh_proj alone while logging success. Decide the drop from the base version. + +history: + born: + date: 2026-07-28 + commit: 0d8d0ff + subject: "fix(image): patch sglang for GLM-5.2 MTP nextn quark-exclude (backport #30265)" + last_modified: + date: 2026-07-28 + commit: 0d8d0ff + subject: "fix(image): patch sglang for GLM-5.2 MTP nextn quark-exclude (backport #30265)" + reason: >- + Written as a deliberately narrow backport in one go — a one-line anchor swap + rather than a port of #30265's new model class. + +upstream_issues: null + +upstream_prs: + - ref: sgl-project/sglang#30265 + url: https://github.com/sgl-project/sglang/pull/30265 + title: "[AMD] Fix GLM-5.2 MTP Quark excludes" + state: MERGED + review_decision: APPROVED + merged_at: 2026-07-08 + author: wangjiaxin99 + ours: false + same_approach: false + approach_note: >- + #30265 is a superset: it adds a dedicated GlmMoeDsaForCausalLMNextN class in + glm4_moe.py plus model_config.py plumbing. Ours is a one-line probe change on the + shared DeepSeek nextn path. We took the narrow form on purpose — this is a + backport onto a frozen release base, not a general fix, and porting a new model + class into a running container is not something to do blind. + requested_action: null + in_pinned_base: false + +related_refs: null + +problem: + what: >- + GLM-5.2's MTP (nextn) layer is entirely bf16/unquantized, but sglang's quark + exclude check does not recognise it as excluded, so eh_proj is built as an MXFP4 + (uint8) parameter and the draft weight-load fails. + why: >- + The quark quantization_config lists exclude entries at SUBMODULE level + ('model.layers..eh_proj'), while DeepseekV3ForCausalLMNextN probes the BARE layer + prefix ('model.layers.'). should_ignore_layer() therefore returns False, the + layer is treated as quantized, and nextn_quant_config is not cleared. + how: >- + Probe the eh_proj submodule — an exact exclude entry — instead of the bare layer, so + the match succeeds, nextn_quant_config becomes None, and the whole bf16 MTP layer is + built bf16. + before_fix: >- + "AssertionError: param.shape=[6144,6144] uint8 vs loaded_weight.shape=[6144,12288] + bf16" during draft weight-load — reported on the status page previously as + '3072 vs 6144'. + after_fix: Coherent GLM-5.2 MTP output on the v0.5.15.post1 base. + context: >- + gfx950 / MI355X, sglang v0.5.15.post1, GLM-5.2 with quark MXFP4 weights and EAGLE + MTP. Not reachable on v0.5.16+, which carries #30265. + call_chain: + - "sglang/srt/models/deepseek_nextn.py :: DeepseekV3ForCausalLMNextN.__init__" + - "should_ignore_layer(mapped_prefix, quant_config.exclude_layers) — probes the bare layer prefix" + - "eh_proj built as an MXFP4 uint8 param" + - "draft weight-load shape assertion fails" + symptom_signature: "param.shape=[6144,6144] uint8 vs loaded_weight.shape=[6144,12288] bf16" + silent: false + +verification: + date: 2026-07-28 + hardware: 8x MI355X (gfx950) + software: sglang v0.5.15.post1, ROCm 7.2.0 + workload: GLM-5.2 quark MXFP4 with EAGLE MTP + result: Draft weight-load succeeds and MTP output is coherent. + notes: null + +scope_limits: >- + Narrow by design: it fixes the probe, not the model class. It does not give GLM-5.2 + the dedicated nextn class #30265 added, and it must not be carried onto a base that + has one. + +open_actions: + - action: >- + Fix this patch's own header, which still claims the anchor disappears once + upstream carries #30265. It does not, and the claim is what makes a silent + misapply possible. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py b/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py index 0a67d8f9..2a2a2279 100644 --- a/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py +++ b/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py @@ -43,13 +43,32 @@ logs confirm the failing prompt is still really split into 4 chunks afterwards. The anchors below are present in both v0.5.15.post1 and v0.5.16. -UPSTREAM: not submitted. The closest existing report, -sgl-project/sglang#25583 (GLM-5-FP8 + NSA + 70k prompt, identical symptom), was -auto-closed with no follow-up; the aggregated-vs-PD A/B above is what it was -missing. Worth measuring when upstreaming: the new `synchronize()` blocks the -transfer worker, trading some transfer overlap for correctness. DROP THIS PATCH -once the base sglang waits on the event in mooncake — this script then reports -"already present" and no-ops. +UPSTREAM: sgl-project/sglang#33970 (OPEN, DRAFT, filed 2026-08-07 by dorado269) +-- "[PD] Make the mooncake KV transfer wait on the prefill forward that wrote +the pages". Same three edits, re-anchored onto main (which has since gained +`TransferKVChunk.staging_counted` and `_prepare_send_indices`, so the anchors +moved). The closest existing report, sgl-project/sglang#25583 (GLM-5-FP8 + NSA + +70k prompt, identical symptom), was auto-closed with no follow-up; the +aggregated-vs-PD A/B above is what it was missing, and is cited in the PR. + +STILL A DRAFT: the 2-node PD pair was unreachable when it was opened, so it has +NOT been re-run end-to-end against this branch. What WAS checked on a single +MI300X: `wait_event` defaults to None so existing construction sites are +unaffected; `send()` forwards it on BOTH arms and clears the sender's copy; and +the wait is a real barrier (event pending after record, `synchronize()` +measurably blocks, a post-barrier read on another stream sees the writes). +STILL UNMEASURED: the new `synchronize()` blocks the transfer worker, trading +some transfer overlap for correctness. The PR says so plainly rather than +implying it is free -- it is the first thing a reviewer will ask. + +WORTH KNOWING when reading the prefill.py edit: that file is transport-agnostic, +so recording the event on the overlap non-final-chunk send also closes the same +gap for MORI, which reads the field but was never handed one on that path. +nixl / ascend / fake / base do not read `_early_send_wait_event` and no sender +uses __slots__, so setting the attribute is inert for them. + +DROP THIS PATCH once the base sglang waits on the event in mooncake -- this +script then reports "already present" and no-ops. Self-locating and idempotent. All three files or none: a half-patched tree still corrupts long prompts, so an anchor that is missing or no longer unique writes diff --git a/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml b/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml new file mode 100644 index 00000000..d926dece --- /dev/null +++ b/deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.upstream.status.yaml @@ -0,0 +1,218 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-07 +verified_by: + - gh-pr-issue-state + - gh-search + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/sglang_disagg/patch_mooncake_early_send_wait_event.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + opt_in_flag: null + +target: + library: sglang + component: disaggregation + repo: sgl-project/sglang + files: + - python/sglang/srt/disaggregation/common/utils.py + - python/sglang/srt/disaggregation/mooncake/conn.py + - python/sglang/srt/disaggregation/prefill.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: null + - surface: Dockerfile.sglang.gfx942 + ref: v0.5.16 + commit: fdebc938f7f4 + pinned_ref_on_main: false + release_branch: release/v0.5.16 + images: + - tag: lmsysorg/sglang:v0.5.16-rocm720-mi30x + digest: null + digest_source: unresolved + note: The anchors are present in both v0.5.15.post1 and v0.5.16. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: disaggregation/mooncake/conn.py contains ZERO + occurrences of `wait_event`, while disaggregation/mori/conn.py has five (the + TransferKVChunk field, the pickup off the sender, the forward, and the + transfer_worker wait). Main is affected, and this is a source read rather than a + search miss. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + Required for correctness, not startup: without it any prompt longer than one + prefill chunk comes back partially wrong over mooncake PD. + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: op + required_to_run: true + evidence: >- + Verified on this base directly — 2x 8x MI325X, GLM-5.2-FP8 1P1D, needle retrieval + 5/9 -> 9/9. + +alive_because: + reason: upstream-pr-open + detail: >- + Ours is now upstream as sglang#33970 (DRAFT, opened 2026-08-07). The closest existing + report, #25583, describes the identical corruption shape but on an AGGREGATED server + with no PD and no mooncake, so a shared root cause is unestablished; it was + auto-closed inactive with no follow-up. The aggregated-vs-PD A/B this patch rests on + is exactly what that issue was missing, and is the core of the PR's argument. + Was `no-upstream-pr` until #33970. + consumers: + - Dockerfile.sglang (any PD deployment with chunked prefill over mooncake) + - Dockerfile.sglang.gfx942 (same) + drop_when: >- + A base sglang synchronizes on the completion event prefill.py already records — the + script then reports "already present" and no-ops. + drop_signal: self-guard-marker-skips + silent_misapply_risk: null + +history: + born: + date: 2026-07-31 + commit: ea07d53 + subject: "fix(image): apply the mooncake PD wait-event patch at image build time" + last_modified: + date: 2026-07-31 + commit: ea07d53 + subject: "fix(image): apply the mooncake PD wait-event patch at image build time" + reason: >- + Moved into the image build in the same change that introduced it, so a PD + deployment cannot silently run without it. Preceded by 786f238, which carried the + patch itself. + +upstream_issues: + - ref: sgl-project/sglang#25583 + url: https://github.com/sgl-project/sglang/issues/25583 + kind: issue + title: "[Bug] Long text response error on GLM-5-FP8 with NSA backend" + state: CLOSED + state_reason: inactive — auto-closed 2026-07-18 with no follow-up + author: dingfen + note: >- + Same corruption shape on GLM-5, but aggregated: no PD and no mooncake. So it may + or may not share this root cause. Treat as suggestive, not confirming. + +upstream_prs: + - ref: sgl-project/sglang#33970 + url: https://github.com/sgl-project/sglang/pull/33970 + title: "[PD] Make the mooncake KV transfer wait on the prefill forward that wrote the pages" + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: dorado269 + ours: true + same_approach: true + approach_note: >- + Opened 2026-08-07 as a DRAFT. Same three edits as this patch, re-anchored onto + main (which has since gained TransferKVChunk.staging_counted and + _prepare_send_indices, so the anchors moved). One behaviour worth knowing that the + local patch header does not spell out: prefill.py is transport-agnostic, so + recording the event on the overlap non-final-chunk send also closes the same gap + for MORI, which reads the field but was never handed one on that path. Checked + that nixl / ascend / fake / base do not read _early_send_wait_event and that no + sender class uses __slots__, so setting the attribute is inert for them. + requested_action: >- + Draft until re-validated on a 2-node PD pair, and until the added synchronize()'s + cost on prefill throughput is measured — the PR states plainly that it is not. + in_pinned_base: false + +related_refs: null + +problem: + what: >- + With chunked prefill over the mooncake KV transport, every prefill chunk except the + last can be RDMA-read while the forward that writes those pages is still running, so + the decode leg receives half-written KV. + why: >- + prefill.py's early-send path already records a completion event as the barrier + (req.disagg_kv_sender._early_send_wait_event), and the comment there says exactly + what it is for — but only mori/conn.py ever reads it. mooncake/conn.py has no + wait_event or synchronize() anywhere, so on mooncake the barrier has never taken + effect. Worse, the overlap-scheduling path that actually moves non-final chunks + (process_batch_result_disagg_prefill) does not even record an event. The final chunk + is always correct because it goes through the sampling path, which has a real + copy_done.synchronize(). + how: >- + Three edits, all mirroring what mori already does. TransferKVChunk gains a + wait_event field so the barrier travels with the work item; mooncake's send() picks + the event up off the sender, add_transfer_request() forwards it, and transfer_worker + synchronizes on it BEFORE reading device memory; and prefill.py's overlap non-final + chunk send records an event on forward_stream. + before_fix: >- + No crash and nothing in any log — output is PARTIALLY wrong for prompts longer than + one prefill chunk. On GLM-5.2 DSA needle-in-a-haystack returns the first digits of + the needle then degenerates: want=2183762 got='21832183218'. + The corruption boundary lands exactly on the chunk boundary. + after_fix: >- + Needle retrieval 5/9 -> 9/9 and a 29k depth sweep 4/9 -> 9/9, with logs confirming + the failing prompt is still really split into 4 chunks afterwards (so the fix is a + barrier, not an accidental disabling of chunking). + context: >- + 2x 8x MI325X (gfx942), ROCm 7.2.0, sglang v0.5.16, GLM-5.2-FP8 1P1D over mooncake + RDMA, overlap scheduling ON, --chunked-prefill-size 131072, --enable-dp-attention. + NOT DSA-specific: any PD deployment running chunked prefill over mooncake with + overlap scheduling is affected. The same model, same backends and same chunk size in + an aggregated single-node server passes 9/9, which is the A/B that localises it to + the transport. + call_chain: + - "sglang/srt/disaggregation/prefill.py :: process_batch_result_disagg_prefill (overlap path)" + - "send_kv_chunk(..., last_chunk=False) — no event recorded on this path" + - "sglang/srt/disaggregation/mooncake/conn.py :: send() -> add_transfer_request()" + - "transfer_worker reads device memory with no synchronize() — races the forward still writing those pages" + symptom_signature: >- + partially-correct long-prompt output with the error boundary on the prefill chunk + boundary; no log line, no crash + silent: true + +verification: + date: 2026-07-31 + hardware: 2x 8x MI325X (gfx942) + software: sglang v0.5.16, ROCm 7.2.0 + workload: GLM-5.2-FP8 1P1D over mooncake RDMA, overlap scheduling on, chunked prefill 131072 + result: needle 5/9 -> 9/9; 29k depth sweep 4/9 -> 9/9; chunk count unchanged at 4 + notes: >- + DSA's sparse retrieval only makes this conspicuous ("retrieved half the digits"); + on a dense model the same defect is a quiet quality drop. + +scope_limits: null + +open_actions: + - action: >- + Re-validate sglang#33970 on a 2-node PD pair and take it out of draft. What WAS + checked on a single MI300X against the PR's exact diff: the wait_event field + defaults to None so existing construction sites are unaffected; send() forwards it + on both the last-chunk and non-last-chunk arms and clears the sender's copy; and + the wait is a real barrier (event pending after record, synchronize() measurably + blocks, a post-barrier read on another stream sees the writes). What was NOT: that + the corruption is gone end-to-end. + owner: dorado269 + blocked_on: 2-node RDMA cluster access (unreachable 2026-08-07) + - action: >- + Measure the added synchronize()'s cost on prefill throughput — it sits on the hot + chunk path. Still unmeasured; #33970 says so explicitly rather than implying it is + free, and it is the first thing a reviewer will ask. + owner: unassigned + blocked_on: same cluster access diff --git a/deploy/docker/patches/sglang_dsa/README.md b/deploy/docker/patches/sglang_dsa/README.md index b6c6c24a..8a2b333a 100644 --- a/deploy/docker/patches/sglang_dsa/README.md +++ b/deploy/docker/patches/sglang_dsa/README.md @@ -36,8 +36,11 @@ established, the upstream issue / third-party PR / our own PR, how it differs from our own upstream PR, and whether the IndexShare workaround substitutes for it. Read the `.diff` before changing it. -Upstream linkage for these and every other patch in the repo is indexed in -[`deploy/docker/patch.upstream.status.md`](../../patch.upstream.status.md). +Each patch also has a machine-checked record beside it — +`.upstream.status.yaml` — carrying its upstream state, per-engine +effect and drop condition. Upstream linkage for these and every other patch in +the repo is indexed in +[`deploy/docker/patch.upstream.status.yaml`](../../patch.upstream.status.yaml). ## Applying diff --git a/deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.upstream.status.yaml b/deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.upstream.status.yaml new file mode 100644 index 00000000..a4d9e857 --- /dev/null +++ b/deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.upstream.status.yaml @@ -0,0 +1,227 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/sglang_dsa/draft_cuda_graph_dp_vote.diff + kind: context-diff + idempotent: false + applied_by: + - Dockerfile.sglang + - deploy/docker/scripts/apply_sglang_dsa_patches.sh + opt_in_flag: APPLY_SGLANG_DSA_PATCHES=1 with DSA_PATCH_SET=full (Dockerfile.sglang only) + +target: + library: sglang + component: speculative-decoding + repo: sgl-project/sglang + files: + - python/sglang/srt/speculative/eagle_worker_v2.py + - python/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py + - python/sglang/srt/layers/dp_attention.py + - python/sglang/srt/model_executor/forward_batch_info.py + - python/sglang/srt/managers/schedule_batch.py + - python/sglang/srt/disaggregation/decode.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: >- + --fuzz=0 against this tag. The gfx942 v0.5.16 image substitutes a runtime + flag instead of taking this diff. + +upstream_main_affected: + value: true + evidence: >- + #32209, the PR carrying this fix, is still OPEN, and the guard it repairs was + introduced by #30839/#31083 which are merged. So main has the guard and not the fix. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + apply_sglang_dsa_patches.sh asserts five bytecode markers for this diff + (dp_attn.py:can_draft_cuda_graph, eagle_worker_v2.py:requires_dp_attention_eager_forward, + eagle_draft_cuda_graph_runner.py:can_run_dp_draft_cuda_graph, + forward_batch_info.py:can_run_dp_draft_cuda_graph, + schedule_batch.py + decode.py:force_disable_draft_cuda_graph). + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: not-applied + required_to_run: false + evidence: >- + Substituted at runtime with + --json-model-override-args '{"index_share_for_mtp_iteration":false}', which makes + the rank-dependent term-4 seed unreachable rather than making the choice a group + decision. Cheaper, narrower, and it costs the IndexShare optimisation. + +alive_because: + reason: upstream-pr-open + detail: >- + #32209 carries this same fix with the same strategy and this diff adopts its + placement verbatim. We deliberately did NOT open a competing PR — the right move is + to converge on #32209, which is better placed. It has been open and unreviewed + since before our fix existed. + consumers: + - Dockerfile.sglang (GLM-5.2 PD decode leg with DP-attention + EAGLE MTP on gfx950) + drop_when: >- + #32209 merges and lands in a base sglang we build against, or the guard introduced + by #30839 is reworked so the choice is no longer per-rank. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-08-01 + commit: c91db76 + subject: "sglang DSA: enable PD + DP-attention + EAGLE MTP for GLM-5.2 on gfx950" + last_modified: + date: 2026-08-01 + commit: c91db76 + subject: "sglang DSA: enable PD + DP-attention + EAGLE MTP for GLM-5.2 on gfx950" + reason: >- + Cut once against v0.5.15.post1 in the change that enabled the topology; no + revision needed since. + +upstream_issues: + - ref: sgl-project/sglang#32527 + url: https://github.com/sgl-project/sglang/issues/32527 + kind: issue + title: "[BUG] EAGLE + DP Attention + PD Disaggregation: Deadlock when index_share_for_mtp_iteration is enabled for GLM-5.2" + state: OPEN + state_reason: null + author: Xavier1994 + note: >- + The same defect, reported independently on 8x Blackwell / GLM-5.2-FP8 on + 2026-07-27 — two days before we fixed it — with the same analysis. It proposes a + THIRD strategy: a dummy all-zeros seed so term 4 is always false. No activity + since it was filed. + +upstream_prs: + - ref: sgl-project/sglang#32209 + url: https://github.com/sgl-project/sglang/pull/32209 + title: Fix PD decode hang with DP attention and GLM-5.2 MTP + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: HZY-Wade + ours: false + same_approach: true + approach_note: >- + Same defect, same strategy — a group-wide decision instead of a per-rank one. This + diff adopts its placement verbatim so the two do not diverge. We opened no PR of + our own on purpose: a competing PR would be noise and #32209 is better placed. + requested_action: null + in_pinned_base: false + +related_refs: + - ref: sgl-project/sglang#30839 + url: https://github.com/sgl-project/sglang/pull/30839 + kind: pr + title: "[bug-fix] Stabilize GLM-5.2 MTP IndexShare across PD and CUDA graph replay" + state: MERGED + state_reason: fixed + author: zRzRzRzRzRzRzR + relevance: introduced-the-defect + note: >- + This is where the per-rank guard came from, cherry-picked to release/v0.5.15 as + #31083. So the deadlock is a regression in our baseline, not a legacy wart. + - ref: sgl-project/sglang#31083 + url: https://github.com/sgl-project/sglang/pull/31083 + kind: pr + title: "[Cherry-pick to release/v0.5.15] Stabilize GLM-5.2 MTP IndexShare across PD and CUDA graph replay (#30839)" + state: MERGED + state_reason: fixed + author: kpham-sgl + relevance: introduced-the-defect + note: The cherry-pick that put the guard into the exact release line we pin. + - ref: sgl-project/sglang#32196 + url: https://github.com/sgl-project/sglang/pull/32196 + kind: pr + title: "[PD] Keep EAGLE DP graph and token metadata consistent" + state: OPEN + state_reason: null + author: weireweire + relevance: adjacent-site + note: Same area, different site. Not a fix for this defect. + - ref: sgl-project/sglang#32722 + url: https://github.com/sgl-project/sglang/pull/32722 + kind: pr + title: "[RED regression] Test GLM-5.2 PD + DP attention + MTP" + state: OPEN + state_reason: null + author: IvanShan177 + relevance: missing-test-coverage + note: >- + A test for exactly this topology. Its existence proves no CI covers it today, + which is why this family of defects reaches a released base. + +problem: + what: >- + EagleDraftWorker.draft() decides PER RANK whether to replay the draft CUDA graph or + run the multi-step draft eagerly. The two paths do not issue the same host-side + collective sequence, so when ranks disagree the whole DP group deadlocks. + why: >- + Two of the guard's four terms are rank-dependent by construction. `not + forward_batch.forward_mode.is_idle()` differs per rank every step. And + `draft_input.dsa_topk_indices is None` is, on the PD DECODE leg, seeded from + RDMA-shipped per-request payloads (eagle_disaggregation.py), so it is a function of + which requests a rank happens to hold. Single-node MIX never hangs because with no + disaggregation the top-k seed is produced locally by identical code on every rank, + so term 4 never flips asymmetrically. + how: >- + Make the choice a GROUP decision using one more int64 slot in the MLP-sync + all-gather the scheduler already performs, min()-reduced so any rank needing eager + takes the whole group eager. Inactive ranks contribute 1 (permissive) and so can + never drag the group into eager on their own. No new collective is introduced. + before_fix: >- + The first routed request hard-deadlocks the PD decode leg. No error, no traceback — + the group simply stops. + after_fix: >- + GLM-5.2 PD with DP-attention and EAGLE MTP serves normally on gfx950, with + IndexShare left enabled. + context: >- + gfx950 / MI355X, sglang v0.5.15.post1, GLM-5.2-FP8, PD disaggregation, DP-attention + with EAGLE MTP. Reproduces only under disaggregation; single-node mixed serving is + unaffected. Independently reported upstream on 8x Blackwell, so it is not + ROCm-specific. + call_chain: + - "sglang/srt/speculative/eagle_worker_v2.py :: EagleDraftWorker.draft()" + - "guard term 2: forward_batch.forward_mode.is_idle() — per-rank occupancy" + - "guard term 4: draft_input.dsa_topk_indices is None — seeded from RDMA payloads on the PD decode leg" + - "divergent branch -> mismatched host-side collective sequence -> group deadlock" + symptom_signature: silent hang on the first routed PD decode request, no log output + silent: false + +verification: + date: 2026-08-01 + hardware: 8x MI355X (gfx950) + software: sglang v0.5.15.post1, ROCm 7.2.0 + workload: GLM-5.2-FP8 PD, DP-attention + EAGLE MTP, IndexShare enabled + result: First-request deadlock cleared; the group takes the same path on every rank. + notes: >- + The all-gather slot rides the sync the scheduler already does, so the fix costs no + extra collective. + +scope_limits: >- + Cut --fuzz=0 against v0.5.15.post1. Not re-cut for v0.5.16 — the gfx942 image works + around the same defect with a runtime flag that disables IndexShare instead. + +open_actions: + - action: >- + Support #32209 upstream rather than opening a competing PR; add the ROCm datapoint + to that thread, and to #32527 which has had no follow-up. + owner: unassigned + blocked_on: "upstream review of #32209" diff --git a/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff b/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff index b1c9b64c..5cf671fa 100644 --- a/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff +++ b/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff @@ -50,10 +50,38 @@ third-party PR NONE for either half. Per-diff greps: #31683 does not touch Weaker than it sounds: `gh search` matches titles and bodies, not diff content, so an upstream PR could touch either site without naming it. -own PR NONE. Not upstreamed. 2a's evidence is runtime-state, not a - revert, so it is harder to present upstream than patch 01 was; - 2b's upstream counterpart would be #32209's other half, which we - could not make work here (see below). +own PR 2a ONLY -- #33973 (OPEN, DRAFT, filed 2026-08-07 by dorado269), + "[DSA] Remove the device-to-host syncs on the decode + DP-divergent branch". Carries 2a AND 2a2. 2b is deliberately + NOT in it: one fix, one PR, and 2b overlaps #32209 while resting + on a concurrency-32 failure whose cause is unidentified (below). + So THIS PATCH OUTLIVES #33973 -- landing it retires only half. + + Still a draft: gfx950 was unreachable when it was opened, so it + has NOT been re-run against the deadlock on hardware that + reproduces it. What WAS checked, on MI300X: `.max().item()` + measurably blocks (~0.5-3 ms run to run, behind 30 queued + 4096^2 matmuls) while `.shape[1]` does not (~3 us) -- the point + is that it blocks AT ALL on a branch only some ranks take, not + the magnitude; widening the page table leaves + every REAL top-k selection and score bit-identical; and + DRAFT_EXTEND_V2 is not is_extend(), so every consumer of the two + removed mirrors is unreachable. + + Two things the PR says that this header did not: + * the strongest argument for 2a is that the backend ALREADY + declares `needs_cpu_seq_lens = False`, with a comment saying + it opts out of the D2H sync -- so the eager fallback + contradicts its own contract; and + * the CPU mirror this branch relies on comes from + eagle_worker_common.prepare_for_draft_extend (line 105), on + its gpu_only branch and for this same reason -- NOT from + base_spec_worker, as stated further down in this file. + A caveat found while validating: a row with seq_len < topk does + return indices past its own seq_len, because top-k must return + topk entries. Those slots carry -inf and occur identically at + BOTH widths, so they are a consequence of seq_len < topk, not of + widening -- the property that holds is about the REAL selections. HOW 2a WAS ESTABLISHED -------------------------------------------------------------------------------- diff --git a/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml b/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml new file mode 100644 index 00000000..bfdc3955 --- /dev/null +++ b/deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.upstream.status.yaml @@ -0,0 +1,230 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-07 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff + kind: context-diff + idempotent: false + applied_by: + - Dockerfile.sglang + - deploy/docker/scripts/apply_sglang_dsa_patches.sh + opt_in_flag: APPLY_SGLANG_DSA_PATCHES=1 with DSA_PATCH_SET=full (Dockerfile.sglang only) + +target: + library: sglang + component: dsa + repo: sgl-project/sglang + files: + - python/sglang/srt/layers/attention/dsa_backend.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: >- + Applied with `patch -p1 --fuzz=0`, cut against this tag. It cannot apply to + the gfx942 image's v0.5.16 base and is deliberately not attempted there. + +upstream_main_affected: + value: true + evidence: >- + No upstream PR exists for 2a at all, and #32209 (the only PR touching 2b) is still + OPEN, so neither half is on main. Established by search plus the absence of any + merged candidate, not by reading main — weaker than this repo's usual standard. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + apply_sglang_dsa_patches.sh asserts the dsa_backend.py:_glm52_match_page_table_rows + bytecode marker and patch 2a's source marker under DSA_PATCH_SET=full. + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: not-applied + required_to_run: false + evidence: >- + DSA_PATCH_SET=indexer skips it: the diff is --fuzz=0 against v0.5.15.post1 and + will not apply to v0.5.16. That image substitutes 2b at runtime with + --json-model-override-args '{"index_share_for_mtp_iteration":false}' and does not + address 2a at all — 2a's deadlock has not been observed on that base. + +alive_because: + reason: upstream-pr-open + detail: >- + SPLIT, and only half is upstream. 2a is now ours as sglang#33973 (DRAFT, opened + 2026-08-07) — it had no upstream issue and no PR of any kind before that. 2b is NOT + in that PR and has no PR of ours: the one third-party PR (#32209) solves the same + row mismatch by trimming q/top-k instead of expanding the page table, and porting + that half here fails at concurrency 32 with the cause unidentified, so we can + neither adopt it nor justify a competing PR. This patch therefore stays until BOTH + halves land. + consumers: + - Dockerfile.sglang (GLM-5.2 DSA PD decode leg with DP-attention + MTP) + drop_when: >- + A base sglang removes the host sync from the DP-divergent branch (2a) and makes the + decode page table token-major under MTP (2b) — or #32209's trimming approach is + made to work at concurrency 32. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-08-01 + commit: c91db76 + subject: "sglang DSA: enable PD + DP-attention + EAGLE MTP for GLM-5.2 on gfx950" + last_modified: + date: 2026-08-01 + commit: c91db76 + subject: "sglang DSA: enable PD + DP-attention + EAGLE MTP for GLM-5.2 on gfx950" + reason: >- + Born and last touched in the same change — the diff was cut once against + v0.5.15.post1 and has not needed a revision. + +upstream_issues: null + +upstream_prs: + - ref: sgl-project/sglang#33973 + url: https://github.com/sgl-project/sglang/pull/33973 + title: "[DSA] Remove the device-to-host syncs on the decode DP-divergent branch" + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: dorado269 + ours: true + same_approach: true + approach_note: >- + Opened 2026-08-07 as a DRAFT, carrying 2a AND 2a2 — and deliberately NOT 2b, which + overlaps #32209 and rests on an unexplained concurrency-32 failure. One fix, one + PR. Adapting it surfaced a stronger upstream argument than this record had: the + backend already declares `needs_cpu_seq_lens = False` with a comment saying it + opts out of the D2H sync, so the eager fallback contradicts its own contract. + Also corrects a citation in this record — the CPU mirror the DRAFT_EXTEND_V2 branch + relies on comes from eagle_worker_common.prepare_for_draft_extend (line 105), on + its gpu_only branch and for this same reason, not from base_spec_worker. + requested_action: >- + Draft until re-validated on gfx950 with PD + DP-attention + MTP. + in_pinned_base: false + - ref: sgl-project/sglang#32209 + url: https://github.com/sgl-project/sglang/pull/32209 + title: Fix PD decode hang with DP attention and GLM-5.2 MTP + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: HZY-Wade + ours: false + same_approach: false + approach_note: >- + Addresses 2b's row mismatch by TRIMMING q and top-k down to the request count, + where this diff EXPANDS the page table to the token count with repeat_interleave. + Porting the trimming half here fails at concurrency 32 and the cause is + unresolved, so we kept the expansion. Nothing upstream covers 2a. + requested_action: null + in_pinned_base: false + +related_refs: + - ref: sgl-project/sglang#32722 + url: https://github.com/sgl-project/sglang/pull/32722 + kind: pr + title: "[RED regression] Test GLM-5.2 PD + DP attention + MTP" + state: OPEN + state_reason: null + author: IvanShan177 + relevance: missing-test-coverage + note: >- + Its existence proves no upstream CI covers this topology today, which is why this + whole family of defects survives in a released base. + +problem: + what: >- + Two independent defects in dsa_backend.py, both on the PD decode leg with MTP. + (2a) A blocking device-to-host sync sits on a branch only SOME DP ranks take, so + the DP collectives desynchronize and the group deadlocks. (2b) The page table has + one row per REQUEST while top-k has one row per TOKEN, so an assert kills every + rank. + why: >- + 2a — `max_seqlen_k = int(forward_batch.seq_lens.max().item())` is a host sync, and + idle DP peers keep their host mirror and take the cheap arm, so the ranks do not + issue the same collective sequence. Two further unconditional `.cpu()` syncs on the + same branch have to go too; with the first fix alone the hang persists. + 2b — `metadata.page_table_1` is per-request, while `topk_indices` has just been + padded to `q.shape[0]`, i.e. per-token. Equal for plain decode, unequal under MTP. + how: >- + 2a — replace the sync with the sync-free `self.req_to_token.shape[1]`. Over- + allocating page-table columns is safe: the table is only indexed THROUGH top-k, + which masks per row by cache_seqlens, so extra columns get -inf and are never + selected. The idiom is already established in-tree three times (triton_backend.py, + trtllm_mha_backend.py, and DSA's own graph path). The removed `.cpu()` mirrors are + dead for DRAFT_EXTEND_V2, which is not is_extend(), so every consumer is + unreachable. 2b — expand the page table with repeat_interleave to reproduce the + token-major layout, at BOTH decode call sites; the rows `_pad_topk_indices` added + hold all -1 and the triton kernel already masks them, so only the row COUNT has to + match. + before_fix: >- + 2a — the whole DP group hangs with no error. 2b — `assert page_table.shape[0] == + topk_indices.shape[0]` in transform_index_page_table_decode_fast, and every rank + dies. + after_fix: >- + GLM-5.2 DSA serves PD with DP-attention and EAGLE MTP on gfx950. Both bytecode + markers assert clean in the build. + context: >- + gfx950 / MI355X, sglang v0.5.15.post1, GLM-5.2 DSA, PD disaggregation, DP-attention + with EAGLE MTP. The traceback for 2b named only the first decode call site; the + second pairs the same unpadded page table with a padded topk_indices, so both are + patched. The prefill sibling solves 2b with an explicit output_num_tokens; the + decode entry point has no equivalent. + call_chain: + - "dsa_backend.py :: init_forward_metadata (decode, DRAFT_EXTEND_V2)" + - "2a: seq_lens.max().item() — host sync on a DP-divergent branch" + - "2b: transform_index_page_table_decode_fast — asserts page_table rows == topk rows" + symptom_signature: "assert page_table.shape[0] == topk_indices.shape[0]" + silent: false + +verification: + date: 2026-08-01 + hardware: 8x MI355X (gfx950) + software: sglang v0.5.15.post1, ROCm 7.2.0 + workload: GLM-5.2-FP8 PD, DP-attention + EAGLE MTP + result: Deadlock and assert both cleared; every consumer of the removed mirrors walked by hand. + notes: >- + 2a2's removals were established dead by reading each consumer: extend_prefix_lens_cpu + is read only inside the is_extend() arm, seq_lens_sum at the is_extend()-gated + capacity check, indexer_seq_lens_cpu is already None here, and both + _cal_indexer_k_start_end and dsa_indexer's k-only path open with + is_extend_without_speculative(). + +scope_limits: >- + Cut --fuzz=0 against v0.5.15.post1 and not re-cut for v0.5.16, so the gfx942 image + runs without it. 2a's deadlock has not been observed on that base, and 2b is worked + around there with a runtime flag rather than fixed. + +open_actions: + - action: >- + Re-validate sglang#33973 (2a + 2a2) on gfx950 with PD + DP-attention + MTP and take + it out of draft. What WAS checked on MI300X against the PR's exact diff: that + `.max().item()` measurably blocks (~0.5-3 ms run to run, behind 30 queued 4096^2 + matmuls) while `.shape[1]` does not (~3 us) — the point being that it blocks at + all on a branch only some ranks take, not the absolute figure; that widening the + page table leaves every REAL top-k + selection and score bit-identical; and that DRAFT_EXTEND_V2 is not is_extend(), so + every consumer of the two removed mirrors is unreachable. What was NOT: that the + group actually stops deadlocking. + owner: dorado269 + blocked_on: MI355X cluster access (unreachable 2026-08-07) + - action: >- + Establish why #32209's trimming approach fails at concurrency 32 here, so 2b can + converge upstream instead of diverging. Until then 2b keeps this patch alive on + its own, even once #33973 lands. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.upstream.status.yaml b/deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.upstream.status.yaml new file mode 100644 index 00000000..dac00fba --- /dev/null +++ b/deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.upstream.status.yaml @@ -0,0 +1,229 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + - deploy/docker/scripts/apply_sglang_dsa_patches.sh + opt_in_flag: APPLY_SGLANG_DSA_PATCHES=1 (default on; DSA_PATCH_SET selects the arm) + +target: + library: sglang + component: dsa + repo: sgl-project/sglang + files: + - python/sglang/srt/layers/attention/dsa_indexer.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: Dockerfile.sglang pins the tag, not a digest. + - surface: Dockerfile.sglang.gfx942 + ref: v0.5.16 + commit: fdebc938f7f4 + pinned_ref_on_main: false + release_branch: release/v0.5.16 + images: + - tag: lmsysorg/sglang:v0.5.16-rocm720-mi30x + digest: null + digest_source: unresolved + note: >- + This is the only one of the three sglang_dsa patches the gfx942 base can + take, because it anchors on source text rather than being a --fuzz=0 diff. + +upstream_main_affected: + value: true + evidence: >- + Our own PR sglang#33059 is still OPEN and unreviewed against main, i.e. the aiter + branch on main still sizes its logits from the padded row count. Not re-read from + main source on this sweep; the open PR is the evidence. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + Without it GLM-5.2 DSA with DP-attention + EAGLE MTP crashes on the very first + batch on gfx950. Verified by the 8-marker bytecode assertion in + apply_sglang_dsa_patches.sh (DSA_PATCH_SET=full). + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: op + required_to_run: true + evidence: >- + DSA_PATCH_SET=indexer asserts the dsa_indexer.py:_p1v2_trim bytecode marker. + Without it DP-attention crashes at concurrency > 1 on this base. + +alive_because: + reason: upstream-pr-open + detail: >- + Our fix is upstream as sglang#33059 but has had no review since it was opened, and + the two engine bases are release-branch pins that would not pick up a main merge + anyway. Three other open PRs touch the same two aiter call sites, so the shape of + the eventual upstream fix is still unsettled. + consumers: + - Dockerfile.sglang (GLM-5.2 DSA PD + DP-attention + MTP on gfx950) + - Dockerfile.sglang.gfx942 (GLM-5.2 DSA on gfx942, any concurrency > 1) + drop_when: >- + A base sglang reconciles the aiter paged-MQA row count with lengths — whether via + #33059, #32762 or the #31480 backend extraction. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-08-03 + commit: "1380228" + subject: "fix(image): carry the DSA indexer row fix onto the gfx942 v0.5.16 base" + last_modified: + date: 2026-08-03 + commit: 9380f07 + subject: "fix(glm5.2): agentic-serving fixes for DSA indexer, ROCm hicache and PD/kvd wiring" + reason: >- + Added the GLM52_P1V3 arm. The original revision guarded only real < padded, + assuming DP padding always makes q_fp8 longer; a DP-idle rank under MTP + draft-extend inverts that and there is nothing to trim, so the lengths side is + clipped through topk_transform's existing ke_offset instead. + supersedes: + - deploy/docker/patches/sglang_dsa/dsa_indexer_hip_dp_padded_rows.diff + +upstream_issues: null + +upstream_prs: + - ref: sgl-project/sglang#33059 + url: https://github.com/sgl-project/sglang/pull/33059 + title: Fix DSA indexer aiter (HIP) padding mismatch under DP-attention + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: dorado269 + ours: true + same_approach: true + approach_note: >- + This is the same fix, submitted upstream. It is written in the shape of the NPU + PR #32762 deliberately, so the two can converge rather than compete. + requested_action: null + in_pinned_base: false + +related_refs: + - ref: sgl-project/sglang#32762 + url: https://github.com/sgl-project/sglang/pull/32762 + kind: pr + title: "[NPU] Fix DSA eager padding mismatch in PD MTP warm-up" + state: OPEN + state_reason: null + author: stellaxcpeng + relevance: same-defect-other-platform + note: >- + Same bug class on NPU. Our diff is written in its shape; the real < padded arm is + exactly its case. + - ref: sgl-project/sglang#32738 + url: https://github.com/sgl-project/sglang/pull/32738 + kind: pr + title: "[Fix] DSA Indexer: pad heads for DeepGEMM paged MQA logits on decode/target-verify" + state: OPEN + state_reason: null + author: aqni + relevance: anchor-collision + note: >- + Pads heads at the same two aiter call sites we edit. Not a fix for our defect — + expect a conflict, not a resolution. + - ref: sgl-project/sglang#31480 + url: https://github.com/sgl-project/sglang/pull/31480 + kind: pr + title: "[DSA] Add an arch-independent torch paged-MQA-logits backend with a fused Triton fast path" + state: OPEN + state_reason: null + author: vroomfondel + relevance: anchor-collision + note: >- + Extracts the paged-MQA backend and restructures the is_aiter() dispatch our + anchors sit in. If this lands first the patch has to be rewritten, not rebased. + - ref: sgl-project/sglang#30378 + url: https://github.com/sgl-project/sglang/pull/30378 + kind: pr + title: "[DSA] Re-enable fused top-k v2 for MTP: clamp padded-row seq_lens to >= 0" + state: MERGED + state_reason: fixed + author: DarkSharpness + relevance: background + note: >- + Already in the base. Clamps padded-row seq_lens VALUES; this patch fixes the + HIP-side row COUNT. Different defect, adjacent code. + +problem: + what: >- + The aiter (HIP) paged-MQA branch of the DSA lightning indexer sizes its logits + output from one row count while `lengths` is sized from the other, so fast_topk_v2 + is handed a score tensor and a lengths tensor that disagree. + why: >- + Under DP-attention the two counts diverge in BOTH directions. When q_fp8 carries DP + padding the real count is smaller, which is the case every CUDA backend already + handles by taking a q_offset= and slicing internally; ROCm's aiter entry point has + no such parameter, so nothing slices. On a DP-idle rank under MTP draft-extend the + inequality inverts: q_offset (= sum of dsa_extend_len_cpu) is 2 while only 1 row is + materialized in q_fp8, so there are FEWER query rows than lengths entries and a + trim cannot help. + how: >- + Reconcile both counts to min(real, padded). When q_fp8 is the longer side, trim + q/weights to the real count and restore the padding after topk_transform — the + contract the CUDA backends already honour. When q_fp8 is the shorter side, clip the + lengths instead, through topk_transform's existing ke_offset parameter. `_p1v2_clip` + and `_p1v2_rows` are bound BEFORE the is_aiter() test because the former is read + unconditionally at the topk_transform call. + before_fix: >- + RuntimeError "Expected lengths.size(0) == B to be true, but got false" — on gfx950 + at the very first batch, on gfx942 at concurrency > 1. In the inverted (P1V3) + direction the scheduler rank dies and the router drops to `active_workers: 1`. + after_fix: >- + GLM-5.2 DSA runs with DP-attention and EAGLE MTP on both bases. The debug hook + SGLANG_DEBUG_DSA_ROWS=1 shows the reconciled counts, including the idle-rank case + mode=IDLE q_fp8=(1,32,128) q_offset=2 -> mqa_q=(1,32,128) with lengths clipped. + context: >- + ROCm-specific: gfx950 (MI355X) and gfx942 (MI325X), GLM-5.2 DSA, DP-attention with + EAGLE MTP. CUDA was never affected because those backends take q_offset= and slice + internally. The GLM52_P1V2 edit sites are byte-identical on v0.5.15.post1 and + v0.5.16; the later GLM52_P1V3 anchor has only been re-read against v0.5.15.post1. + call_chain: + - "sglang/srt/layers/attention/dsa_indexer.py :: forward (aiter/HIP branch)" + - "aiter paged-MQA logits kernel — sizes its output from q_fp8.shape[0]" + - "fast_topk_v2 / topk_transform — asserts lengths.size(0) == B" + symptom_signature: "Expected lengths.size(0) == B to be true, but got false" + silent: false + +verification: + date: 2026-08-03 + hardware: 8x MI355X (gfx950) and 2x 8x MI325X (gfx942) + software: sglang v0.5.15.post1 and v0.5.16, ROCm 7.2.0 + workload: GLM-5.2-FP8, DSA attention, --enable-dp-attention with EAGLE MTP + result: >- + Crash at first batch (gfx950) / conc>1 (gfx942) becomes a clean run. Captured the + inverted direction live with the patch's own SGLANG_DEBUG_DSA_ROWS=1 hook. + notes: >- + Do NOT reintroduce a `0 < q_offset` lower bound on the trim arm — an earlier + revision had one and it made fast_topk_v2 assert on exactly the idle ranks. + +scope_limits: >- + The GLM52_P1V3 anchor (the bare topk_transform call) has been re-read against + v0.5.15.post1 only, not v0.5.16. + +open_actions: + - action: "Get sglang#33059 reviewed, or converge it onto #32762 if that lands first." + owner: dorado269 + blocked_on: upstream review diff --git a/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py index 2e318135..e416ec22 100644 --- a/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py +++ b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py @@ -105,10 +105,29 @@ (host VA is not the device VA on that platform). #32503 / #32792 (both OPEN) add Intel XPU HiCache and will touch the same dict -- a merge conflict risk for this anchor, not a fix for it. - own PR NONE. Not filed. It should be: upstream main is affected, the - one-line form matches an already-merged precedent (#23361), and - the device-pointer measurements above are the evidence. Blocked - only on someone opening it. + own PR #33968 (OPEN, DRAFT, filed 2026-08-07 by dorado269) -- + "[ROCm] Fix HiCache host-pool allocator: hipHostRegister's + device pointer is not the host VA". Same fix as this script, + minus the GLM52_ROCM_HOST_ALLOC bytecode marker. + + STILL A DRAFT because the MI355X cluster was unreachable when it + was opened, so it has NOT been re-run against the original fault + on hardware that reproduces it. What was checked, on MI300X: the + dispatch resolves to alloc_with_pin_memory for every key incl. a + torch.device one, a real 8 MiB allocation comes back pinned with + devPtr == host, and nothing is removed from the module. Do not + read "PR open" as "fixed upstream". + + CORRECTION This header used to call the fix a one-line dispatch override in + the shape of #23361. It is not, and cannot be: + * get_device() returns "cuda" on ROCm (measured), so HIP has no + key of its own to add the way "npu"/"musa" do; and + * memory_pool_host.py:768 and :1257 key the table with a + torch.device OBJECT. torch.device("cuda:0") is NOT + dict-key-equal to "cuda" (measured: different hash), so those + two pools always resolve through the defaultdict default. + That is why both the key and the default move below. A fix that + only added a "cuda" key would silently miss those two pools. Idempotent and self-locating. Run inside the container, then delete stale .pyc. """ diff --git a/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml new file mode 100644 index 00000000..3141a489 --- /dev/null +++ b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.upstream.status.yaml @@ -0,0 +1,228 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-07 +verified_by: + - gh-pr-issue-state + - gh-search + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_host_alloc.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + opt_in_flag: null + +target: + library: sglang + component: hicache + repo: sgl-project/sglang + files: + - python/sglang/srt/mem_cache/pool_host/common.py + versions: + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: gfx950 — where the fault was actually observed. + - surface: Dockerfile.sglang.gfx942 + ref: v0.5.16 + commit: fdebc938f7f4 + pinned_ref_on_main: false + release_branch: release/v0.5.16 + images: + - tag: lmsysorg/sglang:v0.5.16-rocm720-mi30x + digest: null + digest_source: unresolved + note: >- + Carried preventively here. MI300X measures the host and device addresses + EQUAL, so this image has no fault to fix. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05 (contents API, pool_host/common.py): + ALLOC_MEMORY_FUNCS still overrides only "npu" and "musa", with no HIP entry, over a + defaultdict whose default is alloc_with_host_register. Main is affected. This is a + source read, not a search miss. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: op + required_to_run: true + evidence: >- + Without it the first kvd write-back aborts the process with a GPU memory access + fault. Reproduced standalone with the exact write-back kernel. + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: op + required_to_run: false + evidence: >- + It changes the allocator here too, but preventively: MI300X (amdgpu 6.14.14, ROCm + 7.2.0) measures hipHostRegister's device pointer EQUAL to the host VA, so the + fault does not occur on this arch. Do not read this row as evidence the fault was + seen on gfx942. + +alive_because: + reason: upstream-pr-open + detail: >- + Ours is now upstream as sglang#33968 (DRAFT, opened 2026-08-07). No third-party + issue or PR exists. Upstream main is affected, established by reading the file, so + this is not a case of "probably already fixed". Was `no-upstream-pr` until #33968. + consumers: + - Dockerfile.sglang (kvd / hierarchical cache on gfx950 — required) + - Dockerfile.sglang.gfx942 (preventive) + drop_when: >- + A base sglang routes HIP to alloc_with_pin_memory in ALLOC_MEMORY_FUNCS. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-08-03 + commit: 9380f07 + subject: "fix(glm5.2): agentic-serving fixes for DSA indexer, ROCm hicache and PD/kvd wiring" + last_modified: + date: 2026-08-04 + commit: 71fe42c + subject: "fix(sglang-rocm): make hicache survive its first write-back on gfx942" + reason: >- + Carried onto the gfx942 image alongside the staged write-back patch, and its + header corrected: the fault is gfx950-only so far, so this one is preventive on + MI300X rather than a fix for an observed crash. + +upstream_issues: null + +upstream_prs: + - ref: sgl-project/sglang#33968 + url: https://github.com/sgl-project/sglang/pull/33968 + title: "[ROCm] Fix HiCache host-pool allocator: hipHostRegister's device pointer is not the host VA" + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: dorado269 + ours: true + same_approach: true + approach_note: >- + Opened 2026-08-07 as a DRAFT. Same fix as this patch, minus the local + GLM52_ROCM_HOST_ALLOC bytecode marker. Adapting it corrected the record's own + framing twice: get_device() returns "cuda" on ROCm, so HIP cannot take a key of + its own the way #23361's "musa" did; and memory_pool_host.py:768 and :1257 key + the table with a torch.device OBJECT, which is not dict-key-equal to the string + "cuda" (measured: different hash), so those two pools always resolve through the + defaultdict default. Changing only the key would have missed them — which is why + both the key and the default move. + requested_action: >- + Draft until re-validated on gfx950. The MI355X cluster was unreachable when it was + opened, so the write-back repro in the PR body is historical, not from that branch. + in_pinned_base: false + +related_refs: + - ref: sgl-project/sglang#23361 + url: https://github.com/sgl-project/sglang/pull/23361 + kind: pr + title: "[MUSA][19/N] Support HiCache with pin_memory allocator" + state: MERGED + state_reason: fixed + author: yafengio + relevance: shape-we-copied + note: >- + The same one-line dispatch override for the same reason, for MUSA. This patch is + written in its shape, which is also the argument that upstream would accept a HIP + entry. + - ref: sgl-project/sglang#32503 + url: https://github.com/sgl-project/sglang/pull/32503 + kind: pr + title: "[XPU] Enable HiCache support on Intel XPU" + state: OPEN + state_reason: null + author: kpjeeja + relevance: anchor-collision + note: Touches the same ALLOC_MEMORY_FUNCS dict. Expect a conflict, not a fix. + - ref: sgl-project/sglang#32792 + url: https://github.com/sgl-project/sglang/pull/32792 + kind: pr + title: "[XPU]Enable HiSparse hierarchical sparse KV cache on Intel XPU" + state: OPEN + state_reason: null + author: Amrutha-M05 + relevance: anchor-collision + note: Same dict again. + +problem: + what: >- + sglang allocates every hierarchical-cache host pool with anonymous mmap + + hipHostRegister, then stores the resulting HOST virtual addresses in a device-side + pointer table that a GPU kernel dereferences. On ROCm those host VAs are not + dereferenceable from the device. + why: >- + On CUDA, cudaHostRegister maps the pages at the SAME address in the device address + space, so a host VA is directly usable from a kernel. On ROCm, hipHostRegister maps + them at a DIFFERENT device address which you must obtain via + hipHostGetDevicePointer. Measured on gfx950 / ROCm 7.2.0 with an 8 MiB buffer: + pin_memory gives host==devPtr, while mmap+hipHostRegister gives + host=0x7d3bee790000 devPtr=0x7d3bede00000 — and every register-flag variant + (Mapped, Portable|Mapped, MAP_PRIVATE) also differs. gfx950 reports xnack-, so + there is no page-migration path to paper over it. + how: >- + Register ROCm in ALLOC_MEMORY_FUNCS to use alloc_with_pin_memory — torch + pin_memory=True, hipHostMalloc underneath — which returns memory whose device + pointer IS the host pointer. Exactly what the existing "npu" and "musa" entries do, + for the same reason. This is the smallest change that makes the existing + pointer-table design correct on ROCm; translating every host pointer through + hipHostGetDevicePointer is the right upstream shape but touches roughly ten call + sites and is not something to apply blind to a running container. + before_fix: >- + "Memory access fault by GPU node-2 (Agent handle: ...) on address . Reason: + Unknown." on the first kvd write-back. The fault address equals the host pointer + exactly, which is the fingerprint. + after_fix: >- + Standalone repro of the exact write-back kernel — 78 layers, 7.33 GB host indexer + buffer, page ranges head / tail / last — goes from a fault on every mmap variant to + ALL OK with pin_memory. + context: >- + gfx950 / MI355X, ROCm 7.2.0, sglang hierarchical cache with a DSA model + (DSAIndexerPoolHost). MI300X measures the two addresses equal, so the fault is + gfx950-only so far. Cost of the fix: the hugepage path in alloc_mmap + (SGLANG_HUGEPAGE_SIZE) is bypassed on ROCm; that env var is unset here. + call_chain: + - "sglang/srt/mem_cache/pool_host/common.py :: ALLOC_MEMORY_FUNCS[device] -> alloc_with_host_register" + - "alloc_mmap(dims, dtype) then cudaHostRegister(buffer.data_ptr(), n, 0)" + - "memory_pool_host.py :: DSAIndexerPoolHost.init_kv_buffer builds index_k_data_ptrs from host data_ptr()s on the GPU" + - "transfer_kv_all_layer_mla(dst_layers=self.index_k_data_ptrs, ...) — kernel dereferences a host VA" + symptom_signature: "Memory access fault by GPU node-N ... on address " + silent: false + +verification: + date: 2026-08-03 + hardware: MI355X (gfx950, xnack-) and MI300X (amdgpu 6.14.14) for the negative control + software: ROCm 7.2.0, sglang v0.5.15.post1 + workload: standalone repro of the hicache write-back kernel, 78 layers, 7.33 GB host indexer buffer + result: mmap+hipHostRegister faults on every variant; pin_memory is clean on all page ranges + notes: >- + The address-equality measurement is the cheap way to tell whether a given arch is + affected — run it before assuming a new GPU needs this. + +scope_limits: >- + Fixes the allocator, not the design. The pointer tables still hold raw host pointers; + any future pool that builds a device-side table the same way is correct only because + pin_memory happens to return matching addresses. + +open_actions: + - action: >- + Re-validate sglang#33968 on gfx950 and take it out of draft. The equivalence and + scope of the PR's diff were checked on MI300X (dispatch for every key including a + torch.device one, a real 8 MiB allocation with its device pointer measured, module + exports), but MI300X is the NEGATIVE control — it cannot reproduce the fault. Run + the write-back repro on MI355X against that exact branch. + owner: dorado269 + blocked_on: MI355X cluster access (unreachable 2026-08-07) diff --git a/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.upstream.status.yaml b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.upstream.status.yaml new file mode 100644 index 00000000..c433c436 --- /dev/null +++ b/deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.upstream.status.yaml @@ -0,0 +1,238 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/sglang_rocm/patch_hicache_rocm_staged_write_back.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.sglang + - Dockerfile.sglang.gfx942 + opt_in_flag: null + +target: + library: sglang + component: hicache + repo: sgl-project/sglang + files: + - python/sglang/srt/mem_cache/pool_host/mla.py + versions: + - surface: Dockerfile.sglang.gfx942 + ref: v0.5.16 + commit: fdebc938f7f4 + pinned_ref_on_main: false + release_branch: release/v0.5.16 + images: + - tag: lmsysorg/sglang:v0.5.16-rocm720-mi30x + digest: null + digest_source: unresolved + note: >- + The base that actually needs this. #28534 landed the HIP enablement after + v0.5.15.post1 was cut, so this defect exists only from v0.5.16 onward. + - surface: Dockerfile.sglang + ref: v0.5.15.post1 + commit: 0b3bb0cbe318 + pinned_ref_on_main: false + release_branch: release/v0.5.15 + images: + - tag: lmsysorg/sglang:v0.5.15.post1-rocm720-mi35x + digest: null + digest_source: unresolved + note: >- + No pool_host/mla.py exists at this tag — MLA and MHA had not yet moved out of + memory_pool_host.py — so the patch tolerates the absent file and no-ops. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: pool_host/mla.py still says + `self.can_use_write_back_jit = (_is_cuda or _is_hip) and can_use_write_back_jit_kernel(...)` + while DSAIndexerPoolHost and both DeepSeekV4 host pools still gate on _is_cuda alone. + The two gates still disagree on ROCm, so main is affected. + +applies_to: + - engine: sglang + surface: Dockerfile.sglang.gfx942 (gfx942 / MI325X, v0.5.16) + effect: op + required_to_run: true + evidence: >- + Without it the v0.5.16 gfx942 base kills the prefill scheduler on the first reused + prefix, so the image cannot run kvd at all. Measured on MI300X with GLM-5.2-FP8, + DSA, dp8 and the default hicache flags. + - engine: sglang + surface: Dockerfile.sglang (gfx950 / MI355X, v0.5.15.post1) + effect: no-op + required_to_run: false + evidence: >- + Nothing to fix on this base: every can_use_write_back_jit gate at v0.5.15.post1 is + still _is_cuda (MHA, MLA, both V4 pools and DSAIndexerPoolHost, all in + memory_pool_host.py), and _is_hip appears only in the kernel import guard, so the + group's AND and its anchor agree on False. pool_host/mla.py does not exist and the + script exits 0 on that. + +alive_because: + reason: upstream-pr-open + detail: >- + The upstream repair is #30350, which is OPEN with CHANGES_REQUESTED. Note the + inversion: the MERGED PR here (#28534) is what INTRODUCED the disagreement, so this + patch does not drop on a merge — it drops on #30350. #28534's own reasoning argues + for the opposite repair, teaching the remaining pools the JIT rather than gating MLA + down, so upstream will likely close this differently than we did. + consumers: + - Dockerfile.sglang.gfx942 (kvd / hierarchical cache — required to serve at all) + drop_when: >- + A base sglang stops gating DSAIndexerPoolHost on _is_cuda alone — i.e. #30350 or an + equivalent lands. check_group_still_poisoned() tests that precondition and refuses. + drop_signal: precondition-check-refuses + silent_misapply_risk: >- + Anchor drift cannot be the signal here. #30350 never touches pool_host/mla.py, so + our anchor would keep matching and the patch would keep applying on top of the fix. + That is not a crash — both gates read False again — but it silently forfeits the + staged kernel #30350 enables. Hence the precondition check rather than an anchor + check: once DSAIndexerPoolHost stops gating on _is_cuda alone, the script exits 1 and + tells the operator to drop it. + +history: + born: + date: 2026-08-04 + commit: 71fe42c + subject: "fix(sglang-rocm): make hicache survive its first write-back on gfx942" + last_modified: + date: 2026-08-04 + commit: 71fe42c + subject: "fix(sglang-rocm): make hicache survive its first write-back on gfx942" + reason: >- + Written with check_group_still_poisoned() from the start, precisely because the + upstream fix in flight does not touch our anchor and a plain anchor check would + let the patch outlive its reason silently. + +upstream_issues: null + +upstream_prs: + - ref: sgl-project/sglang#30350 + url: https://github.com/sgl-project/sglang/pull/30350 + title: Add HiCache JIT test and benchmark for ROCm/HIP CI support + state: OPEN + review_decision: CHANGES_REQUESTED + merged_at: null + author: Emmanuel0612 + ours: false + same_approach: false + approach_note: >- + Opposite direction, and better. It adds _is_cuda_alike = _is_cuda or _is_hip and + flips the three CUDA-only gates (DSAIndexerPoolHost, DeepSeekV4PagedHostPool, + DeepSeekV4StateHostPool) so the group AND stops reading False on ROCm — including + the V4 stack our patch does not cover. It also teaches staged_write_back.cuh to + accept kDLROCM/kDLROCMHost, which is the TensorMatcher check that emits our crash, + and adds an AMD CI lane. We gate MLA down instead because that is the change that + is safe to apply to a running container; #30350 is the fix that should land. + requested_action: >- + HaiShaw approved on 2026-07-08 then requested changes on 2026-07-13. The conflict + that accompanied the request was cleared the same day and the branch last moved + 2026-07-16, but no re-review was requested, so the block still stands. Our MI300X + datapoint was posted to the thread by llying-001 on 2026-08-04 — the thread's only + non-MI355X evidence, and what this row previously listed as our open action. + in_pinned_base: false + - ref: sgl-project/sglang#28534 + url: https://github.com/sgl-project/sglang/pull/28534 + title: "[AMD] Enable JIT staged HiCache write-back and fix CPU-index crash" + state: MERGED + review_decision: REVIEW_REQUIRED + merged_at: 2026-07-09 + author: AMD-yanfeiwang + ours: false + same_approach: false + approach_note: >- + This is the PR that CREATED the defect, not one that fixes it. It added the HIP + enablement to pool_host/mla.py and pool_host/mha.py and aligned + cache_controller.py and memory_pool_host.py, but never touched the three CUDA-only + pools that share a HostPoolGroup with them. A merged PR that a local patch exists + to undo. + requested_action: null + in_pinned_base: true + +related_refs: null + +problem: + what: >- + Two gates decide one thing and disagree on ROCm, so the hicache controller puts the + destination indices on the GPU while the MLA pool launches the JIT kernel that + requires them on the host. The first write-back kills the scheduler. + why: >- + pool_host/mla.py opts HIP into the staged write-back JIT + (`(_is_cuda or _is_hip) and can_use_write_back_jit_kernel(...)`), and pool_host/mha.py + carries the identical enablement — #28534 did both. But the pools that can share a + HostPoolGroup with them do not: DSAIndexerPoolHost, which any DSA model such as + GLM-5.2 always instantiates alongside the MLA pool, plus DeepSeekV4PagedHostPool and + DeepSeekV4StateHostPool, still gate on _is_cuda alone and read False on ROCm. + HostPoolGroup ANDs the flag over its entries, so one CUDA-only member makes the + group flag False while its ANCHOR entry — the MLA pool — still says True. The two + then feed different decisions: hybrid_cache_controller.start_writing() reads the + GROUP flag, concludes the JIT will not run, and calls move_hybrid_indices() so the + destination indices end up on the GPU; HostPoolGroup.backup_from_device_all_layer() + delegates to the ANCHOR pool, whose own flag is True, so the JIT launches anyway. + how: >- + Make the anchor agree with the group: gate the staged JIT on CUDA only, exactly as + the three CUDA-only pools already spell it. The MLA pool then falls through to + transfer_kv_all_layer_mla_lf_pf — the non-JIT kernel in the same branch, which + asserts its destination indices ARE on the GPU, which is where the controller just + put them. Both gates read False on ROCm and the two decisions match. + before_fix: >- + "tvm.error.InternalError: Tensor match failed for Tensor<1152>[strides=<1>, + dtype=int64, device=rocm:0] at jit_kernel/csrc/kvcacheio/staged_write_back.cuh:248 - + Root cause: Device value [rocm:0] not in the allowed options: [cpu, rocm_host]" and + "Subprocess scheduler_0 crashed with exit code -3", on the first write-back after the + first reusable prefix. + after_fix: >- + The gfx942 image serves kvd. On any model that reaches this crash the fix costs + nothing that worked before: DSAIndexerPoolHost is CUDA-gated, so the group flag was + already False on ROCm for every DSA model and the staged JIT was unreachable + regardless. + context: >- + MI300X (gfx942), ROCm 7.2.0, sglang v0.5.16, GLM-5.2-FP8, DSA attention, dp8, with + --hicache-mem-layout page_first --hicache-io-backend kernel (both defaults). Not + specific to kvd: any hierarchical cache on this path dies the moment it writes a + page back, so the engine survives startup and then crashes on the first real request. + call_chain: + - "hybrid_cache/hybrid_cache_controller.py :: start_writing() reads the GROUP flag (False on ROCm) -> move_hybrid_indices() puts dst indices on the GPU" + - "HostPoolGroup.backup_from_device_all_layer() delegates to the ANCHOR pool" + - "mem_cache/pool_host/mla.py :: backup_from_device_all_layer reads its OWN flag (True) -> launches the JIT" + - "jit_transfer_hicache_all_layer_mla_staged_lf_pf -> staged_write_back.cuh:248 TensorMatcher rejects device=rocm:0" + symptom_signature: "Device value [rocm:0] not in the allowed options: [cpu, rocm_host]" + silent: false + +verification: + date: 2026-08-04 + hardware: MI300X (gfx942) + software: sglang v0.5.16 and current upstream main copies, ROCm 7.2.0 + workload: GLM-5.2-FP8, DSA, dp8, hicache page_first + kernel io-backend + result: >- + Exercised against the v0.5.16 AND current-main copies of both files on throwaway + trees. Stock: applies, exit 0. Re-run: "already applied", exit 0. #30350 simulated by + flipping the three CUDA-only gates: refuses, exit 1, with mla.py byte-identical to + pristine. pool_host/mla.py removed (the v0.5.15.post1 shape): tolerated, exit 0. + Anchor or pool renamed: exit 1. + notes: >- + The five-case exercise is the point — a patch whose drop signal is a precondition + check is only trustworthy if the refusal path is tested. + +scope_limits: >- + This patch gates pool_host/mla.py ONLY. DSAIndexerPoolHost is not the only CUDA-only + member that can poison the group's AND: build_deepseek_v4_hicache_stack puts + DeepSeekV4PagedHostPool in a group anchored by LogicalHostPool, whose flag is + unconditionally True, so expect the same crash on a V4 hicache stack on gfx942. No V4 + stack runs on this branch, so that gate would be untested; the script's SCOPE section + records it for whoever gets there. #30350 does cover the V4 pools. + +open_actions: + - action: >- + Nudge #30350 for re-review — it is CHANGES_REQUESTED with the conflict long cleared + and our MI300X datapoint already on the thread. Landing it retires this patch and + also covers the V4 pools we do not. + owner: llying-001 + blocked_on: HaiShaw re-review diff --git a/deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.upstream.status.yaml b/deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.upstream.status.yaml new file mode 100644 index 00000000..749642d1 --- /dev/null +++ b/deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.upstream.status.yaml @@ -0,0 +1,129 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm-dsv4/patch_aiter_flydsl_moe_memref_bufres.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + opt_in_flag: BUILD_AITER=1 (build ARG; default 1 — the patch targets the aiter this step builds) + +target: + library: aiter + component: flydsl-moe + repo: ROCm/aiter + files: + - aiter/ops/flydsl/kernels/moe_gemm_2stage.py + versions: + - surface: Dockerfile.vllm (build_aiter_rocm.sh) + ref: v0.1.16.post1 + commit: null + pinned_ref_on_main: null + release_branch: null + unpinned_risk: >- + AITER_GIT_REF pins a tag, not a commit, and aiter is built in-container so no digest + exists. The tag is the pin. Note the patch's own header records verification against + aiter 0.1.16.post2 / flydsl 0.2.0, one patch level ABOVE the tag the Dockerfile now + builds — the two have drifted and nobody re-verified. + images: [] + +upstream_main_affected: + value: null + evidence: >- + Not established. aiter main was not read, and no search hit was found on either the + symptom or the symbol across vLLM and ROCm/aiter as of 2026-08-05. Recorded as null + rather than guessed: this is a fast-moving kernel library where a version-specific + fx.ptrtoint behaviour may simply have changed without a traceable fix. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (vllm-dsv4-patches loop) + effect: unverified + required_to_run: false + evidence: >- + Load-bearing for Kimi-K2.6 int4 W4A16 MoE when the anchors are present, and a + documented no-op for DSv4 MXFP4. Marked unverified because the patch was verified + against aiter 0.1.16.post2 while the Dockerfile now builds v0.1.16.post1, and nobody + has confirmed the anchors still match what that tag produces. + +alive_because: + reason: no-upstream-pr + detail: >- + No upstream issue and no upstream PR, ours included. Searched again on 2026-08-05 across + vLLM and ROCm/aiter, on the symptom and on the symbol. + consumers: + - Dockerfile.vllm, for Kimi-K2.6 int4 W4A16 MoE + drop_when: >- + aiter's flydsl accepts memref-typed pointer arguments in fx.ptrtoint, or the built aiter + version no longer contains the buffer-resource helper this reverts. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + The patch REVERTS gemm1/gemm2 to an older buffer_ops formulation. If aiter later changes + those kernels for reasons unrelated to the memref problem, the anchors could still match + while the revert now discards a genuine improvement. The marker guard prevents double + application but cannot detect that case. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged, while AITER_GIT_REF moved underneath it. That drift is the open action below. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + vLLM passes tensors into aiter's flydsl 2-stage MoE GEMM, which arrive as flydsl memrefs, + and aiter 0.1.16's fx.ptrtoint rejects memrefs — the kernel build dies with an MLIRError. + why: >- + The gemm1/gemm2 buffer-resource construction was changed to go through fx.ptrtoint, which + assumes a raw pointer. A memref is not one, so the conversion raises during MLIR + construction rather than at a type boundary where it would be obvious. + how: >- + Revert the gemm1/gemm2 buffer-resource builds to the memref-friendly buffer_ops calls + (create_buffer_resource / extract_base_index). gemm3 is deliberately untouched. + before_fix: MLIRError crash in the 2-stage MoE GEMM on Kimi-K2.6 int4 W4A16. + after_fix: The MoE GEMM builds and runs. No-op for DSv4 MXFP4, which does not take this path. + context: >- + Kimi-K2.6 int4 W4A16 MoE on ROCm. Verified against aiter 0.1.16.post2 with flydsl 0.2.0 on + a vLLM dev748 base — note that BOTH of those differ from what Dockerfile.vllm builds today. + call_chain: + - "vLLM MoE layer passes tensors to the aiter flydsl 2-stage GEMM" + - "aiter/ops/flydsl/kernels/moe_gemm_2stage.py :: _ptr_buffer_resource -> fx.ptrtoint(ptr)" + - "ptr is a flydsl memref -> MLIRError during MLIR construction" + symptom_signature: MLIRError from fx.ptrtoint in moe_gemm_2stage + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm + software: aiter 0.1.16.post2, flydsl 0.2.0, vLLM dev748 + workload: Kimi-K2.6 int4 W4A16 MoE + result: MLIRError cleared. + notes: >- + This verification is now stale relative to the Dockerfile — recorded as-is rather than + restated against v0.1.16.post1, which would be a claim nobody has checked. + +scope_limits: gemm1 and gemm2 only; gemm3 is left alone. + +open_actions: + - action: >- + Re-verify against the aiter version Dockerfile.vllm actually builds (v0.1.16.post1). The + patch was verified on 0.1.16.post2 and the two have drifted apart. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.upstream.status.yaml b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.upstream.status.yaml new file mode 100644 index 00000000..5fdfb735 --- /dev/null +++ b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.upstream.status.yaml @@ -0,0 +1,134 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_hybrid_blocksize.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + opt_in_flag: null + +target: + library: vllm + component: moriio + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. The + patch's own header records verification against vllm 0.23.x, two minor versions back. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: null + evidence: >- + Not established — main was not read for this one and no search hit exists. MoRIIO attracts + little third-party traffic, so absence of a report says little either way. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (vllm-dsv4-patches loop) + effect: unverified + required_to_run: false + evidence: >- + Verified load-bearing on vllm 0.23.x for DSv4-Pro MoRIIO 1P1D — without it the prefill + worker dies at KV registration. Not re-verified on the v0.25.1 base, which refactored + the MoRIIO connector enough to retire three sibling patches, so op-vs-no-op here is + genuinely open. + +alive_because: + reason: no-upstream-pr + detail: >- + Nothing upstream and nothing of ours. All four DSv4 MoRIIO patches were searched again on + 2026-08-05 across vLLM and ROCm/aiter, on the symptom and on the symbol, with no hits. + consumers: + - Dockerfile.vllm, for DeepSeek-V4 MoRIIO PD + drop_when: >- + A base vLLM drops the global block_size equality check, or the v0.25.1+ refactor is + confirmed to have removed the anchors (in which case the patch is already a no-op). + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + This patch REMOVES a check and DEMOTES a raise to a debug log. If upstream later adds a + genuinely load-bearing block-size validation at the same site, the patch would silence it. + The narrow anchors make that unlikely rather than impossible. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged across two base bumps. Its three GLM-5.1 siblings were deleted on the v0.25.1 + move for having become no-ops; this one was kept without being re-verified. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + A global block_size equality check in the MoRIIO connector kills the prefill worker at KV + registration for DeepSeek-V4, which legitimately registers per-layer caches with different + block sizes. + why: >- + The connector asserts `first_geometry.block_size == self.block_size` and raises on any + per-layer mismatch, but DSv4's KV geometry is hybrid by design — c128a sparse layers use + block size 2 while SWA and state layers use 256. The check is spurious because the actual + offset arithmetic already consults the per-layer self.block_lens dict, so nothing + downstream depends on the global equality. + how: >- + Drop the global assert and demote the per-layer mismatch raise to a debug log, leaving the + per-layer offset path untouched. + before_fix: The prefill worker dies at KV registration, so DSv4 MoRIIO PD never starts. + after_fix: DSv4-Pro serves over MoRIIO 1P1D. + context: >- + DeepSeek-V4-Pro with MoRIIOConnector 1P1D on ROCm; verified on vllm 0.23.x. Only hybrid + per-layer block-size models reach the check. + call_chain: + - "moriio_connector.py :: register_kv_caches" + - "assert first_geometry.block_size == self.block_size (global check)" + - "per-layer mismatch raise" + - "prefill worker dies before serving" + symptom_signature: block_size assertion at MoRIIO KV registration on a hybrid-KV model + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm 1P1D + software: vllm 0.23.x ROCm, MoRIIOConnector + workload: DeepSeek-V4-Pro MoRIIO PD + result: Prefill worker survives registration and PD serves. + notes: Verification predates the v0.25.1 base by two minor versions. + +scope_limits: >- + Relaxes validation only; it does not make any transfer path block-size aware, because the + per-layer path already was. + +open_actions: + - action: >- + Re-verify all four DSv4 MoRIIO patches on the v0.25.1 base. The refactor that landed + moriio_layout.py already turned three sibling patches into no-ops, and these four were + carried across on the assumption they still apply. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.upstream.status.yaml b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.upstream.status.yaml new file mode 100644 index 00000000..a6366227 --- /dev/null +++ b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.upstream.status.yaml @@ -0,0 +1,122 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_noncontig_register.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + opt_in_flag: null + +target: + library: vllm + component: moriio + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_engine.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref. The patch's own header + records verification against vllm 0.23.x. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: null + evidence: Not established — main was not read, and no search hit exists as of 2026-08-05. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (vllm-dsv4-patches loop) + effect: unverified + required_to_run: false + evidence: >- + Verified load-bearing on vllm 0.23.x for DSv4-Pro 1P1D. Not re-verified on v0.25.1. + +alive_because: + reason: no-upstream-pr + detail: >- + Nothing upstream and nothing of ours. Worth noting the fix is not novel — sglang's mori + connector already registers this way, so the shape is established; only vLLM's MoRIIO + lacks it. + consumers: + - Dockerfile.vllm, for DeepSeek-V4 MoRIIO PD + drop_when: >- + A base vLLM's MoRIIO engine registers non-contiguous tensors by storage span, as sglang's + mori connector already does. + drop_signal: self-guard-marker-skips + silent_misapply_risk: null + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged across two base bumps. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + MoRI's register_torch_tensor rejects DeepSeek-V4's KV cache because it is a non-contiguous + view, and the obvious workaround — calling .contiguous() — would be silently wrong. + why: >- + DSv4's fp8_ds_mla KV cache is a 576-byte-aligned non-contiguous view over a larger buffer. + register_torch_tensor forces contiguity and raises. .contiguous() would copy to a NEW + address, so the registered memory would no longer be the buffer the model forward writes + into — registration would succeed and transfers would move stale data. + how: >- + For a non-contiguous tensor, register the storage span through the low-level + register_memory(data_ptr, storage_nbytes - storage_offset_bytes, dev, loc), using the same + base that _compute_block_transfer_offsets already uses so the offset arithmetic stays + consistent. Contiguous tensors keep the original path. This mirrors sglang's mori connector. + before_fix: >- + register_torch_tensor raises on the non-contiguous DSv4 KV view, so registration fails. + after_fix: DSv4-Pro registers its KV cache and MoRIIO 1P1D serves. + context: >- + DeepSeek-V4-Pro (fp8_ds_mla, 576B-aligned non-contiguous KV view) with MoRIIO on ROCm, + verified on vllm 0.23.x. Contiguous fp16/bf16 caches are unaffected. + call_chain: + - "moriio_engine.py :: register_torch_tensor(kv_view)" + - "non-contiguous 576B-aligned fp8_ds_mla view -> raises" + - "(.contiguous() would copy to a new address, detaching from the forward's buffer)" + symptom_signature: register_torch_tensor rejecting a non-contiguous tensor at MoRIIO registration + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm 1P1D + software: vllm 0.23.x ROCm, MoRIIOConnector + workload: DeepSeek-V4-Pro MoRIIO PD + result: Registration succeeds against the live forward buffer. + notes: >- + The .contiguous() trap is the transferable lesson: it would have "worked" and moved stale + KV, which is far worse than the exception. + +scope_limits: Non-contiguous registration only; contiguous tensors keep the stock path. + +open_actions: + - action: Re-verify on the v0.25.1 base together with the other three DSv4 MoRIIO patches. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.upstream.status.yaml b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.upstream.status.yaml new file mode 100644 index 00000000..de7505e6 --- /dev/null +++ b/deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.upstream.status.yaml @@ -0,0 +1,128 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm-dsv4/patch_moriio_dsv4_sparse_backend.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + opt_in_flag: null + +target: + library: vllm + component: moriio + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref. The patch's own header + records verification against vllm 0.23.x. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: null + evidence: Not established — main was not read, and no search hit exists as of 2026-08-05. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (vllm-dsv4-patches loop) + effect: unverified + required_to_run: false + evidence: >- + Verified load-bearing on vllm 0.23.x for DSv4-Pro 1P1D. Not re-verified on v0.25.1. The + anchor is matched verbatim so drift skips rather than mis-patches. + +alive_because: + reason: no-upstream-pr + detail: Nothing upstream and nothing of ours. + consumers: + - Dockerfile.vllm, for DeepSeek-V4 sparse MLA MoRIIO PD + drop_when: >- + A base vLLM's ROCm backend selector recognises fp8_ds_mla sparse MLA, or MoRIIO stops + routing a handshake tag through the full backend-selection path. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + The patch hardcodes backend_name for one model family. If a future base changes what + backend_name means — from a P/D handshake tag to something the transfer path actually + dispatches on — a hardcoded value would be wrong rather than merely cosmetic. That + assumption is the load-bearing part of this fix and should be re-checked on a base bump. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged across two base bumps. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + MoRIIO asks the generic ROCm backend selector for a backend name purely to fill a P/D + handshake tag, and for DSv4 sparse MLA that query raises and kills the prefill worker. + why: >- + The generic selector returns ROCM_AITER_MLA_SPARSE, which does not support fp8_ds_mla, so + get_attn_backend() raises. The irony is that the return value is only used as a handshake + STRING — nothing dispatches on it — so a fatal query is being made for a cosmetic purpose. + how: >- + For DSv4 sparse MLA specifically (use_mla and hf_config.index_topk and cache_dtype + fp8_ds_mla), skip the generic query and set backend_name directly to + ROCM_FLASHMLA_SPARSE_DSV4. Every other model keeps the original path. The old code is + matched verbatim so drift skips loudly rather than mis-patching. + before_fix: >- + get_attn_backend() raises during MoRIIO setup and the prefill worker dies, so DSv4 sparse + MLA PD never starts. + after_fix: DSv4-Pro serves over MoRIIO 1P1D with the correct handshake tag. + context: >- + DeepSeek-V4-Pro sparse MLA (fp8_ds_mla, index_topk) with MoRIIOConnector 1P1D on ROCm, + verified on vllm 0.23.x. + call_chain: + - "moriio_connector.py :: setup -> get_attn_backend(...) to obtain backend_name" + - "generic ROCm selector returns ROCM_AITER_MLA_SPARSE, which lacks fp8_ds_mla -> raises" + - "prefill worker dies" + symptom_signature: get_attn_backend raising for fp8_ds_mla during MoRIIO handshake setup + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm 1P1D + software: vllm 0.23.x ROCm, MoRIIOConnector + workload: DeepSeek-V4-Pro sparse MLA MoRIIO PD + result: Prefill worker survives setup and PD serves. + notes: null + +scope_limits: >- + DSv4 sparse MLA only, gated on use_mla + index_topk + fp8_ds_mla. Other models are untouched. + +open_actions: + - action: Re-verify on the v0.25.1 base together with the other three DSv4 MoRIIO patches. + owner: unassigned + blocked_on: null + - action: >- + Confirm backend_name is still only a handshake tag on the current base — the whole fix + rests on that. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_defer_kv_register.upstream.status.yaml b/deploy/docker/patches/vllm/patch_defer_kv_register.upstream.status.yaml new file mode 100644 index 00000000..7840fc66 --- /dev/null +++ b/deploy/docker/patches/vllm/patch_defer_kv_register.upstream.status.yaml @@ -0,0 +1,162 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_defer_kv_register.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: mooncake-connector + repo: vllm-project/vllm + files: + - vllm/v1/worker/gpu_model_runner.py + - vllm/v1/worker/gpu_worker.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: true + evidence: >- + Weakest evidence of any record here. No upstream issue or PR was found by search, and + main was NOT read for this one, so "affected" is an inference from the defect never + having been reported rather than an observation. Treat as unverified until someone reads + compile_or_warm_up_model on main. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: op + required_to_run: false + evidence: >- + Load-bearing at high GPU utilisation with bare ibv_reg_mr registration; at lower util + the crash it avoids does not fire, so the reordering is inert in effect. + - engine: vllm + surface: deploy/overlay/infera-exec + effect: op + required_to_run: false + evidence: Same patch on the runtime-overlay path. + +alive_because: + reason: no-upstream-pr + detail: >- + Nothing upstream, and nothing of ours. This one is also the hardest to upstream honestly: + what we carry is a reordering that AVOIDS a crash whose mechanism is not fully explained + — the None return from one TP worker's compile_or_warm_up_model is a symptom we routed + around rather than diagnosed. + consumers: + - >- + Dockerfile.vllm and the overlay payload, for high-util Mooncake PD with bare + ibv_reg_mr registration (MOONCAKE_HIP_DMABUF=0) + drop_when: >- + The high-util boot crash is diagnosed properly and fixed at its cause, or a base vLLM + stops aggregating compilation_times in a way that a None can kill. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + The patch reorders when registration happens. If a future base changes the warmup + sequence such that the deferred point is no longer before the engine reports ready, P/D + pairing could break in a way this record's "registration still completes before ready" + claim would no longer cover. The anchors are narrow enough that a rewrite of that + function drops the patch rather than misplacing it, but the invariant is worth re-checking + on a base bump rather than assumed. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + With bare ibv_reg_mr registration, registering the Mooncake KV pool at its normal point — + inside gpu_model_runner.initialize_kv_cache, before compile_or_warm_up_model — trips a + decode-boot crash at high GPU utilisation. + why: >- + Not fully established, and this record should not pretend otherwise. The observable is + that one TP worker's compile_or_warm_up_model returns None, and vLLM's + `max(t.language_model for t in compilation_times)` aggregation then dies on it. Why + early registration makes that worker return None is unexplained; the working theory is + memory pressure from the pinned pool during warmup, which is consistent with it being + util-dependent but is not proven. + how: >- + Move the Mooncake register_kv_caches call to the very END of compile_or_warm_up_model, + after all warmup, cudagraph capture and the post-capture sampler/pooler dummy runs. + gpu_model_runner stashes the caches and the transfer group on self instead of registering; + gpu_worker performs the registration at the end. Registration still completes before the + engine reports ready, so P<->D pairing is unaffected. + before_fix: >- + "AttributeError: 'NoneType' object has no attribute 'language_model'" during decode boot + at high util. + after_fix: >- + Kimi-K2.6 at --gpu-memory-utilization 0.8 boots stably and DeepSeek-V4-Pro output is + correct, confirming the deferral does not break pairing or transfer. + context: >- + Reachable with bare ibv_reg_mr GPU registration (MOONCAKE_HIP_DMABUF=0, the default, + with host-libionic injection) at high --gpu-memory-utilization. Not observed at lower + util. + call_chain: + - "gpu_model_runner.py :: initialize_kv_cache -> kv_transfer_group.register_kv_caches(kv_caches)" + - "gpu_worker.py :: compile_or_warm_up_model — one TP worker returns None" + - "max(t.language_model for t in compilation_times) -> AttributeError on the None" + symptom_signature: "AttributeError: 'NoneType' object has no attribute 'language_model'" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm multi-node PD + software: vLLM with the Mooncake connector, bare ibv_reg_mr registration + workload: Kimi-K2.6 at --gpu-memory-utilization 0.8, and DeepSeek-V4-Pro for correctness + result: >- + Boot crash cleared and DSv4-Pro output correct, which is the check that the deferral did + not break pairing. + notes: >- + Validating correctness as well as boot was the right instinct: a reordering of KV + registration is exactly the kind of change that could boot fine and transfer nothing. + +scope_limits: >- + A workaround, not a fix — it changes WHEN registration happens, not why early registration + breaks warmup. + +open_actions: + - action: >- + Diagnose why one TP worker's compile_or_warm_up_model returns None under early + registration. Without that, this cannot be upstreamed and cannot be safely dropped. + owner: unassigned + blocked_on: null + - action: >- + Read compile_or_warm_up_model on upstream main to replace this record's inferred + upstream_main_affected with an observed one. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.upstream.status.yaml b/deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.upstream.status.yaml new file mode 100644 index 00000000..48bf45d6 --- /dev/null +++ b/deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.upstream.status.yaml @@ -0,0 +1,161 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_mooncake_mamba_unpack.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: mooncake-connector + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref, so no upstream commit + is recorded; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: mooncake_connector.py still has + `conv, _ = cache_or_caches` at line 1678 — the exact line and line number from the + Kimi-K3 traceback. Main is affected. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: op + required_to_run: true + evidence: >- + Kimi-K3 PD cannot register KV without it — every rank raises at once during + registration. + - engine: vllm + surface: deploy/overlay/infera-exec (applied at container start over an unpatched base) + effect: op + required_to_run: true + evidence: >- + This is the surface the patch was written for: the overlay applies patches/vllm/*.py + at container start so an unpatched vendor image can still serve. + +alive_because: + reason: no-upstream-pr + detail: >- + No upstream issue and no upstream PR — including none of ours, for a crash that takes + down every rank. The nearest PR, #47638, edits the very next line and leaves the + arity assumption intact, so it will not fix this even if it merges. + consumers: + - Dockerfile.vllm and the overlay payload, for Kimi-K3 (KDA linear attention) PD + drop_when: >- + Upstream stops destructuring MambaSpec caches as a fixed 2-tuple. The patch then + reports "already applied" or fails to find its anchor and no-ops. + drop_signal: self-guard-marker-skips + silent_misapply_risk: null + +history: + born: + date: 2026-08-02 + commit: 5b31fe0 + subject: "feat(overlay): apply vendor engine patches at container start" + last_modified: + date: 2026-08-02 + commit: 5b31fe0 + subject: "feat(overlay): apply vendor engine patches at container start" + reason: >- + Born with the overlay mechanism it was written for. It was never indexed on the + previous status page, which is the gap this record system exists to prevent. + +upstream_issues: null + +upstream_prs: null + +related_refs: + - ref: vllm-project/vllm#47638 + url: https://github.com/vllm-project/vllm/pull/47638 + kind: pr + title: "[KV Connector] Normalize list/tuple KV cache input in register_kv_caches" + state: OPEN + state_reason: null + author: jianzs + relevance: adjacent-site + note: >- + Edits the very next line — it appends an `elif isinstance(cache_or_caches, (list, + tuple))` branch for Ascend compressed MLA — and leaves the MambaSpec arity + assumption alone. Our three-line anchor survives it, so the two can coexist; do not + read this PR as an incoming fix. + +problem: + what: >- + The Mooncake connector's register_kv_caches() destructures every MambaSpec layer as a + 2-tuple, so any Mamba-family layer that carries a different number of state tensors + kills KV registration on every rank. + why: >- + `conv, _ = cache_or_caches` encodes Mamba2's shape (conv state, SSM state) as if it + were the contract. It is not: MambaSpec.shapes is already a variable-length tuple of + shapes, so the two-tensor assumption belongs to the connector, not the spec. Kimi-K3 + uses KDA linear attention, whose layers carry a different count. + how: >- + Take the first state tensor — exactly what the original code kept — without + constraining how many follow: `cache_list = [cache_or_caches[0]]`. + before_fix: >- + "mooncake_connector.py:1678 in register_kv_caches / conv, _ = cache_or_caches / + ValueError: too many values to unpack (expected 2)", on every rank simultaneously + during KV registration, so PD never starts. + after_fix: Kimi-K3 registers KV and PD starts. + context: >- + Kimi-K3 with KDA linear attention over Mooncake PD. Mamba2 models were never + affected, which is why the assumption survived. + call_chain: + - "mooncake_connector.py :: register_kv_caches" + - "isinstance(layer_spec, MambaSpec) -> conv, _ = cache_or_caches" + - "KDA layer carries != 2 state tensors -> ValueError on every rank" + symptom_signature: "ValueError: too many values to unpack (expected 2)" + silent: false + +verification: + date: 2026-08-02 + hardware: AMD ROCm multi-node PD + software: vLLM v0.25.1 base with the Mooncake connector + workload: Kimi-K3 PD KV registration + result: Registration completes; the all-rank ValueError is gone. + notes: null + +scope_limits: >- + Keeps the first state tensor and no more, which is what the original code did. It does + not teach the connector to transfer the additional KDA state tensors — if those turn + out to matter for correctness, that is a separate and larger fix. + +open_actions: + - action: >- + Submit upstream. Main is affected (confirmed by source read), the fix is three + lines, and there is no competing PR. + owner: unassigned + blocked_on: null + - action: >- + Establish whether dropping the remaining KDA state tensors from the transfer is + correct, or merely non-crashing. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_moriio_pagelen.upstream.status.yaml b/deploy/docker/patches/vllm/patch_moriio_pagelen.upstream.status.yaml new file mode 100644 index 00000000..d0b06b9f --- /dev/null +++ b/deploy/docker/patches/vllm/patch_moriio_pagelen.upstream.status.yaml @@ -0,0 +1,165 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-search + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_moriio_pagelen.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: moriio + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_layout.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: >- + v0.25.1 refactored the MoRIIO connector into moriio_layout.py, which is why + three sibling patches became no-ops and were dropped while this one stayed + load-bearing. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: moriio_layout.py's MLA branch still computes + `block_len = block_size * slot_size_bytes` and `block_stride = stride[0]` from the + tensor shape. Main is affected. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: op + required_to_run: true + evidence: >- + Required for DeepSeek-V4 and GLM-5.1 over MoRIIO PD — without it the output is wrong + while direct prefill is correct. Dockerfile.vllm's step-2 header calls it out by name + as ACTIVE ON v0.25.1. + - engine: vllm + surface: deploy/overlay/infera-exec + effect: op + required_to_run: true + evidence: Same patch on the runtime-overlay path. + +alive_because: + reason: no-upstream-pr + detail: >- + No upstream issue and no upstream PR, ours included, for a silent correctness bug on + main. Searched again on 2026-08-05 across vLLM on both the symptom and the symbol. + MoRIIO attracts little third-party traffic, so read this as genuinely unreported rather + than as a stale search. + consumers: + - Dockerfile.vllm and the overlay payload, for DSv4 and GLM-5.1 MoRIIO PD + drop_when: >- + A base vLLM derives the MLA per-block geometry from spec.page_size_bytes instead of the + tensor shape. + drop_signal: anchor-drift-fails-loudly + silent_misapply_risk: null + +history: + born: + date: 2026-07-23 + commit: 78fd0ff + subject: "Fix MoRIIO MLA page-length transfer geometry on v0.25.1 (DSv4 / GLM-5.1 PD)" + last_modified: + date: 2026-07-23 + commit: 78fd0ff + subject: "Fix MoRIIO MLA page-length transfer geometry on v0.25.1 (DSv4 / GLM-5.1 PD)" + reason: >- + Written against v0.25.1's refactored moriio_layout.py and unchanged since. Its three + GLM-5.1 siblings were dropped in the same era for becoming no-ops on this base; this + one survived because the refactor did not fix the geometry. + +upstream_issues: null + +upstream_prs: null + +related_refs: null + +problem: + what: >- + MoRIIO's MLA transfer path derives the per-block transfer SIZE and STRIDE from the + tensor shape rather than from the authoritative page size, so block-scaled fp8 MLA + caches move the wrong bytes and PD output is silently wrong. + why: >- + get_layer_transfer_geometry's MLA (3-dim) branch computes slot_size_bytes = latent_dim * + element_size, block_len = block_size * slot_size_bytes and block_stride = stride[0]. + The authoritative per-scheduler-block page is spec.page_size_bytes, and the + shape-derived values disagree with it in two independent ways. For DeepSeek-V4 + fp8_ds_mla (UE8M0 block-scaled, 576-byte aligned) the page is PADDED, so page_size_bytes + > block_size*inner*es and the dropped tail is exactly the per-block scale — decode + dequantizes with a stale scale, so facts garble while structure survives. For GLM-5.1 + the cache is laid out per KERNEL block of size 1 (shape[1] == 1, ~1.25M blocks) while + the scheduler pages at block_size 16, so the shape-derived values are 16x too small and + only 1/16 of each block moves, to the wrong offset — total garbage. + how: >- + Use spec.page_size_bytes, which is both alignment-aware and kernel/logical-block-ratio + aware. block_stride is in ELEMENTS, so it becomes page_size_bytes // element_size. + This is what Mooncake already does, which is why Mooncake PD is correct on the same + nodes and model — that differential is how the bug was localised. + before_fix: >- + DSv4 gives right structure with wrong facts ("The capital of France is" -> "a good + idea..."); GLM-5.1 gives total garbage ("is is is is..."). READ and WRITE modes fail + identically, and prefill queried directly is correct. + after_fix: >- + Verified 2026-07-22 on TP4 2-node MoRIIO PD at temp=0: DSv4-Pro France->Paris, + China->Beijing, PD output equal to prefill-direct. GLM-5.1-FP8 likewise correct. + context: >- + TP4 2-node MoRIIO PD, DeepSeek-V4 and GLM-5.1 (block-scaled fp8 MLA + DSA lightning + indexer, --kv-cache-dtype fp8). Byte-for-byte no-op for contiguous matched-block caches + (fp16/bf16 K/V — Qwen, Kimi), where page_size_bytes == stride[0]*es == + block_size*inner*es. The K/V (5-dim) branches are untouched; they already handle the + ratio via kernel_blocks_per_block. + call_chain: + - "moriio_layout.py :: get_layer_transfer_geometry (MLA 3-dim branch)" + - "slot_size_bytes = latent_dim * element_size; block_len = block_size * slot_size_bytes; block_stride = stride[0]" + - "MoRIIO transfer moves block_len bytes at block_stride elements — disagrees with spec.page_size_bytes" + - "decode attends over partially-transferred / mis-scaled KV" + symptom_signature: >- + PD output wrong while prefill-direct is correct; DSv4 right-structure/wrong-fact, + GLM-5.1 degenerate repetition + silent: true + +verification: + date: 2026-07-22 + hardware: TP4 across 2 nodes + software: vLLM v0.25.1, MoRIIO connector + workload: DSv4-Pro and GLM-5.1-FP8 MoRIIO PD, temp=0 + result: >- + DSv4-Pro France->Paris and China->Beijing with PD == prefill-direct; GLM-5.1-FP8 + correct. Diagnosed by differential against Mooncake on the same nodes plus live + per-layer geometry instrumentation. + notes: >- + The Mooncake differential is the transferable technique here: two connectors on the + same nodes and model, one correct, isolates the defect to the transport layer. + +scope_limits: >- + MLA (3-dim) branch only. The K/V (5-dim) branches already handle the kernel/logical ratio + and are deliberately untouched. + +open_actions: + - action: >- + Submit upstream. This is a silent correctness bug present on main with no issue and no + PR, and the fix is to use a value upstream already computes. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_moriio_write.upstream.status.yaml b/deploy/docker/patches/vllm/patch_moriio_write.upstream.status.yaml new file mode 100644 index 00000000..229eec8c --- /dev/null +++ b/deploy/docker/patches/vllm/patch_moriio_write.upstream.status.yaml @@ -0,0 +1,164 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - upstream-source-read + - gh-search + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_moriio_write.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: moriio + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/moriio/moriio_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: >- + Whether THIS base is one of the ROCm images that still hardcodes + is_producer=True is the unresolved question — see applies_to. + +upstream_main_affected: + value: false + evidence: >- + Read from upstream main on 2026-08-05: moriio_connector.py has is_producer=False at + BOTH call sites. Main is fixed. The patch's docstring records the same for + v0.22.1rc0 source. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: unverified + required_to_run: false + evidence: >- + The patch's docstring enumerates v0.20.2 / v0.21.0 / v0.22.0 / v0.22.1 as ROCm images + that still hardcode is_producer=True, but v0.25.1 postdates all of them and main was + fixed, so this base plausibly carries the fix already. Not checked. If it does, the + patch prints "already patched" and exits 0. One grep inside the image settles it. + - engine: vllm + surface: deploy/overlay/infera-exec + effect: unverified + required_to_run: false + evidence: Same question, same base. + +alive_because: + reason: released-but-infera-not-bumped + detail: >- + Fixed upstream on source (main and v0.22.1rc0) but historically absent from every + published AMD ROCm image, which is why the patch exists at all. Our base has since moved + to v0.25.1, so the likely truth is that this is now redundant — a drop candidate whose + only blocker is that nobody has looked. + consumers: + - >- + possibly none on the v0.25.1 base; required on any ROCm image at v0.22.1 or earlier + running MoRIIO WRITE (push) mode + drop_when: >- + The pinned base is confirmed to carry is_producer=False. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + The guard is `if new in src` — i.e. it keys on the FIXED text, so on a fixed base it + reports "already patched" and exits 0. Safe, but the build log line reads like success + rather than like redundancy. Note the sharper edge: if upstream ever ships a base with + a DIFFERENT count of is_producer=True occurrences, the script exits 1, which the + Dockerfile's patch loop swallows into "[vllm-patch] skipped" — so a real failure is + indistinguishable from a benign skip in the log. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged. The base has moved twice under it without the patch being re-examined, + which is precisely how a patch becomes invisible dead weight. + +upstream_issues: + - ref: internal#67 + url: https://github.com/AMD-AGI/Infera/issues/67 + kind: issue + title: "MoRIIO WRITE mode: decode addresses itself for block-allocate notification" + state: OPEN + state_reason: null + author: null + note: >- + Our tracker item, quoted in the patch docstring. Resolves against an internal tracker, + so a 404 is expected. + +# Deliberately null rather than a placeholder ref. Upstream reaches the same end state — +# is_producer=False at both call sites on main and on v0.22.1rc0 source — but the PR that +# did it was never identified, and a guessed number is worse than none. The source read +# under upstream_main_affected is the citation. +upstream_prs: null + +related_refs: null + +problem: + what: >- + In MoRIIO WRITE (push) mode the decode sends the block-allocate notification to ITSELF, + the consumer-side handler asserts, the notify thread dies and the request hangs. + why: >- + The decode calls get_peer_zmq_from_request_id(..., is_producer=True), which resolves to + the producer — itself. The receiving handler asserts get_role() == PRODUCER ("Only + prefill can get block messages") and raises, killing the moriio-notify thread. The + decode must address the PREFILL, i.e. is_producer=False. + how: >- + One-token change: is_producer=True -> is_producer=False at the single call site, guarded + on the occurrence count being exactly 1 so a layout change fails rather than guesses. + before_fix: The request hangs and the client sees HTTP 000; the notify thread is dead. + after_fix: WRITE-mode PD requests complete. + context: >- + MoRIIO WRITE (push) mode only. PULL (read) mode never takes this branch and does not + need the patch. + call_chain: + - "moriio_connector.py :: decode sends block-allocate notification" + - "get_peer_zmq_from_request_id(request.request_id, is_producer=True) -> resolves to self" + - "consumer handler asserts get_role() == PRODUCER -> moriio-notify thread dies" + - "request hangs, HTTP 000" + symptom_signature: '"Only prefill can get block messages" followed by a hung request / HTTP 000' + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm 2-node PD + software: vLLM ROCm image, MoRIIO WRITE mode + workload: MoRIIO push-mode PD request + result: Hang cleared; the notification reaches the prefill. + notes: null + +scope_limits: WRITE (push) mode only. + +open_actions: + - action: >- + Grep the pinned v0.25.1 base for is_producer= and drop this patch if it is already + False. Main is fixed and the base has moved twice since the patch was written. + owner: unassigned + blocked_on: null + - action: >- + Identify the upstream PR that fixed this, so the record cites a change rather than a + source state. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_sched_guard.upstream.status.yaml b/deploy/docker/patches/vllm/patch_sched_guard.upstream.status.yaml new file mode 100644 index 00000000..97a647d7 --- /dev/null +++ b/deploy/docker/patches/vllm/patch_sched_guard.upstream.status.yaml @@ -0,0 +1,183 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - gh-search + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_sched_guard.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: scheduler + repo: vllm-project/vllm + files: + - vllm/v1/core/sched/scheduler.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: v1/core/sched/scheduler.py still has the bare + `assert req_id in self.requests` at both call sites, with three open issues against it + and no PR. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: op + required_to_run: false + evidence: >- + Keeps the decode EngineCore alive under concurrent PD load. Deliberately NOT + required to run: it is applied for throughput runs and should be considered + unwanted for correctness testing — see scope_limits. + - engine: vllm + surface: deploy/overlay/infera-exec + effect: op + required_to_run: false + evidence: Same patch on the runtime-overlay path. + +alive_because: + reason: no-upstream-pr + detail: >- + Three independent upstream issues describe this exact assert and all three are OPEN + with no PR against any of them. Nobody has fixed it upstream, ourselves included — + though in our case that is defensible, because what we carry is a symptom guard and + not a fix, and upstreaming a symptom guard would be the wrong contribution. + consumers: + - Dockerfile.vllm and the overlay payload, for PD throughput runs + drop_when: >- + Upstream replaces the assert with real handling of late KV-transfer-finished events — + which means fixing the race, not guarding it. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + This patch converts a crash into a warning while the underlying race still loses the + racing request's KV transfer. Carrying it on a correctness run therefore HIDES a + correctness defect. That is the risk here — not that the patch becomes stale, but that + it is applied on a run whose purpose is to surface what it suppresses. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: + - ref: vllm-project/vllm#43226 + url: https://github.com/vllm-project/vllm/issues/43226 + kind: issue + title: "assert req_id in self.requests in _update_from_kv_xfer_finished for an aborted/freed request" + state: OPEN + state_reason: null + author: null + note: The most discussed of the three (5 comments). Same assert, aborted-request trigger. + - ref: vllm-project/vllm#46240 + url: https://github.com/vllm-project/vllm/issues/46240 + kind: issue + title: "Scheduler assert when finished_recving and finished_sending arrive in one step" + state: OPEN + state_reason: null + author: null + note: Same assert, different trigger — both events in a single step. + - ref: vllm-project/vllm#49089 + url: https://github.com/vllm-project/vllm/issues/49089 + kind: issue + title: "Late KV-transfer-finished event for an already-failed request trips the scheduler assert" + state: OPEN + state_reason: null + author: null + note: >- + Same assert again. Three independent reporters and no PR is the useful signal here: + the race is real and unowned upstream. + - ref: internal#69 + url: https://github.com/AMD-AGI/Infera/issues/69 + kind: issue + title: "PD: decode EngineCore dies on assert req_id in self.requests under concurrent load" + state: OPEN + state_reason: null + author: null + note: >- + Our tracker item, quoted in the patch docstring. Resolves against an internal + tracker, so a 404 is expected rather than stale. + +upstream_prs: null + +related_refs: null + +problem: + what: >- + Under concurrent PD load the decode EngineCore dies on + `assert req_id in self.requests` in scheduler._update_from_kv_xfer_finished — a + KV-transfer-finished event arrives for a request that has already been removed. + why: >- + The scheduler assumes a KV-transfer-finished event can only refer to a live request. + It can't guarantee that: the request can be aborted, freed, or completed between the + transfer being issued and the event arriving, and both the recving and sending paths + have the same assumption. + how: >- + Skip the stale event with a warning instead of asserting, at both call sites. This is + explicitly a symptom guard, not a repair — the racing request still loses its KV + transfer. + before_fix: The decode EngineCore process dies, taking the deployment down mid-run. + after_fix: >- + The engine stays up and logs "KV xfer finished for unknown req (recving/sending); + skipping stale event". The affected request is still wrong. + context: >- + Concurrent PD load on vLLM. Applied for THROUGHPUT runs where correctness checking is + off; it must not be applied to correctness tests, which should surface the race. + call_chain: + - "vllm/v1/core/sched/scheduler.py :: _update_from_kv_xfer_finished" + - "finished_recving / finished_sending event for a req_id already removed from self.requests" + - "bare assert -> EngineCore process death" + symptom_signature: "assert req_id in self.requests" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm multi-node PD + software: vLLM + workload: concurrent PD throughput + result: EngineCore survives; the stale event is logged and skipped. + notes: >- + "Works" here means the process stays up, which is the whole intent and also the whole + limitation. + +scope_limits: >- + Does NOT fix the underlying correctness race — the racing request still loses its KV + transfer. Apply for throughput runs; leave it off for correctness runs so the race + stays visible. + +open_actions: + - action: >- + Decide whether to pursue the real fix upstream. Three open issues and no PR means + the race is unowned; a proper repair would be a genuine contribution where this + guard would not. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.upstream.status.yaml b/deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.upstream.status.yaml new file mode 100644 index 00000000..b2231929 --- /dev/null +++ b/deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.upstream.status.yaml @@ -0,0 +1,180 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_vllm_mooncake_blocksize.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: mooncake-connector + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + No upstream commit is recorded here: the base is consumed as a published image + digest, not a git ref, so the exact vLLM commit is whatever v0.25.1 was cut from. + The digest below is the real pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: >- + vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. The tag in the ref is for readability; + the @sha256 is what pins it. + +upstream_main_affected: + value: false + evidence: >- + vllm#46807 is MERGED (2026-06-30) and landed the marker name + _physical_blocks_per_logical_kv_block on main — seven hits in its diff. Main is fixed. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: unverified + required_to_run: false + evidence: >- + Whether the pinned v0.25.1 base already carries #46807 has not been checked + directly, and it decides between op and no-op. If it does, the patch prints + "already patched" and exits; if not, it is still doing the work. Checking is one + grep inside the image and is the open action below. + - engine: vllm + surface: deploy/overlay/infera-exec (patches applied at container start) + effect: unverified + required_to_run: false + evidence: Same question, same base. + +alive_because: + reason: released-but-infera-not-bumped + detail: >- + Upstream fixed this in passing on 2026-06-30 as part of #46807 (GDN + MLA PD + support). Our own PR proposed the identical repair first and was closed as + superseded. So nothing is outstanding upstream — this is the cleanest drop + candidate in the tree, waiting only on confirmation that the pinned base carries it. + consumers: + - >- + possibly none — this is the next patch to drop, pending a grep of the pinned base + drop_when: >- + The pinned vLLM base is confirmed to carry #46807. + drop_signal: self-guard-marker-skips + silent_misapply_risk: >- + Note the inversion that makes this safe: the guard marker + _physical_blocks_per_logical_kv_block is what UPSTREAM added, not what we edit. So on + a fixed base the patch self-skips and cannot double-apply — but equally there is no + `git apply` failure and no build-log complaint to tell you it has become dead weight. + The build log's "already patched" line is the only signal, and it reads like success. + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: >- + Unchanged. The upstream conversation moved (our PR was superseded) without the + local patch needing an edit. + +upstream_issues: null + +upstream_prs: + - ref: vllm-project/vllm#46807 + url: https://github.com/vllm-project/vllm/pull/46807 + title: Support GDN + MLA PD with the Mooncake connector + state: MERGED + review_decision: null + merged_at: 2026-06-30 + author: null + ours: false + same_approach: true + approach_note: >- + Fixes this in passing as part of a larger change, using the same logical/kernel + ratio handle we did — the name _physical_blocks_per_logical_kv_block is literally + the marker this patch guards on. + requested_action: null + in_pinned_base: null + - ref: vllm-project/vllm#46334 + url: https://github.com/vllm-project/vllm/pull/46334 + title: "Mooncake connector: register KV at logical-page granularity" + state: CLOSED + review_decision: null + merged_at: null + author: llying-001 + ours: true + same_approach: true + approach_note: >- + Ours, proposing the identical repair, and it went in FIRST. Closed 2026-07-01 as + superseded by #46807 rather than rejected — the outcome we wanted, by a different + PR. + requested_action: null + in_pinned_base: null + +related_refs: null + +problem: + what: >- + Some attention backends force a KERNEL block size of 1 even when the user pins + --block-size 16. The KV tensor is then allocated at per-token granularity while the + scheduler still hands the connector LOGICAL page block_ids, so the connector's + address arithmetic is off by the ratio. + why: >- + The stock connector registers num_blocks = cache.shape[0] and block_len = + stride(0)*elemsize. When cache.shape[0] == num_logical_pages * ratio, the address + base + page_id*block_len lands `ratio` times too early. Affected backends are ROCm + Aiter MLA and the DeepSeek-V3.2 / GLM-5.1 DSA lightning indexer. + how: >- + Mirror vLLM's NIXL connector: compute ratio = logical_block_size // kernel_block_size + in _sync_block_size_with_kernel and register at logical-page granularity + (num_blocks //= ratio, block_len *= ratio). ratio == 1 for dense and matched-block + models, so this is byte-for-byte inert there. + before_fix: >- + RDMA transfers empty rows, the decode engine attends over all-zero prompt KV and + emits garbage from the first token. The transfer also overflows, producing + "destination transfer region exceeds remote KV block size". + after_fix: GLM-5.1 fp8 mooncake PD produces correct output from the first token. + context: >- + Observed with GLM-5.1 fp8 over mooncake PD. Applies to any backend that forces + kernel block size 1; no-op for dense or matched-block models. + call_chain: + - "mooncake_connector.py :: register_kv_caches -> num_blocks = cache.shape[0], block_len = stride(0)*elemsize" + - "scheduler hands LOGICAL page block_ids" + - "base + page_id*block_len lands ratio x too early -> empty rows transferred" + symptom_signature: "destination transfer region exceeds remote KV block size" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm multi-node PD + software: vLLM with the Mooncake connector, GLM-5.1 fp8 + workload: mooncake PD, --block-size 16 with a kernel-block-1 backend + result: Garbage-from-first-token cleared. + notes: null + +scope_limits: null + +open_actions: + - action: >- + Grep the pinned v0.25.1 base for _physical_blocks_per_logical_kv_block and, if + present, delete this patch. It is the least contentious retirement available: + merged upstream, our own repair, nothing outstanding. + owner: unassigned + blocked_on: null diff --git a/deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.upstream.status.yaml b/deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.upstream.status.yaml new file mode 100644 index 00000000..6eb747ed --- /dev/null +++ b/deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.upstream.status.yaml @@ -0,0 +1,193 @@ +# yaml-language-server: $schema=../_schema/patch.upstream.status.schema.json +schema_version: 1 +status_updated: 2026-08-05 +verified_by: + - gh-pr-issue-state + - upstream-source-read + - local-repro + +patch: + path: deploy/docker/patches/vllm/patch_vllm_mooncake_prom_metrics.py + kind: python-anchor-script + idempotent: true + applied_by: + - Dockerfile.vllm + - deploy/overlay/Dockerfile.payload + - deploy/overlay/infera-exec + opt_in_flag: null + +target: + library: vllm + component: mooncake-connector + repo: vllm-project/vllm + files: + - vllm/distributed/kv_transfer/kv_connector/v1/mooncake/mooncake_connector.py + versions: + - surface: Dockerfile.vllm + ref: v0.25.1 + commit: null + pinned_ref_on_main: false + release_branch: null + unpinned_risk: >- + Consumed as a published image digest rather than a git ref; the digest is the pin. + images: + - tag: vllm/vllm-openai-rocm:v0.25.1 + digest: sha256:84459732ca98b40fe2f5338a3f050be6d522504e47a484a5180d58fb75956f86 + digest_source: dockerfile-pin + note: vllm 0.25.1, torch 2.11.0, ROCm 7.2.3. + +upstream_main_affected: + value: true + evidence: >- + Read from upstream main on 2026-08-05: mooncake_connector.py still has NO + build_prom_metrics. Main is affected; #50374 is the candidate fix and is unreviewed. + +applies_to: + - engine: vllm + surface: Dockerfile.vllm (v0.25.1 base) + effect: op + required_to_run: true + evidence: >- + Required for the PD + kvd-L3 recipe specifically: MultiConnector(InferaKvdConnector + + MooncakeConnector) with engine metrics enabled dies on the first stats collection + after the first request without it. + - engine: vllm + surface: deploy/overlay/infera-exec + effect: op + required_to_run: true + evidence: Same condition on the runtime-overlay path. + +alive_because: + reason: upstream-pr-open + detail: >- + Upstream now has a candidate, #50374, opened 2026-07-30 and still REVIEW_REQUIRED, plus + a second PR (#43836) that would solve it from the other side by making MultiConnector + skip children without metrics. Either would retire this. Neither has moved. + consumers: + - >- + Dockerfile.vllm and the overlay payload, for PD + kvd-L3 with engine metrics enabled + (i.e. without --disable-log-stats) + drop_when: >- + A base vLLM gives MooncakeConnector a build_prom_metrics (#50374), or makes + MultiConnector tolerant of children without one (#43836). + drop_signal: self-guard-marker-skips + silent_misapply_risk: null + +history: + born: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + last_modified: + date: 2026-07-20 + commit: 89c86fb + subject: Infera v0.1.0 + reason: Unchanged since it was written. + +upstream_issues: null + +upstream_prs: + - ref: vllm-project/vllm#50374 + url: https://github.com/vllm-project/vllm/pull/50374 + title: Add MooncakePromMetrics and wire via build_prom_metrics + state: OPEN + review_decision: REVIEW_REQUIRED + merged_at: null + author: pavithranrao + ours: false + same_approach: true + approach_note: >- + Same shape as ours — give the connector a build_prom_metrics — but a real metrics + class rather than our no-op adapter. Better; ours only has to satisfy the + registration contract because Mooncake's telemetry already flows through + get_kv_connector_stats. + requested_action: null + in_pinned_base: false + - ref: internal#178 + url: https://github.com/AMD-AGI/Infera/pull/178 + title: "Give InferaKvdConnector build_prom_metrics" + state: MERGED + review_decision: null + merged_at: null + author: null + ours: true + same_approach: true + approach_note: >- + Ours, and the direct ancestor of this patch: it fixed the InferaKvd half of the same + bug and stopped there, so KVD=1 with stats still died. That is why the Kimi-K2.6 PD + benchmarks had to run with L3 off or stats off, and why kvd's own L3 telemetry was + invisible. Worth recording as the shape of the mistake: fixing one child of a + MultiConnector fixes nothing, because the assert is over all of them. + requested_action: null + in_pinned_base: null + +related_refs: + - ref: vllm-project/vllm#43836 + url: https://github.com/vllm-project/vllm/pull/43836 + kind: pr + title: "MultiConnector: skip children without Prometheus metrics support" + state: OPEN + state_reason: null + author: null + relevance: alternative-repair + note: >- + The opposite side of the same fix — make the container tolerant instead of making + every child comply. Either landing retires this patch. + +problem: + what: >- + MooncakeConnector implements get_kv_connector_stats but not build_prom_metrics, so + under MultiConnector the Prometheus registration contract is violated and the engine + dies on the first stats collection after the first request. + why: >- + Without its own build_prom_metrics, MooncakeConnector inherits the base classmethod + that returns None. MultiConnector.build_prom_metrics only registers children whose + build_prom_metrics returns non-None, so Mooncake never lands in the registry — and + then MultiKVConnectorPromMetrics.observe asserts that every child IS registered. + Reachable only under MultiConnector; a single-connector setup never takes that path. + how: >- + Add a build_prom_metrics returning a no-op observe() adapter, mirroring + InferaKvdPromMetrics. It invents no counters and alters no existing metric — Mooncake's + transfer telemetry already flows through get_kv_connector_stats, so only the + registration contract needs satisfying. Doing this at image-build time rather than + monkey-patching from infera's Python at run time is deliberate: the runtime approach + loses an import-order race, because MultiConnector calls Mooncake's build_prom_metrics + before the infera module that would patch it is imported. Editing the vendored file + removes the race entirely. + before_fix: >- + "AssertionError: MooncakeConnector is not contained in the list of registered + connectors with Prometheus metrics support: dict_keys([...])" and the engine dies. + after_fix: >- + PD + kvd-L3 runs with engine metrics on, so the vllm:*prefix_cache_* counters and + kvd's L3 telemetry are visible — they were never missing, just gated behind engine + metrics being enabled at all. + context: >- + Exactly the PD + kvd-L3 recipe: MultiConnector(InferaKvdConnector + MooncakeConnector) + with engine metrics enabled. Unreachable with a single connector, and unreachable with + --disable-log-stats, which is how the Kimi-K2.6 benchmarks worked around it. + call_chain: + - "MultiConnector.build_prom_metrics -> registers only children returning non-None" + - "MooncakeConnector inherits the base build_prom_metrics -> None -> never registered" + - "MultiKVConnectorPromMetrics.observe asserts every child is registered -> engine dies" + symptom_signature: "MooncakeConnector is not contained in the list of registered connectors with Prometheus metrics support" + silent: false + +verification: + date: 2026-07-20 + hardware: AMD ROCm PD with kvd L3 + software: vLLM with MultiConnector(InferaKvd + Mooncake), engine metrics enabled + workload: PD + kvd-L3 with stats on + result: >- + The first-request assert is gone and L3 telemetry is visible for the first time. + notes: null + +scope_limits: >- + A no-op adapter: it satisfies registration and emits nothing of its own. If real + Mooncake Prometheus counters are wanted, that is #50374's job, not this patch's. + +open_actions: + - action: >- + Support #50374 with our datapoint rather than opening a competing PR, and note on + the thread that #43836 would also resolve it. + owner: unassigned + blocked_on: upstream review diff --git a/pr.done.md b/pr.done.md new file mode 100644 index 00000000..3fb14fd6 --- /dev/null +++ b/pr.done.md @@ -0,0 +1,150 @@ +# sglang GLM-5.2 patches — upstream PRs opened + +Companion to `work.todo.md`, which inventories all seven sglang GLM-5.2 patches +and establishes that exactly three needed a new upstream PR. This file records +what was opened, on what evidence, and what is still outstanding. + +Date: 2026-08-07. Upstream base: `sgl-project/sglang` `main` @ `7395ee833e`. +All three are **drafts** — see "Deferred validation" for why. + +## Opened + +| PR | Title | Patch it upstreams | State | +|----|-------|--------------------|-------| +| [#33968](https://github.com/sgl-project/sglang/pull/33968) | `[ROCm] Fix HiCache host-pool allocator: hipHostRegister's device pointer is not the host VA` | `sglang_rocm/patch_hicache_rocm_host_alloc.py` | draft | +| [#33970](https://github.com/sgl-project/sglang/pull/33970) | `[PD] Make the mooncake KV transfer wait on the prefill forward that wrote the pages` | `sglang_disagg/patch_mooncake_early_send_wait_event.py` | draft | +| [#33973](https://github.com/sgl-project/sglang/pull/33973) | `[DSA] Remove the device-to-host syncs on the decode DP-divergent branch` | `sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff` — **2a half only** | draft | + +Branches live on `dorado269/sglang`. Working folder with the adapted diffs, +validation scripts and PR bodies: `work/upstream-glm52-sglang-prs/`. + +## What each PR changes, and what was found while adapting it + +### #33968 — HiCache ROCm allocator + +`ALLOC_MEMORY_FUNCS` defaults to `alloc_with_host_register` (anonymous `mmap` + +`hipHostRegister`), but the host pools hand host `data_ptr()`s to GPU kernels +through a device-side pointer table. On ROCm those addresses are not the same, +so the first write-back faults at the host VA. + +**The record described this as a one-line override in the shape of the merged +MUSA PR #23361. It cannot be**, for two reasons established first-hand: + +1. `get_device()` returns `"cuda"` on ROCm — measured in the container — so HIP + has no key of its own to add, unlike `"npu"`/`"musa"`. +2. `memory_pool_host.py:768` and `:1257` key the table with a `torch.device` + **object**. Measured: `torch.device("cuda:0") == "cuda"` is `False` and the + hashes differ, so those two pools *always* resolve through the `defaultdict` + default. Adding only a `"cuda"` key would silently miss them. + +So the default has to change too. That widens behaviour on a ROCm host to any +other device string (e.g. `"xpu"`); raised with the user, who chose to keep +equivalence with the proven fix rather than ship an unvalidated narrower one. + +### #33970 — mooncake early-send wait event + +Silent correctness bug: chunked prefill over mooncake RDMA-reads KV pages while +the forward writing them is still running, so long prompts come back partially +wrong with no crash and no log line. + +Confirmed on `main`: `mooncake/conn.py` contains `wait_event` **0 times** while +`mori/conn.py` contains it **6 times**, and the overlap non-final-chunk send in +`prefill.py` records no event at all. + +**Cross-transport effect worth knowing:** `prefill.py` is transport-agnostic, so +recording the event there also closes the same gap for `mori`, which reads the +field but was never handed one on that path. Checked that `nixl`, `ascend`, +`fake` and `base` do not read `_early_send_wait_event` and that no sender uses +`__slots__`, so setting the attribute is inert for them. + +### #33973 — DSA decode DP host-sync (2a only) + +`seq_lens.max().item()` is a D2H sync on a branch only some DP ranks enter, so +the group deadlocks on the first routed request. + +The strongest upstream argument turned out to be one the local record did not +use: the backend **already declares** `needs_cpu_seq_lens = False`, with a +comment saying it opts out of the D2H sync — so the eager fallback contradicts +its own contract. Also corrected a citation: the CPU mirror this path relies on +comes from `eagle_worker_common.prepare_for_draft_extend` (line 105), not +`base_spec_worker` as the record said. + +**Scope deliberately narrowed.** The local diff bundles a second, independent +defect (2b, the per-request vs per-token page-table row mismatch). It is **not** +in this PR: it overlaps #32209, and porting that PR's trimming approach here +fails reproducibly at concurrency 32 with the cause unidentified. One fix, one +PR; 2b stays local until that is understood. + +## Validation + +Everything below was run **this session**, on 8× MI300X (gfx942), ROCm 7.2.0, +torch 2.9.1, inside `lmsysorg/sglang-rocm:v0.5.15-rocm720-mi30x-20260713` with +the adapted upstream tree on `PYTHONPATH`. Scripts are in the working folder. + +| Script | What it establishes | Result | +|--------|--------------------|--------| +| `probe_host_devptr.py` | host VA vs device pointer per allocation strategy | PASS — all four strategies **equal** on MI300X, reproducing the documented negative control | +| `validate_A.py` | #33968 equivalence + scope vs the proven local fix | PASS | +| `validate_B.py` | #33970 plumbing + that the barrier is real, against live CUDA events | PASS | +| `validate_C.py` | #33973 — the sync is real; widening does not change top-k; the removed mirrors are unreachable | PASS | + +Two findings from those runs that changed the PRs rather than just confirming +them: + +- `.max().item()` measurably blocks — **~0.5–3 ms** run to run, behind 30 queued + 4096² matmuls — versus **~3 µs** for `.shape[1]`. First-hand evidence that the + branch really does desynchronize ranks, quoted in #33973. The number moved + during cleanup: an earlier revision measured 44 ms because the timed call was + also the first one, so it carried lazy-init cost. Warmed, it is 0.5–3 ms. The + PR was corrected; what matters is that it blocks at all, not the magnitude. +- `validate_C.py` initially **failed** an assertion I had written. Diagnosis: a + row with `seq_len < topk` legitimately returns indices past its own `seq_len`, + because top-k must return `topk` entries; those slots carry `-inf` and occur + identically at both widths. My test's model was wrong, not the fix — but the + correct property (every *real* selection is bit-identical narrow vs wide) is + now asserted, and stated in the PR. + +## Deferred validation — why all three are drafts + +The original faults need hardware this session does not have. The vultr MI355X +cluster is **unreachable**: `149.28.124.225` answers neither ping (100% loss) +nor `:22`; `chi2865` / `chi2866` the same. The local box is gfx942, which is the +*negative control* for #33968 and cannot reproduce any of the three faults. + +So this session established **equivalence and scope** against the proven local +fixes, and nothing more. Each PR body says so explicitly and marks which evidence +is historical. + +**TODO when the cluster returns** — re-run against the exact PR diffs, then flip +each draft to ready: + +| PR | Needs | Run | +|----|-------|-----| +| #33968 | 1× gfx950 | HiCache write-back repro: stock must fault at the host VA, patched must not. Re-run `probe_host_devptr.py` there — expect `same=False`, the positive control. | +| #33970 | 2 nodes with RDMA | GLM-5.2-FP8 1P1D over mooncake, `--chunked-prefill-size 131072`, overlap on. Needle retrieval, expect 5/9 → 9/9. **Plus** the added `synchronize()`'s cost on prefill throughput — flagged as unmeasured in the PR, and the first thing a reviewer will ask. | +| #33973 | 1× gfx950, PD + DP-attention + MTP | Group must not deadlock on the first routed request; `py-spy` should show no rank inside `dsa_backend`. | + +## CI + +All three show `pr-gate` red. Cause verified via the job steps API: the failing +step is **"Block draft PR"** — a repository policy that fails the gate for any +draft, and the `*-finish` jobs just aggregate it. Not a lint, format, or compile +failure; `pre-commit` passes clean locally on every changed file. These go green +when the drafts are marked ready. + +## Not upstreamed, and why + +From `work.todo.md`, four of the seven sglang patches correctly get no PR: + +- `patch_glm52_nextn_quark_exclude.py` — #30265 already **merged**; ours is a + backport onto a frozen release base. +- `patch_dsa_indexer_hip_dp_padded_rows.py` — already ours as **#33059** (open). +- `draft_cuda_graph_dp_vote.diff` — **#32209** carries the same fix with the same + strategy; a competing PR would be noise. Add a ROCm datapoint to that thread + and to #32527 instead. +- `patch_hicache_rocm_staged_write_back.py` — **#30350** is the upstream repair + and is better than ours (covers the DeepSeekV4 pools too). It is + `CHANGES_REQUESTED` with the conflict long cleared; the action is a re-review + nudge. Note the inversion: the *merged* #28534 is what introduced the defect. +- `dsa_backend_dp_sync_and_page_table_rows.diff` **2b half** — see #33973's scope + note above. diff --git a/pyproject.toml b/pyproject.toml index e71ed3ed..ea45ce4f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -32,6 +32,9 @@ dev = [ "pre-commit>=3", "pytest>=8", "pytest-asyncio>=0.23", + # scripts/validate-patch-status.py (deploy/docker/patches/ status records) + "jsonschema>=4.18", + "PyYAML>=6", ] # all = ["amd-infera[sglang]", "amd-infera[vllm]", "amd-infera[atom]"] diff --git a/scripts/validate-patch-status.py b/scripts/validate-patch-status.py new file mode 100644 index 00000000..07595b7a --- /dev/null +++ b/scripts/validate-patch-status.py @@ -0,0 +1,288 @@ +#!/usr/bin/env python3 +"""Validate the patch ↔ upstream status records under deploy/docker/patches/. + +Three schemas, three kinds of file: + + _schema/patch.upstream.status.schema.json -> .upstream.status.yaml + _schema/patch.upstream.index.schema.json -> deploy/docker/patch.upstream.status.yaml + _schema/patch.archived.schema.json -> patches/archived/patch.archived.yaml + +Schema validation alone would let the set drift out of agreement with the tree, so +this also cross-checks them against what is actually on disk: + + * every applied patch file has a record, unless the index lists it under not_patches + * every record points at a patch file that exists, and only speaks for its own directory + * every record file on disk belongs to a patch — a rename cannot strand one + * every index entry points at a record that exists, and every record is indexed + * the index totals match the files found + * archived entries point at a file that exists, or say `deleted` and name the commit + +Dates are written unquoted in the YAML, so PyYAML hands us `datetime.date`; those +are normalised to ISO strings before validation, which means the schema's date +pattern still rejects anything written as a malformed string. + +Usage: + python3 scripts/validate-patch-status.py [--repo-root .] [-v] +""" + +from __future__ import annotations + +import argparse +import datetime as dt +import json +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + sys.exit("validate-patch-status: PyYAML is required (pip install -e '.[dev]')") + +try: + from jsonschema import Draft202012Validator, FormatChecker +except ImportError: # pragma: no cover + sys.exit("validate-patch-status: jsonschema is required (pip install -e '.[dev]')") + +PATCH_ROOT = Path("deploy/docker/patches") +SCHEMA_DIR = PATCH_ROOT / "_schema" +INDEX_PATH = Path("deploy/docker/patch.upstream.status.yaml") +ARCHIVED_PATH = PATCH_ROOT / "archived" / "patch.archived.yaml" + +RECORD_SUFFIX = ".upstream.status.yaml" +# Extensions that can carry a fix and therefore need a record. +PATCH_EXTS = {".py", ".diff", ".patch", ".sh"} + + +def normalise(node): + """Turn YAML's native dates back into ISO strings so the schema can check them.""" + if isinstance(node, dict): + return {k: normalise(v) for k, v in node.items()} + if isinstance(node, list): + return [normalise(v) for v in node] + if isinstance(node, dt.datetime): + return node.date().isoformat() + if isinstance(node, dt.date): + return node.isoformat() + return node + + +class Report: + def __init__(self, verbose: bool) -> None: + self.errors: list[str] = [] + self.verbose = verbose + + def fail(self, where: str, message: str) -> None: + self.errors.append(f"{where}: {message}") + + def ok(self, message: str) -> None: + if self.verbose: + print(f" ok {message}") + + +def load_yaml(path: Path, report: Report): + try: + return normalise(yaml.safe_load(path.read_text())) + except yaml.YAMLError as exc: + report.fail(str(path), f"not valid YAML: {exc}") + return None + + +def validate_against(schema_path: Path, doc, where: str, report: Report) -> None: + schema = json.loads(schema_path.read_text()) + validator = Draft202012Validator(schema, format_checker=FormatChecker()) + for err in sorted(validator.iter_errors(doc), key=lambda e: list(e.path)): + loc = "/".join(str(p) for p in err.path) or "(root)" + report.fail(where, f"{loc}: {err.message}") + + +def patch_files(root: Path) -> list[Path]: + """Every file under patches/ that could carry a fix, excluding schemas and archive.""" + found = [] + for path in sorted((root / PATCH_ROOT).rglob("*")): + if not path.is_file() or path.suffix not in PATCH_EXTS: + continue + rel = path.relative_to(root) + parts = rel.parts + if "_schema" in parts or "archived" in parts: + continue + found.append(rel) + return found + + +def record_for(patch: Path) -> Path: + """patches/vllm/patch_x.py -> patches/vllm/patch_x.upstream.status.yaml""" + return patch.with_suffix("").with_name(patch.stem + RECORD_SUFFIX) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--repo-root", default=".", type=Path) + ap.add_argument("-v", "--verbose", action="store_true") + args = ap.parse_args() + root = args.repo_root.resolve() + report = Report(args.verbose) + + record_schema = root / SCHEMA_DIR / "patch.upstream.status.schema.json" + index_schema = root / SCHEMA_DIR / "patch.upstream.index.schema.json" + archived_schema = root / SCHEMA_DIR / "patch.archived.schema.json" + for schema in (record_schema, index_schema, archived_schema): + if not schema.is_file(): + report.fail(str(schema.relative_to(root)), "schema missing") + if report.errors: + print("\n".join(f"FAIL {e}" for e in report.errors)) + return 1 + + # ---- index ------------------------------------------------------------- + index = load_yaml(root / INDEX_PATH, report) + if index is None: + print("\n".join(f"FAIL {e}" for e in report.errors)) + return 1 + validate_against(index_schema, index, str(INDEX_PATH), report) + + not_patches = {Path(e["path"]) for e in index.get("not_patches", [])} + indexed: dict[Path, dict] = {} + for lib, block in index.get("libraries", {}).items(): + for entry in block.get("patches", []): + indexed[Path(entry["patch"])] = {"library": lib, **entry} + + for entry in index.get("not_patches", []): + if not (root / entry["path"]).is_file(): + report.fail(str(INDEX_PATH), f"not_patches lists a missing file: {entry['path']}") + + # ---- per-patch records ------------------------------------------------- + found = patch_files(root) + records_seen: set[Path] = set() + + # A patch can be several files (a .patch plus the fragment a script appends). The + # record names them under extra_files, and they must not also demand a record of + # their own — so collect those before deciding what is missing one. + owned_extras: set[Path] = set() + on_disk_records: set[Path] = set() + for record in sorted((root / PATCH_ROOT).rglob(f"*{RECORD_SUFFIX}")): + rel_record = record.relative_to(root) + on_disk_records.add(rel_record) + doc = load_yaml(record, report) or {} + for entry in doc.get("patch", {}).get("extra_files", []): + extra = Path(entry) + # extra_files is the rest of ONE patch, so a record may only name files + # beside it. Unbounded, one line in any record excuses a patch anywhere + # in the tree from having its own — which is the whole gate. + if extra.parent != rel_record.parent: + report.fail( + str(rel_record), + f"patch.extra_files entry {extra} is outside {rel_record.parent}", + ) + continue + owned_extras.add(extra) + + for patch in found: + if patch in not_patches: + report.ok(f"{patch} (declared not-a-patch)") + continue + if patch in owned_extras: + report.ok(f"{patch} (covered as extra_files)") + continue + record = record_for(patch) + if not (root / record).is_file(): + report.fail( + str(patch), + f"no status record — expected {record}, or list it under not_patches in the index", + ) + continue + records_seen.add(record) + doc = load_yaml(root / record, report) + if doc is None: + continue + validate_against(record_schema, doc, str(record), report) + + declared = Path(doc.get("patch", {}).get("path", "")) + if declared != patch: + report.fail(str(record), f"patch.path is {declared}, file is {patch}") + for extra in doc.get("patch", {}).get("extra_files", []): + if not (root / extra).is_file(): + report.fail(str(record), f"patch.extra_files entry missing: {extra}") + if patch not in indexed: + report.fail(str(patch), "has a record but is not listed in the index") + elif Path(indexed[patch]["record"]) != record: + report.fail( + str(INDEX_PATH), + f"{patch}: record is {indexed[patch]['record']}, expected {record}", + ) + else: + report.ok(f"{patch} <-> {record}") + + for patch in indexed: + if patch not in found: + report.fail(str(INDEX_PATH), f"indexes a patch that is not on disk: {patch}") + + # The walk above only goes patch -> record, so a record whose patch was renamed or + # deleted is never loaded and never checked. Close the loop the other way. + for stale in sorted(on_disk_records - records_seen): + report.fail( + str(stale), + "does not correspond to any patch file — delete it, or restore the patch", + ) + + # ---- archived ---------------------------------------------------------- + archived_declared = Path(index["archived_record"]) + if archived_declared != ARCHIVED_PATH: + report.fail(str(INDEX_PATH), f"archived_record should be {ARCHIVED_PATH}") + archived = load_yaml(root / ARCHIVED_PATH, report) + if archived is not None: + validate_against(archived_schema, archived, str(ARCHIVED_PATH), report) + for entry in archived.get("patches", []): + src = entry.get("source", {}) + current = src.get("current_path") + where = f"{ARCHIVED_PATH}[{entry.get('name')}]" + if current == "deleted": + if not src.get("last_commit_with_file"): + report.fail( + where, + "current_path is 'deleted' but last_commit_with_file is unset, " + "so the file cannot be recovered", + ) + elif not (root / current).is_file(): + report.fail(where, f"current_path does not exist: {current}") + for extra in src.get("extra_files", []): + if not (root / extra).is_file(): + report.fail(where, f"extra_files entry missing: {extra}") + if entry.get("retired_reason") == "upstream-fixed-and-in-base" and not entry.get( + "upstream_fix" + ): + report.fail(where, "retired as upstream-fixed but upstream_fix is null") + + # ---- totals ------------------------------------------------------------ + totals = index.get("totals", {}) + active = len([p for p in found if p not in not_patches and p not in owned_extras]) + if totals.get("active_patches") != active: + report.fail( + str(INDEX_PATH), + f"totals.active_patches is {totals.get('active_patches')}, found {active}", + ) + if totals.get("records") != len(records_seen): + report.fail( + str(INDEX_PATH), + f"totals.records is {totals.get('records')}, found {len(records_seen)}", + ) + n_archived = len((archived or {}).get("patches", [])) + if totals.get("archived_patches") != n_archived: + report.fail( + str(INDEX_PATH), + f"totals.archived_patches is {totals.get('archived_patches')}, found {n_archived}", + ) + + # ---- result ------------------------------------------------------------ + if report.errors: + for err in report.errors: + print(f"FAIL {err}") + print(f"\n{len(report.errors)} problem(s)") + return 1 + print( + f"patch status OK — {len(records_seen)} record(s), " + f"{len(indexed)} indexed, {n_archived} archived" + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/unit/test_patch_status_records.py b/tests/unit/test_patch_status_records.py new file mode 100644 index 00000000..1f9d431a --- /dev/null +++ b/tests/unit/test_patch_status_records.py @@ -0,0 +1,175 @@ +############################################################################### +# Copyright (c) 2026, Advanced Micro Devices, Inc. All rights reserved. +# +# SPDX-License-Identifier: MIT +############################################################################### +"""The patch status records under deploy/docker/patches/ hold, and the gate bites. + +A patch that outlives its upstream fix is not obviously broken: it keeps applying +cleanly and the build log says nothing. The records exist so that state is written +down, and scripts/validate-patch-status.py exists so the writing-down cannot be +skipped. A validator that passes on a tree with a missing record would be worse +than none, so these tests check the REFUSALS as well as the happy path: + + * the real tree validates + * a patch added without a record fails + * a record whose date is not a date fails + * an index whose totals disagree with the tree fails + * an archived entry that claims `deleted` without a recovery commit fails + * a record that reaches outside its own directory to cover a patch fails + * a record left behind by a renamed or deleted patch fails +""" + +from __future__ import annotations + +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] +VALIDATOR = REPO_ROOT / "scripts" / "validate-patch-status.py" +PATCH_ROOT = Path("deploy/docker/patches") +INDEX = Path("deploy/docker/patch.upstream.status.yaml") +ARCHIVED = PATCH_ROOT / "archived" / "patch.archived.yaml" + +pytestmark = pytest.mark.skipif( + not VALIDATOR.is_file(), reason="validator script not present in this checkout" +) + + +def run(root: Path) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [sys.executable, str(VALIDATOR), "--repo-root", str(root)], + capture_output=True, + text=True, + ) + + +@pytest.fixture +def tree(tmp_path: Path) -> Path: + """A throwaway copy of just the parts the validator reads.""" + root = tmp_path / "repo" + (root / PATCH_ROOT.parent).mkdir(parents=True) + shutil.copytree(REPO_ROOT / PATCH_ROOT, root / PATCH_ROOT) + shutil.copy(REPO_ROOT / INDEX, root / INDEX) + (root / "scripts").mkdir() + shutil.copy(VALIDATOR, root / "scripts" / VALIDATOR.name) + return root + + +def test_repo_records_validate() -> None: + """The tree as committed passes, so a later failure means a real regression.""" + done = run(REPO_ROOT) + assert done.returncode == 0, done.stdout + done.stderr + + +def test_copied_tree_validates(tree: Path) -> None: + """Guards the fixture itself — the mutations below only mean something from a clean base.""" + done = run(tree) + assert done.returncode == 0, done.stdout + done.stderr + + +def test_patch_without_record_is_rejected(tree: Path) -> None: + (tree / PATCH_ROOT / "vllm" / "patch_brand_new.py").write_text("# no record\n") + done = run(tree) + assert done.returncode == 1 + assert "no status record" in done.stdout + + +def test_record_with_non_date_is_rejected(tree: Path) -> None: + record = tree / PATCH_ROOT / "vllm" / "patch_sched_guard.upstream.status.yaml" + record.write_text( + record.read_text().replace("status_updated: 2026-08-05", 'status_updated: "soon"') + ) + done = run(tree) + assert done.returncode == 1 + assert "status_updated" in done.stdout + + +def test_record_not_in_index_is_rejected(tree: Path) -> None: + index = tree / INDEX + text = index.read_text() + # Drop the whole entry for one patch, keeping the file valid YAML. + start = text.index(" - patch: deploy/docker/patches/vllm/patch_sched_guard.py") + end = text.index(" - patch:", start + 1) + index.write_text(text[:start] + text[end:]) + done = run(tree) + assert done.returncode == 1 + assert "not listed in the index" in done.stdout + + +def test_wrong_totals_are_rejected(tree: Path) -> None: + index = tree / INDEX + index.write_text(index.read_text().replace("active_patches: 25", "active_patches: 24")) + done = run(tree) + assert done.returncode == 1 + assert "totals.active_patches" in done.stdout + + +def test_deleted_archive_entry_needs_a_recovery_commit(tree: Path) -> None: + archived = tree / ARCHIVED + archived.write_text( + archived.read_text().replace( + "last_commit_with_file: 89c86fb", "last_commit_with_file: null", 1 + ) + ) + done = run(tree) + assert done.returncode == 1 + assert "cannot be recovered" in done.stdout + + +def test_record_pointing_at_the_wrong_patch_is_rejected(tree: Path) -> None: + record = tree / PATCH_ROOT / "vllm" / "patch_sched_guard.upstream.status.yaml" + record.write_text( + record.read_text().replace( + "path: deploy/docker/patches/vllm/patch_sched_guard.py", + "path: deploy/docker/patches/vllm/patch_somewhere_else.py", + ) + ) + done = run(tree) + assert done.returncode == 1 + assert "patch.path is" in done.stdout + + +def test_extra_files_cannot_reach_into_another_directory(tree: Path) -> None: + """extra_files is for the rest of one patch, not a way to vouch for someone else's. + + Left unbounded it is a hole straight through the gate: the new patch below needs a + record, and one line in an unrelated record would otherwise be enough to excuse it. + """ + (tree / PATCH_ROOT / "vllm" / "patch_brand_new.py").write_text("# no record\n") + record = tree / PATCH_ROOT / "atom" / "patch_gdn_pd_state_transfer.upstream.status.yaml" + record.write_text( + record.read_text().replace( + "patch:\n", + "patch:\n extra_files:\n - deploy/docker/patches/vllm/patch_brand_new.py\n", + 1, + ) + ) + done = run(tree) + assert done.returncode == 1 + assert "is outside" in done.stdout + # and the patch it tried to cover is still asked for a record of its own + assert "no status record" in done.stdout + + +def test_orphan_record_is_rejected(tree: Path) -> None: + """Nothing walks record -> patch, so an unclaimed record is never even parsed.""" + (tree / PATCH_ROOT / "vllm" / "patch_ghost.upstream.status.yaml").write_text( + "status_updated: not-a-date\n" + ) + done = run(tree) + assert done.returncode == 1 + assert "does not correspond to any patch file" in done.stdout + + +def test_renamed_patch_may_not_leave_its_record_behind(tree: Path) -> None: + """The realistic version of the above: `git mv` the patch, forget the record.""" + vllm = tree / PATCH_ROOT / "vllm" + (vllm / "patch_sched_guard.py").rename(vllm / "patch_sched_guard_v2.py") + done = run(tree) + assert done.returncode == 1 + assert "does not correspond to any patch file" in done.stdout diff --git a/work.todo.md b/work.todo.md new file mode 100644 index 00000000..d3971f1c --- /dev/null +++ b/work.todo.md @@ -0,0 +1,142 @@ +# sglang GLM-5.2 patches — upstream PR inventory + +Scope: every patch in `deploy/docker/patches/` whose `target.library` is `sglang` +and whose problem was found on, or verified against, GLM-5.2. That is all seven +sglang patches in the tree — no sglang patch in this repo is unrelated to GLM-5.2. + +Source of truth: each patch's `.upstream.status.yaml`, re-checked against +live upstream on 2026-08-07 (`gh` for PR/issue state, `git show origin/main` for +source). vLLM / mooncake / aiter / atom patches are out of scope here even where +they mention GLM. + +Upstream `main` read at `3ed2a0adf3` (2026-08-07). + +## The table + +| # | Patch | Component | Existing upstream PR | State (2026-08-07) | Ours? | main affected? | Needs a NEW PR | +|---|-------|-----------|----------------------|--------------------|-------|----------------|----------------| +| 1 | `sglang/patch_glm52_nextn_quark_exclude.py` | quantization | sglang#30265 | MERGED 2026-07-08 | no | **no** — fixed | **no** | +| 2 | `sglang_disagg/patch_mooncake_early_send_wait_event.py` | disaggregation | *none* | — | — | **yes** | **YES** | +| 3 | `sglang_dsa/patch_dsa_indexer_hip_dp_padded_rows.py` | dsa | sglang#33059 | OPEN, REVIEW_REQUIRED | **yes** | yes | no — already filed | +| 4 | `sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff` (2a) | dsa | *none* | — | — | **yes** | **YES** | +| 4b | `sglang_dsa/dsa_backend_dp_sync_and_page_table_rows.diff` (2b) | dsa | sglang#32209 | OPEN, REVIEW_REQUIRED | no | yes | **no** — see below | +| 5 | `sglang_dsa/draft_cuda_graph_dp_vote.diff` | speculative-decoding | sglang#32209 | OPEN, REVIEW_REQUIRED | no | yes | no — deliberate | +| 6 | `sglang_rocm/patch_hicache_rocm_host_alloc.py` | hicache | *none* | — | — | **yes** | **YES** | +| 7 | `sglang_rocm/patch_hicache_rocm_staged_write_back.py` | hicache | sglang#30350 | OPEN, CHANGES_REQUESTED | no | yes | no — nudge #30350 | + +Three patches need a new upstream PR: **#2, #4 (2a half only), #6**. + +## Per-patch notes + +### 1 — `patch_glm52_nextn_quark_exclude.py` — no PR needed +sglang#30265 ("[AMD] Fix GLM-5.2 MTP Quark excludes", wangjiaxin99) is MERGED and +is a superset of ours: it gives GLM-5.2 a dedicated `GlmMoeDsaForCausalLMNextN`. +Our patch is a one-line backport onto the frozen `v0.5.15.post1` base, which was +cut from `release/v0.5.15` without it. Nothing to upstream — it is upstream. + +Carry-forward warning (already in the record): the anchor string +`ckpt_prefix = f"model.layers.{config.num_hidden_layers}"` is STILL on main +(`deepseek_nextn.py:328`), so the patch does not self-disable on a fixed base. +Decide the drop from the base version, not from the build log. + +### 2 — `patch_mooncake_early_send_wait_event.py` — NEEDS A PR +No upstream PR, and no upstream issue that establishes the root cause (#25583 is +the same corruption shape on an *aggregated* GLM-5-FP8 server, auto-closed +inactive — suggestive only). + +Re-verified on main today: +- `disaggregation/mooncake/conn.py` — `wait_event` occurs **0 times**, while + `disaggregation/mori/conn.py` has **6**. The barrier exists only for mori. +- `disaggregation/prefill.py:1113` records `_early_send_wait_event` on the + radix early-send path; the overlap non-final-chunk send at line 812 records + nothing at all. + +So main is affected on both halves. This is a **silent** correctness bug — no +crash, no log line, output partially wrong past the first prefill chunk. Best +candidate in the tree for an upstream PR. + +Before opening: the record's own open action asks for the cost of the added +`synchronize()` on prefill throughput. Not measured. Will be stated as an open +question in the PR rather than claimed either way. + +### 3 — `patch_dsa_indexer_hip_dp_padded_rows.py` — already filed +Ours: sglang#33059, OPEN, `REVIEW_REQUIRED`, `MERGEABLE`/`BLOCKED`, last touched +2026-08-07. No action beyond review chasing. + +Note the file moved upstream: `layers/attention/dsa_indexer.py` → +`layers/attention/dsa/dsa_indexer.py`. #33059 is already against the new path. + +### 4 — `dsa_backend_dp_sync_and_page_table_rows.diff` — SPLIT +This diff bundles two independent defects. They upstream differently. + +**2a — DP host-sync deadlock. NEEDS A PR.** No upstream issue, no upstream PR, +ours or anyone's. Confirmed live on main: +- `dsa_backend.py:794` — `max_seqlen_k = int(forward_batch.seq_lens.max().item())`, + a blocking D2H sync on a branch only some DP ranks take. +- `dsa_backend.py:854-861` — the two further unconditional `.cpu()` mirrors + (2a2) are still there. With 2a alone the hang persists, so both go together. + +**2b — page-table row mismatch. NO new PR.** sglang#32209 (HZY-Wade) is OPEN and +already addresses this row mismatch, by TRIMMING q/top-k where we EXPAND the page +table. Our record documents a negative result: porting #32209's trimming approach +onto this HIP/tilelang path fails reproducibly at concurrency 32 (0/32 across +seven runs), with the root cause **not identified**. Opening a competing PR on an +unexplained failure would be noise. Correct move is to finish that investigation +first; keep 2b local until then. + +### 5 — `draft_cuda_graph_dp_vote.diff` — no PR, deliberate +sglang#32209 carries this exact fix with the same strategy (group decision rather +than per-rank), and our diff adopts its placement verbatim so the two converge. +sglang#32527 (Xavier1994) reports the same deadlock independently on 8× Blackwell +— not ROCm-specific — and is still OPEN with no activity. + +The value we can add is a datapoint on those threads, not a fourth PR. + +### 6 — `patch_hicache_rocm_host_alloc.py` — NEEDS A PR +No upstream issue and no upstream PR. Confirmed live on main +(`mem_cache/pool_host/common.py:177-183`): + +```python +ALLOC_MEMORY_FUNCS = defaultdict( + lambda: alloc_with_host_register, + {"npu": alloc_with_pin_memory, "musa": alloc_with_pin_memory}, +) +``` + +No HIP entry. The merged precedent to copy is sglang#23361 ("[MUSA][19/N] Support +HiCache with pin_memory allocator") — same one-line dispatch override, same +reason. This is the clearest missing-PR gap in the whole tree: a crash-class fix, +main affected by direct source read, and an already-merged PR of the same shape. + +Anchor-collision risk when it lands: #32503 and #32792 (both OPEN, Intel XPU) +touch the same dict. + +### 7 — `patch_hicache_rocm_staged_write_back.py` — no new PR +sglang#30350 (Emmanuel0612) is the upstream repair and is strictly better than +ours: it flips the three CUDA-only gates via `_is_cuda_alike`, covers the +DeepSeekV4 pools we do not, and teaches `staged_write_back.cuh` to accept +`kDLROCM`. It is OPEN with `CHANGES_REQUESTED` (HaiShaw, 2026-07-13); the +conflict was cleared the same day and no re-review has been requested. + +Our MI300X datapoint is already on the thread (llying-001, 2026-08-04). Correct +action is a re-review nudge, not a competing PR. Note the inversion: the MERGED +PR here (#28534) is what *introduced* the disagreement. + +## Plan + +1. Open an upstream PR for **#6** (hicache ROCm allocator) — smallest, strongest + evidence, merged precedent. +2. Open an upstream PR for **#2** (mooncake early-send wait event) — silent + correctness bug, three files, mirrors what mori already does. +3. Open an upstream PR for **#4 / 2a** (DP host-sync deadlock) — 2a + 2a2 only. + 2b stays local. +4. All three as **drafts**, via the `open-source-pr` workflow: check upstream + main, strip local-repo semantics (`GLM52_*` markers, the `MARKER = "applied"` + bytecode literal, infera path references), re-validate the adapted patch + locally for equivalence and scope, then open. +5. Record the outcome in `pr.done.md` and update each patch's + `upstream_prs` / `open_actions` in its `.upstream.status.yaml`. + +Not doing, and why: #1 is already merged upstream; #3 is already our open PR; +#5 and #7 have a better third-party PR in flight that we should support rather +than compete with; #4 / 2b rests on an unexplained concurrency-32 failure.