[memory] PCIe comm slabs + indexer scratch: per-capture channel slabs, per-call fold allocs, compressed-wire ring oversizing, fp32 tile_logits HWM

[malaiwah]

Found during a memory review of the v20 r12 release image voipmonitor/vllm:gilded-gnosis-v20-vllmb46c3aa-si35aebc6-fi801d57a-cu132-20260730-r12. Code refs are against the composed r12 tree; all cited sparkinfer files are byte-identical to public commit de04125508c75e748016493f851e6bd4513b515b (reachable from pr/92), so permalinks use that SHA. vLLM caller refs are from the composed r12 vLLM tree (base f978d009 + release-lock PR heads). Deployment context for the impact math: GLM-5.2 753B MoE, EXL3 3.0bpw, 4x RTX 6000 Pro Blackwell 96GB (sm120), TP4; release gate = DCP4 / fp8 KV / V2 runner / max_num_batched_tokens=8192 / VLLM_DCP_A2A_MAX_TOKENS=64; production turnkey = DCP2 / nvfp4 KV / V1 runner / batched 3072 / MTP k=5 / GMU 0.957 (~4.1 GiB total margin per GPU). These are proposals from a source review, not demands — happy to be corrected on any of the mechanics.

1. DCP A2A pool mints a fresh IPC channel + slab per graph-capture context, retained until pool close

Where: sparkinfer/comm/pcie/pcie_dcp_a2a.py#L766-L789

    @contextmanager
    def capture(self, stream: object = None):
        """Bind nested CUDA captures to the enclosing stream's channel."""
        if self.single_channel:
            channel = self.for_stream(stream)
        else:
            ...
            # Keep a graph-owned channel even when CUDA recycles the enclosing
            # stream handle used by a previously captured graph manager.
            channel = self._new_channel(stream_key)
            self._channels[key] = channel

What/Why: With single_channel=False (what vLLM's dcp_alltoall._get_b12x_dcp_a2a_pool passes), every entry into PCIeDCPA2APool.capture() unconditionally allocates a new channel, i.e. a new IPC staging slab (_staging_layout, lines 61-106: two slots of max_batch × total_heads × max(head_dim, query_head_dim) × 2 B output + fp32 LSE). Channels are appended to _all_channels and only freed in close() (lines 791-801); checkpoint/rollback_channels is used by vLLM solely around throwaway profiling captures.

One correction to our own initial reading, in fairness to the design: vLLM enters exactly one capture context per cudagraph-manager capture pass, not one per capture size — vllm/v1/worker/gpu/cudagraph_utils.py:464 wraps the whole PIECEWISE+FULL size loop in a single graph_capture(...), which enters capture_b12x_dcp_a2a once. So the multiplier is the number of manager passes (target model + MTP drafter prefill + drafter decode on the V2 runner), not the 16+ capture sizes.

Impact (recomputed): Per-channel slab with VLLM_DCP_A2A_MAX_TOKENS=64, query_head_dim=576, head_dim=512:

• DCP4 (gate, total_heads=96 = 24 heads/TP-rank × 4): 13.55 MiB/channel/rank. Warm channel + 3 capture passes ≈ 4 channels ≈ 54 MiB/rank, of which ~40 MiB exists only for graph-lifetime isolation.
• DCP2 (turnkey, total_heads=48): 6.77 MiB/channel; warm + 1 capture pass (V1 runner captures everything under one graph_capture) ≈ 13.5 MiB/rank.

Not huge, but it is IPC-pinned cudaMalloc memory outside the torch allocator, invisible to the profiler's KV budget, and it grows by one slab any time another capture pass is added (e.g. more drafter managers). At turnkey's ~4.1 GiB total margin/GPU, 54 MiB is ~1.3% of margin.

Suggested fix: Key capture channels by capture context and reuse them across manager passes that cannot replay concurrently (target FULL vs drafter decode graphs can overlap, but target prefill/drafter prefill cannot), or free transient capture channels when their graphs are destroyed rather than at pool close.

Verification: Count PCIeDCPA2A.from_exchange_group calls (or log in _new_channel) during a TP4/DCP4 startup with MTP enabled on the V2 runner; observe 1 warmup + 1 per manager capture pass, none freed before shutdown.

2. Two-level fold candidate slab is torch.empty'd per index_topk_fp8 call, escaping the scratch plan

Where: sparkinfer/attention/nsa_indexer/paged.py#L955-L964

    if two_level_slices:
        fold_values = torch.empty(
            (q_rows * total_slices, topk), dtype=torch.float32, device=q_fp8.device
        )
        fold_indices = torch.empty(
            (q_rows * total_slices, topk), dtype=torch.int32, device=q_fp8.device
        )
        fold_lengths = torch.full(
            (q_rows,), total_slices * topk, dtype=torch.int32, device=q_fp8.device
        )

What/Why: Everything else this route touches is plan-reserved (tile_logits, top-k buffers, the streaming-carry double buffer at scratch.py:948-962), but the two-level fold candidate slab is a per-call allocation, sized q_rows × (total_slices × topk × 8 + 4) bytes (_plan_two_level_fold, paged.py:198), bounded only by the 256 MiB SPARKINFER_INDEXER_TWO_LEVEL_FOLD_MAX_MIB budget. On GLM (32 indexer heads, topk=2048) the fused route only covers decode rows ≤ 16 (fused_indexer.py:138, _GLM32_FUSED_MAX_ROWS = 16), so any decode batch of 17–64 rows takes the tiled route and hits this allocation — per sparse layer, per step.

Impact (recomputed): At long context (≥512k tokens the slice cap makes total_slices = 32), per call: 48 rows (turnkey: 8 seqs × MTP k=5+1) → 24 MiB; 64 rows (gate capture max) → 32 MiB. In eager mode that is a 24 MiB empty+free per sparse layer per step — exactly the pattern that expandable_segments:False (forced under LMCache) turns into fragmentation pressure. Under FULL CUDA-graph capture the slab lands in the graph pool; since the V2 runner captures all sizes into one shared pool largest-first, retained pool growth is roughly the max-size slab (~32 MiB), not a per-size sum — we could not substantiate a 100+ MiB accumulation from the code alone, so we flag the graph-pool magnitude as uncertain; the per-call churn is not.

Suggested fix: Move the fold candidate slab into the paged scratch layout (it is bounded by min(budget, max_q_rows × 32 × topk × 8), i.e. ≤ 32 MiB at decode caps — cheaper than one extra graph's worth of churn), sharing one reservation across capture sizes like the carry double-buffer already does.

Verification: torch.cuda.memory_stats() alloc/free counters across one decode step at 48 rows / long context show per-layer 24 MiB allocation pairs attributable to index_topk_fp8; the scratch plan (plan_indexer_scratch) contains no matching reservation.

3. DMA ring slab keeps full bf16 shard width in compressed wire modes, and adds the fp8 stage on top

Where: sparkinfer/comm/pcie/pcie_dma.py#L179-L229

        self.shard_capacity = _align_up(
            (self.max_bytes + self.world_size - 1) // self.world_size, SCRATCH_ALIGN
        )
        steps = 2 * (self.world_size - 1)
        flags_bytes = FLAG_SLOTS * FLAG_STRIDE
        slab_bytes = flags_bytes + steps * self.shard_capacity
        ...
        if self._fp8:
            max_shard_elems = self.max_bytes // 2 // self.world_size
            stride = _align_up(
                max_shard_elems + max_shard_elems // FP8_QUANT_BLOCK * 4,
                SCRATCH_ALIGN,
            )
            self._fp8_stage = torch.empty(
                self.world_size * stride, dtype=torch.uint8, device=self.device
            )

What/Why: shard_capacity is computed from max_bytes (bf16 sizing) before and regardless of the wire mode. In the compressed ring/a2a modes every hop copies piece_slice_bytes = piece_elems + piece_elems/128*4 (~0.516× of the bf16 width — verified against the step loop at lines 496-524, where compressed sends write into slots addressed by piece * piece_slice_bytes but strided by the full shard_capacity). So in i8_ring/mx_ring/ring mode, all 2(W−1) slots are ~48% dead space. In the AG-only modes (ag/i8/mx) the reduce-scatter hops still carry full-width bf16, so only the (W−1) allgather slots could shrink. On top of that, _fp8_stage is allocated eagerly at init, in addition to the full-width slab.

Impact (recomputed, TP4, max_bytes = max_num_batched_tokens × 6144 × 2):

• Gate (8192 tok): slab = 32 KiB flags + 6 × 24 MiB = 144.03 MiB/rank IPC-pinned; +49.5 MiB _fp8_stage if a compressed mode is enabled.
• Turnkey (3072 tok): slab = 54.03 MiB/rank; +18.6 MiB stage if compressed.
• Sizing shard_capacity from the wire payload in fully-compressed modes would save ~70 MiB (gate) / ~26 MiB (turnkey) per rank; in AG-only modes ~35 MiB / ~13 MiB (allgather slots only).

Note the default deployment (no SPARKINFER_PCIE_DMA_FP8/VLLM_PCIE_DMA_FP8) runs bf16 wire, where the slab is correctly sized — this finding applies to the compressed opt-in.

Suggested fix: Compute a per-mode slot width (ring/a2a variants: ~0.52 × shard_capacity for all steps; ag variants: full width for RS steps, compressed width for AG steps) and derive _fp8_stage from the same widths instead of adding it on top.

Verification: Instantiate PCIeDmaAllReduce(max_bytes=96 MiB, fp8="i8_ring") on 4 ranks and compare slab_bytes against the sum of bytes actually addressed by fp8_scratch_piece across all steps (~52% of the slab).

4. Latent: full-context fp32 logits blob on the direct (non-supertile) paged logits route

Where: sparkinfer/attention/nsa_indexer/kernel.py#L2323-L2333 (inside run_paged_logits_kernel)

        if preinitialize_invalid_logits:
            logits = torch.full(
                (rows, width_tokens),
                float("-inf"),
                dtype=torch.float32,
                device=q_fp8.device,
            )
        else:
            logits = torch.empty(
                (rows, width_tokens), dtype=torch.float32, device=q_fp8.device
            )

What/Why: Both branches materialize a (rows, width_tokens) fp32 blob — 128 MiB at 512k context × 64 rows (and the -inf branch also pays a full fill). We confirmed production GLM decode does not reach this: index_topk_fp8 routes through the fused kernel (rows ≤ 16) or the supertile tile-logits scratch, and run_paged_logits_kernel's only callers are _impl.paged_decode_logits and the MSA block-scores path. Filing it as a latent hazard: any future caller (debug, reference parity, MSA at long context) silently allocates context-proportional fp32.

Suggested fix: Guard the direct route with a byte ceiling (mirroring the two-level fold budget), or require a caller-provided output buffer above a threshold.

Verification: Call paged_decode_logits with a 512k-token page table and 64 rows; observe the single 128 MiB fp32 allocation.

5. tile_logits fp32 plan sizing: up to 384-512 MiB of shared-workspace high-water mark at prefill caps

Where: sparkinfer/attention/nsa_indexer/scratch.py#L883-L893 and #L936-L937

    num_q_tiles = (
        max_q_rows + _PAGED_INDEX_TILE_BLOCK_Q - 1
    ) // _PAGED_INDEX_TILE_BLOCK_Q
    num_k_tiles = supertile_tokens // _PAGED_INDEX_TILE_BLOCK_K
    tile_logits_elements = max(
        1,
        num_q_tiles
        * num_k_tiles
        * _PAGED_INDEX_TILE_BLOCK_Q
        * _PAGED_INDEX_TILE_BLOCK_K,
    )
    ...
    cursor += tile_logits_elements * dtype_nbytes(torch.float32)

What/Why: tile_logits = round32(max_q_rows) × supertile_tokens × 4 B (supertile default 32768). We traced what vLLM actually plans (per the "verify the row cap" note in our review): the composed vLLM's sparse_attn_indexer.py reserves both a decode plan at capture rows and a prefill plan at profile_q_rows = min(q_rows, VLLM_SPARSE_INDEXER_MAX_LOGITS_MB·MiB/4 / 32768) (_get_b12x_paged_indexer_profile_q_rows, default cap 512 MB → 4096 rows), bound into the shared workspace via current_workspace_manager().get_simultaneous(*plan.shapes_and_dtypes()).

Impact (recomputed): decode plan at 64 rows: 8 MiB (fine). Prefill plan: turnkey 3072 rows → 384 MiB; gate 8192 rows → capped at 4096 → 512 MiB contributed to the shared workspace high-water mark per rank. (The 1 GiB figure in our internal review only occurs with a raised VLLM_SPARSE_INDEXER_MAX_LOGITS_MB; the default env cap bounds it at 512 MiB by construction.) bf16 tile logits would halve this to 192/256 MiB — but tiled_topk.py:1616 hard-requires fp32 (row_logits must have dtype torch.float32), so this needs a kernel-side change, not just a dtype swap. Given the indexer scores are consumed only through a top-k selection, fp32→bf16 precision impact is plausibly acceptable but needs validation on the 3.0 bpw model.

Suggested fix: Accept bf16 tile_logits in run_tiled_topk (keep fp32 carry/output), or expose a per-plan dtype so integrators can trade the workspace HWM.

Verification: Log plan.shapes_and_dtypes() from the prefill reservation on a 3072-token profile run: the tile_logits entry is (3072/32768-tile shape) fp32 = 384 MiB.

6. Oneshot channels: 8 MiB rank_data per channel for ≤84 KiB dispatch payloads

Where: sparkinfer/comm/pcie/pcie_oneshot.py#L27 and #L325-L327

DEFAULT_RANK_DATA_BYTES = 8 * 1024 * 1024
...
        self.rank_data = torch.empty(
            rank_data_bytes, dtype=torch.uint8, device=self.device
        )

What/Why: Every oneshot channel allocates an 8 MiB rank_data tensor; the composed vLLM never overrides rank_data_bytes. Channels multiply like the DCP A2A ones: one warm channel + one per graph_capture() entry (custom_all_reduce.py:672 enters pool.capture() per capture pass) → ~4 channels on the V2 gate config → ~32 MiB/rank of registration bookkeeping for a dispatcher whose payloads are capped at 84 KiB (VLLM_PCIE_ONESHOT_ALLREDUCE_MAX_SIZE).

One important correction to our internal review: the "2×8 MiB eager slabs per channel" part does not apply to the vLLM deployment path. The composed vLLM already right-sizes the eager IPC slabs to the dispatch cutoff — custom_all_reduce.py:78-89 passes eager_buffer_bytes = max(84 KB, 84 KB, 16), so the per-channel IPC slab is ~169 KiB, and the in-tree comment explicitly documents why. The 2×8 MiB cost only hits callers that use from_exchange_group defaults (DEFAULT_MAX_SIZE at line 26).

Suggested fix: Scale rank_data_bytes with max_size (a ~1 MiB default comfortably covers 84 KiB dispatch registration), and/or lower DEFAULT_MAX_SIZE so default-constructed channels don't reserve 16 MiB of IPC staging for an 84 KiB dispatcher.

Verification: On a TP4 V2-runner boot with MTP, sum channel.rank_data.numel() across pool._all_channels after capture: 4 × 8 MiB, against max_size=86016.

[malaiwah]

Independent re-review by a second reviewer against the same trees (sparkinfer de041255, composed r12 vLLM, launcher 513bd84), re-deriving every number from the shipped launcher env rather than vLLM defaults. Retractions and corrections included; two of the six claims change materially.

Claim	Verdict	Note
1. Per-capture A2A channel slabs	HOLDS-WITH-CORRECTION	13.55 MiB/channel and the gate's 4 channels are exact. Turnkey is 3 channels / 20.32 MiB, not 2 / 13.5 — the V1 runner's throwaway profiling capture mints a channel and never rolls it back. Profiler-visibility claim needs splitting (warm channel is budgeted).
2. Per-call fold slab	HOLDS-WITH-CORRECTION	24 MiB @48 rows / 32 MiB @64 rows exact. Unreachable at the release gate (seq=1 ⇒ ≤6 decode rows ⇒ fused route). At turnkey, uniform decode is FULL-captured, so the churn is capture-time; eager exposure is limited to PIECEWISE mixed batches. Suggested fix bound is wrong at the prefill plan (see below).
3. DMA ring sized for bf16 wire	HOLDS-WITH-CORRECTION (both impact numbers wrong)	Wire mode is bf16 in every shipped config — confirmed from the launcher, not inferred; the compressed-mode claims are correctly descoped to opt-in users, but max_bytes is 3072 tokens in both configs, not 8192: slab 54.03 MiB (not 144.03), stage 18.56 MiB (not 49.5), compressed-mode savings 26.2 / 13.1 MiB (not 70 / 35).
4. Full-context fp32 -inf blob	HOLDS	128 MiB arithmetic exact; the two callers and the "not reached in production" scoping both re-verified.
5. fp32 tile_logits HWM	HOLDS-WITH-CORRECTION	The reservation is a shared-arena high-water contribution, not additive. Gate is also 3072 rows ⇒ 384 MiB, not 512 MiB. The fp32 contract for tile_logits is at tiled_topk.py:1518, not the row_logits validator at :1616 (different function) — conclusion unchanged, citation wrong. The genuinely additive axis is the V2 two-lane workspace, which the issue does not mention.
6. Oneshot rank_data	HOLDS, sharpened	8 MiB is per channel and never overridden; gate 4×8 = 32 MiB, turnkey 3×8 = 24 MiB. Realized utilization is 2 RankData = 256 bytes (0.003%): the graph-buffer registration path is unreachable once eager buffers are configured, so the shrink has no registry-bound risk.

---

3 — wire mode resolved: bf16, in both shipped configs

This is the question worth settling, because the whole finding hinges on it. The launcher chain, not the docs:

# serve-glm52-v19.sh:25 and serve-gilded-gnosis.sh:42 (glm52-exl3 family)
F8_DMA="${F8_DMA:-0}"
...
configure_glm52_pcie_runtime_env "${B12X_PCIE_DMA}" "${F8_DMA}"

# glm52-pcie-runtime-env.sh:16-18
export VLLM_PCIE_DMA_FP8="${dma_wire_mode}"
export B12X_PCIE_DMA_FP8="${dma_wire_mode}"
export SPARKINFER_PCIE_DMA_FP8="${dma_wire_mode}"

F8_DMA is unset in the release-gate invocation and absent from the turnkey container's 36 user-set variables, so all three exports are 0. custom_all_reduce.py:537-552 forwards that string verbatim as fp8= to PCIeDmaAllReduce, and _normalize_fp8_mode("0") returns "" (pcie_dma.py:78-79) ⇒ self._fp8 falsy ⇒ _fp8_stage is not allocated and wire_mode == "bf16". Corroborating evidence in-tree: the launcher's own preflight requires it (tests/fake-glm52-pcie-calibrator.sh:17 → require_env VLLM_PCIE_DMA_FP8 0), and the calibration policy explicitly refuses to auto-select compressed transport ("compressed_dma_requires_explicit_opt_in": true; serve-glm52-v19.sh:146 → skipped:explicit-compressed-dma).

So: the slab is correctly sized in the shipped configuration, there is no eager _fp8_stage, and both the oversizing and the "adds the fp8 stage on top" sub-claims apply only to compressed opt-in. The issue's closing note already says this; the impact bullets should carry the same scope, because as written they read as shipped costs.

Separately, the reachability of the ~48% dead space within compressed modes re-verified and holds: step slots stride by shard_capacity (_scratch_ptr, pcie_dma.py:308-309) while compressed payloads pack at piece * piece_slice_bytes (:447-448, piece_slice_bytes = piece_elems + piece_elems/128*4 at :410), i.e. 0.5156× of the bf16 width.

3b — both configs are 3072 batched tokens, so the gate column is 2.67× too large

max_bytes comes from the scheduler config, not a default:

# custom_all_reduce.py:453-465
max_num_batched_tokens = vllm_config.scheduler_config.max_num_batched_tokens
...
pcie_buffer_size = max_num_batched_tokens * model_hidden_size * 2

The EXL3 family pins MAX_BATCHED_TOKENS=3072 (serve-gilded-gnosis.sh:29), overriding v16's 8192 default, and turnkey passes --max-num-batched-tokens 3072. Recomputed for TP4, max_bytes = 3072·6144·2 = 36 MiB:

	shard_capacity	slab (32 KiB flags + 6 slots)	_fp8_stage if compressed	full-compressed saving	AG-only saving
gate and turnkey	9.00 MiB	54.03 MiB	18.56 MiB	26.2 MiB	13.1 MiB

The 144.03 / 49.5 / 70 / 35 MiB figures correspond to 8192 tokens, which no shipped profile uses. One caveat worth a log check rather than an assumption: the except branch at custom_all_reduce.py:456-464 falls back to 8192 and emits "vLLM config unavailable during CustomAllreduce init". That branch should not fire — WorkerBase.init_device wraps the call in set_current_vllm_config (v1/worker/worker_base.py:329) and the communicator is built inside it — but the warning is a one-line grep on any real boot, so please confirm rather than take my word.

5 — additive vs. shared: shared, one arena per (ubatch, lane)

WorkspaceManager owns exactly one buffer per (ubatch, lane) and only ever grows it to the largest single get_simultaneous request:

# v1/worker/workspace.py:71-73, 139, 168
self._current_workspaces: list[torch.Tensor | None] = [None] * (self._num_ubatches * self._num_lanes)
...
current_workspace = self._ensure_workspace_size(total_bytes)
...
if current_size < required_bytes:   # grow only

So the 384 MiB is a high-water contribution, not an allocation that sits beside the other workspace consumers. Two consequences for the impact framing:

• The prefill indexer plan's total is what sets the arena, and tile_logits is 72% of it. Recomputing the whole layout at turnkey/gate (max_q_rows=3072, topk=2048, supertile 32768, max_page_table_width=8192): tile_logits 384.0 MiB, topk_values+topk_indices 48.0 MiB, fold carry double-buffer 96.0 MiB, gather-K + lengths ≈ 4.0 MiB ⇒ 532.1 MiB plan total. The decode plan is 15.1 MiB (64 rows) and is fully absorbed. Halving tile_logits shrinks the plan to 340.1 MiB, so the realized saving is 532.1 − max(340.1, largest_other_lane-0_demand) — bounded above by 192 MiB and possibly zero. That "largest other consumer" is the number to measure (VLLM_DEBUG_WORKSPACE=1 prints every resize with its caller), not to assume.
• Gate profile_q_rows is min(q_rows, 512 MiB/4/32768) = min(3072, 4096) = 3072, so the 512 MiB / 4096-row cap is never hit at the gate either. Both configs are 384 MiB. The default env cap is confirmed 512 (envs.py:60), so the framing "capped at 512 MiB by construction" is right; it just isn't the binding constraint here.
• The genuinely additive axis is one you didn't file: num_workspace_lanes = 2 whenever the V2 runner has a speculative config (gpu_worker.py:419-424), and the drafter runs under use_workspace_lane(1) (gpu/model_runner.py:756, 973, 2019). Lane 1 is an independent arena. If the MTP draft's sparse-MLA layer takes the same prefill reservation path, the gate pays the plan twice. I could not settle that statically — it is the higher-value measurement of the two.

fp32 assert: the citation is off by function. run_tiled_topk performs no dtype check on tile_logits; the hard fp32 contract is the kernel binding at tiled_topk.py:1518, _to_kernel_tensor(flat_input, cutlass.Float32, assumed_align=4), plus the radix helpers that consume the fp32 bit pattern (_float_as_uint32, _convert_to_uint8, _convert_to_coarse_bin). :1616 (row_logits must have dtype torch.float32) belongs to run_row_topk, the direct route from claim 4. Conclusion is unchanged and if anything stronger: bf16 needs a new dtype template, a bf16 ordered-key mapping for the radix bins, and a bf16 store path in the producing logits kernel.

1 — channel count and lifetime

Re-derived independently. One pool per DCP group (_get_b12x_dcp_a2a_pool keys on (group, device, total_heads, head_dim, query_head_dim, max_batch_size), and warmup's all-gather and LSE-reduce probes resolve to the same key because both clamp max_batch_size to VLLM_DCP_A2A_MAX_TOKENS=64). Slab arithmetic reproduces exactly: align(64·96·576·2) + align(64·96·4) per slot, ×2 slots + signal ⇒ 14,205,952 B = 13.5479 MiB at DCP4; 7,103,488 B = 6.7744 MiB at DCP2.

graph_capture() entries, each of which unconditionally mints a channel:

• Gate (V2, MTP3) — target manager, drafter prefill manager, drafter decode manager (spec_decode/autoregressive/speculator.py:108, 125; the decode manager exists because num_speculative_steps != 1). Profiling captures are checkpointed and rolled back (gpu/model_runner.py:957, 989). 1 warm + 3 = 4 channels = 54.19 MiB. Confirmed.
• Gate MTP0 — no speculator ⇒ 1 warm + 1 = 27.10 MiB.
• Turnkey (V1) — checkpoint_b12x_graph_channels has exactly one caller and it is on the V2 runner. The V1 path enters graph_capture for the throwaway profiling capture at gpu_model_runner.py:6745 with no checkpoint, and that capture always runs (gpu_worker.py:558-563 gates only on cudagraph_mode != NONE; VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS gates whether the estimate is applied, not whether the capture happens). Its channel leaks for the process lifetime. 1 warm + 1 leaked + 1 production = 3 channels = 20.32 MiB, so turnkey is ~50% worse than filed, and one of the three slabs buys nothing at all.

Profiler-visibility needs splitting. The warm channel is created in kernel_warmup, which runs inside the memory_profiling context (gpu_worker.py:487 via _profile_model_with_kernel_warmup, called at :545), so it lands in non_torch_increase and is budgeted. On V1 the leaked profiling-capture channel is also inside that context and is never freed, so it is budgeted too. What escapes every term is the production-capture channel(s): 40.64 MiB at the MTP3 gate, 6.77 MiB at turnkey. Same accounting hole as vllm#201/GG-12, worth cross-referencing.

2 — reachability and the fix bound

Route gate re-verified: _GLM32_FUSED_MAX_ROWS = 16 for topk==2048 and num_heads==32 (fused_indexer.py:138, 554-555). Gate runs MAX_NUM_SEQS=1, so decode rows are 1 (MTP0) or 4 (MTP3, graph_step = MTP+1) — always fused. Claim 2 is turnkey-only.

Fold eligibility also required checking output_physical_slots, which disables the two-level fold outright (paged.py:161-165): deepseek_v2.py:818-820 sets it only when decode_context_parallel_size == 1, so at DCP2/DCP4 it is False and the fold is live. Good.

Slab size confirmed at 32 slices for a 524288-token context (_TWO_LEVEL_SLICE_TOKENS=16384, _TWO_LEVEL_MAX_SLICES=32): 48 rows → 25,166,016 B = 24.00 MiB; 64 rows → 32.00 MiB. With 57 sparse layers in the shipped pattern (FFFSSS…, 78 chars, 57 S), an eager step is ~1.34 GiB of allocate/free traffic — worth stating, since "24 MiB per layer per step" understates it.

But vllm::sparse_attn_indexer is in _attention_ops (config/compilation.py:776) and therefore in default splitting_ops, so it runs outside piecewise graphs and inside FULL graphs. The EXL3 launcher emits "cudagraph_mode":"FULL_AND_PIECEWISE" (serve-glm52-v16.sh:345), and turnkey captures [4,8,…,64], which includes 48. So uniform decode should take the FULL graph: the torch.empty executes once at capture, its address is baked into the graph, and per-step churn is zero. The eager window is then only mixed/non-uniform batches carrying 17–64 decode rows — genuinely narrower than filed. (Turnkey supplies its own CUDAGRAPH_CAPTURE_SIZES, so confirm its final --compilation-config still carries FULL_AND_PIECEWISE rather than plain PIECEWISE; that single field decides whether this claim is a per-step cost or a one-off.) Prefill chunks never allocate the slab at all: at 3072 rows candidate_nbytes = 3072·(32·2048·8+4) ≈ 1.50 GiB exceeds the 256 MiB budget, so mode == "auto" disables the fold (paged.py:199-204).

6 — rank_data

from_exchange_group never overrides rank_data_bytes, and each _new_channel builds a channel with its own torch.empty(rank_data_bytes) (pcie_oneshot.py:325-327, 1053-1099), so 8 MiB is per channel — confirmed. Channel counts follow claim 1's graph_capture accounting exactly (custom_all_reduce.py:672 enters pool.capture() per pass; checkpoint_pcie_channels is reached only through the V2-only checkpoint_b12x_graph_channels): 4 × 8 MiB gate, 3 × 8 MiB turnkey.

The utilization figure is much starker than filed, and it also answers the registry-bound question. RankData is kMaxRanks=16 pointers = 128 B, so 8 MiB is 65,536 slots. In this deployment exactly two are consumed, by register_pcie_buffers — because PCIeOneshotAllReduce.capture() only calls register_graph_buffers() when self._eager_ptrs is None (pcie_oneshot.py:768-769), and vLLM always configures eager buffers. _prepare_input correspondingly never calls register_buffer on that path (:616). 256 bytes of 8 MiB.

Minor: the 84 KiB cutoff comes from VLLM_PCIE_ONESHOT_FUSED_ADD_RMS_NORM_MAX_SIZE, not from VLLM_PCIE_ONESHOT_ALLREDUCE_MAX_SIZE — the launcher sets those to 84KB and 64KB respectively (glm52-pcie-runtime-env.sh:13-14), and buffer_size = max(allreduce, fused, 16) (custom_all_reduce.py:88). max_size=86016 is right; the attribution isn't.

---

Fix-safety notes

1 — reuse capture channels across non-overlapping managers. Capture-safe in the sense that matters: channel creation is IPC cudaMalloc + handle exchange, which already must happen before capture begins, and capture() fast-fails if entered mid-capture (pcie_dcp_a2a.py:772-776). The hazard is the sharing predicate, not the allocation. Sharing one channel between the target FULL graph and the drafter decode graph is unsafe if those graphs can be in flight concurrently; the target-prefill / drafter-prefill pair is safe. Note this interacts directly with sparkinfer#95: two graphs sharing a channel also share its host-side staging-slot parity counter, so a shared channel makes that latent parity bug reachable rather than accidental. Sequence #95's device-side slot derivation before any channel sharing. The strictly safer half of the fix — free transient capture channels when their graphs are destroyed — has no such coupling, and the machinery already exists (rollback_channels + _coordinated_close_channels); it needs a rank-collective close, which rollback_channels already provides. Cheapest correct first step: call checkpoint_b12x_graph_channels/rollback_b12x_graph_channels around the V1 profiling capture too. That is a two-line change recovering 6.77 MiB at turnkey with zero new invariants, since it reuses the V2 path verbatim. Measure: log in _new_channel and count survivors after capture_model.

2 — move the fold slab into the scratch plan. Do not use the bound in the issue. min(budget, max_q_rows·32·topk·8) evaluates to 1.5 GiB on the prefill plan (max_q_rows=3072) and would over-reserve by ~1.47 GiB for a slab the runtime never allocates there, and the constant 32 is wrong above 1 M context: total_slices is max(32, width/32768) — 32 at 524288, 64 at 2 M, 128 at 4 M. No capacity-independent bound exists. The right formulation is to call _plan_two_level_fold from _indexer_paged_scratch_layout: it is a pure function of (q_rows, topk, page_size, page_table_width, supertile_pages, output_physical_slots) plus the env policy, all of which the layout already has, so the reservation becomes exact — candidate_nbytes when the fold is enabled, zero when the budget or physical-slot check disables it. Capture-safe (allocation moves to profile time, before lock_workspace()); numerically identical (same shapes/dtypes, fold_lengths still needs its full() fill, which is a graph-legal kernel). Best part: because the workspace arena is shared and already ≥532 MiB from the prefill plan, adding 32 MiB to the decode plan (15.1 → 47.1 MiB) grows the arena by zero. Verify with VLLM_DEBUG_WORKSPACE=1 — no new resize line should appear.

3 — per-mode slot widths. Compressed-only, so no shipped-config risk, but two hazards worth naming. (a) The step-slot stride is load-bearing for peer addressing: _scratch_ptr(owner, step) = scratch_base[owner] + step·shard_capacity (pcie_dma.py:308-309) is evaluated on the sending rank against the receiver's IPC base, so every rank must compute an identical stride. Derive the compressed stride from (mode, max_bytes, world_size) only — never from a per-call quantity. _pick_pieces returns 2 or 1 depending on the actual shard_elems (:312-320, :405), and although pieces · piece_slice_bytes happens to be pieces-invariant (= shard_elems·(1 + 4/128)), a stride derived from any per-call value would drift between ranks that see different tensor sizes in flight. Rank disagreement here is silent cross-rank corruption, not a crash. (b) Size the compressed slot from max_bytes/world_size, not from the current call: should_allreduce admits anything up to max_bytes (:333), so the slot must hold the largest admissible shard's compressed width, and a2a modes take a different addressing path entirely (_all_reduce_fp8) that needs the same treatment or it will read a slot sized for the ring layout. Deriving _fp8_stage from the same widths is safe — it is a private device tensor, not IPC, and _fp8_stage_stride is already independent of pieces.

4 — byte ceiling on the direct route. Safe; it is a plain torch.empty/torch.full on an eager path with no IPC or graph exposure. Prefer the caller-provided-buffer variant: raising above a threshold changes paged_decode_logits from "slow" to "fails", and the MSA block-scores caller already demonstrates the page_scores-style out-param pattern (kernel.py:2289-2318).

5 — bf16 tile_logits. Not a dtype swap. Needs (i) a Float16/BFloat16 template for flat_input at :1518, (ii) a bf16 analogue of the monotone float→radix-key mapping used by _convert_to_uint8 / _convert_to_coarse_bin / _float_as_uint32, and (iii) a bf16 store path in the producing logits kernel. Numerics do change observably: 8 mantissa bits will reorder near-equal indexer scores, so the selected 2048-token set differs run-to-run against fp32 — this is a model-output change, not a rounding detail, and the fp32 carry/output does not protect it because the comparison happens on the bf16 values. -inf masking survives bf16. Given the sharing analysis above, the win may also be zero: measure the lane-0 arena high-water and its second-largest consumer first, and check whether lane 1 holds a second copy. If lane 1 does, sharing/eliminating that duplicate is a larger, exactness-preserving win than the dtype change — and the same shape as vllm#203/GG-02.

6 — shrink rank_data_bytes. Safe with a large margin, and the registry bound is not the constraint: the only consumer is register_pcie_buffers (2 slots, 256 B), and register_graph_buffers is unreachable while eager buffers are configured. check_rank_data_capacity throws rather than corrupting, so an under-size mistake fails loudly at startup, not silently at runtime. 1 MiB leaves 8,192 slots; even a hypothetical future graph-registration path (156 A2A ops × 16 captured sizes ≈ 2,496 buffers) fits. Lowering DEFAULT_MAX_SIZE is the riskier half — it is the default for all callers, including the autotune/benchmark paths in this file, so scale rank_data_bytes from max_size and leave DEFAULT_MAX_SIZE alone. Not capture-relevant: rank_data is allocated at channel construction, never during capture.

Priority triage

Trivial + safe (do now, no measurement):

• Wrap the V1 profiling capture in checkpoint_b12x_graph_channels/rollback_b12x_graph_channels — 6.77 MiB + 8 MiB rank_data at turnkey, reusing the V2 code path verbatim (claims 1, 6).
• rank_data_bytes scaled from max_size — 24–32 MiB/rank across configs, provably 256 B in use (claim 6).
• Byte ceiling / out-param on the direct paged-logits route — latent-only, zero shipped risk (claim 4).
• Fold slab into the scratch layout via _plan_two_level_fold, not via the filed bound — free at turnkey because the arena is already larger (claim 2).

Needs measurement before it earns a change:

• Lane-0 workspace high-water and its second-largest consumer; whether lane 1 holds a duplicate indexer plan on the V2 gate. This decides whether bf16 tile_logits is worth its numerics risk, and may surface a bigger exactness-preserving win (claim 5).
• Channel-reuse-across-managers, gated on sparkinfer#95's device-side slot derivation landing first (claim 1).
• Compressed-mode slot widths — only if a compressed wire mode is actually being considered for deployment; today it is opt-in and off (claim 3).

Drop / rescope:

• Claim 3's impact table as written: 144.03 / 49.5 / 70 / 35 MiB are 8192-token numbers and no shipped profile uses 8192. Substitute 54.03 / 18.56 / 26.2 / 13.1, and mark the whole finding compressed-opt-in.
• Claim 2's gate row (64 rows / 32 MiB): unreachable at seq=1.
• Claim 5's 512 MiB gate figure: gate is 3072 rows, same 384 MiB as turnkey.