[bug] W4A16 fused MoE: runtime launch resolve has no CUDA-graph capture guard, so a first-use CuTe compile can run inside a captured graph

[malaiwah]

Found during a source review of the v20 r12 release image voipmonitor/vllm:gilded-gnosis-v20-vllmb46c3aa-si35aebc6-fi801d57a-cu132-20260730-r12. All cited sparkinfer files are byte-identical to public commit de04125508c75e748016493f851e6bd4513b515b (git diff de04125 r12-composed -- <path> empty for kernel.py, fused_moe/_impl.py, _lib/compiler.py, w4a16/route_pack.py, w4a16/host.py; composed tree 35aebc6), so permalinks use that SHA. vLLM caller refs are from the composed r12 vLLM tree (commit b31667d3, tree b46c3aa) and are quoted inline without permalinks.

Deployment context: GLM-5.2 753B MoE (hidden 6144, 75 MoE layers, top-k 8), EXL3 TR3 3.0 bpw, TP4, cudagraph_mode=FULL_AND_PIECEWISE; launcher defaults VLLM_EXL3_TRELLIS_MAX_M=32, VLLM_EXL3_TRELLIS_BLOCK_M=8, VLLM_EXL3_PREFILL_TRELLIS=1, VLLM_EXL3_PREFILL_BLOCK_M=64, MAX_BATCHED_TOKENS=3072.

This is a defensive-hardening report, filed with an honest reachability verdict: the hazard is LATENT on both shipped configs, not live. It is filed anyway because (1) the compile entry point demonstrably sits inside the captured region on the shipped EXL3 path, (2) every layer of protection is external to sparkinfer while every sibling first-use site in the same file self-defends, (3) the one in-process backstop is bypassed on a disk-cache hit, and (4) the fix as originally proposed is wrong at the proposed location.

Where

compile_w4a16_fused_moe is consulted on every run_w4a16_moe call, including from inside a captured CUDA graph, and has no capture guard: sparkinfer/moe/_shared/kernels/w4a16/kernel.py#L10246-L10259

    if int(prepared.workspace.numel()) < sms * 4 + 2:
        raise ValueError("prepared W4A16 workspace is too small for fused FC1+FC2")
    if fused_launch is None:
        fused = compile_w4a16_fused_moe(
            size_m=m,
            hidden_size=hidden_size,
            ...
            moe_block_size=block_size_m,
            max_m_blocks=int(required_m_blocks),

fused_launch is always None on the vLLM path — sparkinfer/moe/fused_moe/_impl.py#L937-L959 (TPMoEScratchPlan.bind):

            route_expert_map=route_expert_map,
            output_expert_map=output_expert_map,
            fused_launch=None,
            topk_sum_launch=None,
        )

The whole of compile_w4a16_fused_moe (kernel.py:7026-7731) contains no capture check. Exhaustive listing of is_current_stream_capturing in kernel.py — three occurrences, none in either compile entry point nor at the resolve site:

8446         if torch.cuda.is_current_stream_capturing():   # rotation placeholder
9179     if torch.cuda.is_current_stream_capturing():       # _get_c_tmp scratch
9439         if torch.cuda.is_current_stream_capturing():   # Trellis dense storage

Contrast the planner, which does guard — sparkinfer/moe/fused_moe/_impl.py#L6183-L6186:

    if not torch.cuda.is_available():
        raise RuntimeError("CUDA must be available before planning Trellis MoE")
    if torch.cuda.is_current_stream_capturing():
        raise RuntimeError("Trellis launch planning cannot run during capture")

and the sibling first-use compile sites elsewhere in the tree that all self-defend: gemm/_shared/mxfp8_bmm.py:1393 and :1490 (raise on "compile miss during CUDA-graph capture"), gemm/mla_query_projection/_bf16.py:199-203, gemm/bf16_gemv/_kernel.py:289-292 (falls back to cuBLAS under capture), w4a16/route_pack.py:96, moe/ep_moe/_impl.py:106, moe/fused_moe/_impl.py:5900 and :5964. The largest and slowest compile in the MoE stack (the fused FC1→act→FC2 megakernel plus two GEMM kernels) is the one hot-path first-use site with no guard.

Mechanism

1. bind() hard-codes fused_launch=None, so the launch is resolved lazily inside the kernel call rather than being carried on the binding. The captured graph body therefore executes compile_w4a16_fused_moe(...) as Python on every MoE layer.
2. The requested cache key is ("w4a16_fused_moe", device, kernel.__cache_key__) (kernel.py:7423-7428). size_m is not in the key (it is dataclasses.replaced onto a hit at L7430-7435), but max_m_blocks, moe_block_size, cta_m_blocks, blocks_per_sm, tile_n/tile_k and shared_words of both GEMMs are (kernel.py:1000-1037, nested via L4810-4811).
3. max_m_blocks comes from required_m_blocks = int(block_expert_ids.numel()) (kernel.py:10201), which pack_topk_routes_by_expert slices to capacity_route_blocks derived from route_pack_numel_capacity(numel, topk) → route_pack_token_capacity → 1 << (tokens-1).bit_length() (host.py:232-240, route_pack.py:307-331, route_pack.py:108-112). So the key is a deterministic function of the next-pow2 token bucket of m, plus host-side constants.
4. On a _FUSED_CACHE miss (kernel.py:6606, an unbounded module-level dict) the function proceeds straight into sparkinfer_compile(...) (kernel.py:7659, 8146, 8236) → sparkinfer._lib.compiler.compile, i.e. an NVVM/ptxas JIT plus CUDA module load, with a CUDA graph capture in progress.

Reachability (LATENT on both shipped configs)

I tried to construct a live path and could not. Recorded in full so the margin is auditable:

The entry point is genuinely inside capture. Shipped EXL3 MoE routes exl3.py:_apply_rank_sliced → api.bind(plan, scratch=...) → TPMoEScratchPlan.bind (fused_launch=None) → run_w4a16_moe, for every decode step, inside the FULL decode graphs. Not eager-only.

But vLLM warms every key first. GPUModelRunner._warmup_and_capture (vllm/v1/worker/gpu_model_runner.py:6913-6938) runs cudagraph_num_of_warmups eager _dummy_runs at exactly desc.num_tokens immediately before capturing that descriptor, and vllm/config/vllm.py:1430 forces cudagraph_num_of_warmups = 1 whenever cudagraphs are enabled (the CompilationConfig default at config/compilation.py:643 is 0). The FULL path adds a second eager re-warm (vllm/v1/worker/gpu/cudagraph_utils.py:500-509), and CUDAGraphWrapper optionally a third (vllm/compilation/cuda_graph.py:317-329). Since the key is a pure function of m = desc.num_tokens plus constants, and _FUSED_CACHE is unbounded (no eviction between warmup and capture), every key capture requests was already populated eagerly.

Recomputed key coverage (top-k 8, E=160 local experts, max_m_blocks(m) = ceil(max_packed_route_slots(next_pow2(m)*topk, bs, E) / bs); capture sizes are MTP+1 .. GRAPH in steps of MTP+1, serve-glm52-v16.sh:337-345):

capture m	pow2 bucket	plan	bs	max_m_blocks
4	4	decode	8	32
8	8	decode	8	64
12, 16	16	decode	8	128
20, 24, 28, 32	32	decode	8	172
36…64 (m > TRELLIS_MAX_M=32)	64	prefill	64	166

Five distinct GEMM keys for a GRAPH=32 capture set; six with GRAPH=64. Per-size eager warmup is therefore load-bearing, not incidental — nothing else populates these.

Second line of defence, also external. Both capture sites wrap torch.cuda.graph(...) in guard_b12x_kernel_resolution (vllm/compilation/cuda_graph.py:332-341 for PIECEWISE, vllm/v1/worker/gpu/cudagraph_utils.py:510-516 for FULL), which calls sparkinfer.freeze_kernel_resolution. It is active on the shipped configs because serve-glm52-v16.sh:227-239 unconditionally exports VLLM_USE_B12X_WO_PROJECTION=1, VLLM_USE_B12X_MHC=1, VLLM_USE_B12X_SPARSE_INDEXER=1 (plus VLLM_USE_B12X_MOE=1 for MOE_BACKEND=b12x), satisfying b12x_cuda_graph_prewarm_enabled(). A genuine cute.compile inside capture therefore hits raise_if_kernel_resolution_frozen (_lib/compiler.py:2727 / :2777) → KernelResolutionFrozenError.

Where the margin is thin — three independent ways it disappears:

• The freeze does not cover a disk-cache hit. _lib/compiler.py:2659-2679 returns the loaded object before raise_if_kernel_resolution_frozen is ever reached:

      if disk_cache_enabled:
          compiled = _load_cute_compile_from_disk(cache_key)
          if compiled is not None:
              ...
              _memory_cache_put(memory_cache_key, compiled)
              return compiled

  SPARKINFER_COMPILE_DISK_CACHE defaults to "1" (_lib/compiler.py:655-657) and the image pins a persistent SPARKINFER_COMPILE_CACHE_DIR=/cache/jit/${CACHE_FINGERPRINT}/sparkinfer/compile (Dockerfile.vllm-b12x-cu132:1234). So on a warm host cache a first-touch-under-capture takes _load_cute_compile_from_disk → tempfile.TemporaryDirectory + shutil.copy2 + ExternalBinaryModule(...) (a CUDA module load) inside the capture region, silently, with no diagnostic at all.
• Protection (i) is a vLLM upstream default, not a sparkinfer invariant: cudagraph_num_of_warmups is 0 in CompilationConfig, and any embedder of sparkinfer that captures without an eager pass at the same m has zero protection.
• Protection (ii) is gated on VLLM_USE_B12X_*; a config with all of them 0 silently drops the freeze.

Failure mode if it fires

Two sub-cases, honestly separated:

• Disk-cache miss: raise_if_kernel_resolution_frozen fires first (when the freeze is active) → clean KernelResolutionFrozenError before any CUDA work. Without the freeze: a multi-hundred-ms-to-seconds NVVM/ptxas JIT plus module load inside the capture window.
• Disk-cache hit: module load inside capture, freeze bypassed. Whether ExternalBinaryModule construction performs work that CUDA rejects during stream capture (and hence whether cudaStreamEndCapture fails loudly or a subtly broken graph is produced) is not statically provable from this tree — ExternalBinaryModule lives in cutlass.base_dsl.export, outside both repos. Measurement needed (see Verification below). I am not claiming silent corruption.

The project already treats compile-inside-capture as a defect class, but only in offline validation, never at runtime: validation/cutlass_migration/acceptance/w4a16_serving.py:416-418 ("graph capture compiled a CUTLASS kernel"), .../compute_exceptions.py:748-749, .../residual_composite.py:558-560, .../diagnostics/paired/w4a16_serving.py:820.

Severity

bug — defensive. No silent numeric corruption is demonstrated, and no live path exists on the shipped gate or turnkey configs. The impact is a missing invariant on the one unguarded first-use compile that provably executes inside captured graphs, whose entire safety today is supplied by the host framework. Cost when it fires ranges from a loud startup error (best case) to an unaudited CUDA module load inside a capture region (disk-hit case). No memory cost, inside or outside the profiled budget.

Suggested fix

Not the obvious one. Mirroring the planner's guard inside compile_w4a16_fused_moe would be wrong: _prewarm_w4a16_planned_launches deliberately calls it while capturing — _impl.py#L6519-L6520 captures is_capturing = torch.cuda.is_current_stream_capturing() and then at #L6625-L6643 skips only the route-pack launch under capture, keeping the compile_w4a16_fused_moe call at #L6565 on purpose ("The real route-pack launch happens in run_w4a16_moe. During CUDA graph capture this prewarm-only launch would be recorded as useless work"). A guard in the compile function would break that supported path.

Correct placement is the runtime resolve site, where the miss is unintended:

    if fused_launch is None:
        if torch.cuda.is_current_stream_capturing():
            key_probe = _fused_moe_cache_probe(...)   # existing key, no compile
            if key_probe is None:
                raise RuntimeError(
                    "W4A16 fused MoE launch is not resolved for CUDA graph capture "
                    f"(m={m}, moe_block_size={block_size_m}, "
                    f"max_m_blocks={required_m_blocks}); run an eager warmup at this "
                    "token count before capturing"
                )
        fused = compile_w4a16_fused_moe(...)

Raising is right here, not falling back: unlike _get_c_tmp (kernel.py:9164-9184, which raises under capture and allocates in eager) and unlike bf16_gemv/_kernel.py:290 (which has a legitimate cuBLAS fallback), there is no alternative MoE launch to fall back to — the only options are compile or fail, and compiling is what we are trying to prevent. It is also capture-safe (pure host-side dict probe, no CUDA API call, no allocation) and changes neither numerics nor any ABI. Cheapest correct variant: cache-probe first and raise only on a genuine miss, so the guard never rejects a warm key.

Two adjacent items, deliberately out of scope here:

• _lib/compiler.py:2659-2679 should consult raise_if_kernel_resolution_frozen before the disk-cache load, not after, so freeze_kernel_resolution means what its name says.
• Bucket mismatch corroborating [perf] W4A16 launch path + indexer env parsing: full kernel rebuild before every compile-cache probe, prewarmed launches unreachable at runtime, per-call getenv on the paged-indexer prefill route #96 part B: the plan-time prewarm computes capacity_m_blocks from capacity_tokens * num_topk exactly (_impl.py:6219-6226) while the runtime resolve uses the pow2 bucket. Recomputed at the turnkey prefill capacity (m=3072, bs=64, topk=8, E=160): planner capacity_m_blocks = 542, runtime required_m_blocks = 670 (bucket 4096). So the prewarmed key is unreachable from the runtime probe by two independent mechanisms, not just the tile-config pin.

Verification / repro hint

1. Prove the entry point is inside capture (no GPU needed beyond a normal boot): set SPARKINFER_LOG_CUTE_COMPILES_AFTER_ENGINE_START=1 and boot the turnkey; every compile_w4a16_fused_moe disk-hit/miss during Capturing CUDA graphs is logged with its cache key.
2. Prove the latency of the warmup dependency: launch with --compilation-config '{"cudagraph_num_of_warmups":0,...}' (overriding config/vllm.py:1430) plus VLLM_USE_B12X_MOE=0 VLLM_USE_B12X_MHC=0 VLLM_USE_B12X_WO_PROJECTION=0 VLLM_USE_B12X_SPARSE_INDEXER=0 to drop the freeze, and SPARKINFER_COMPILE_DISK_CACHE=0 to force a true compile. Capture should then attempt a CuTe compile for the largest capture size. Observe whether cudaStreamEndCapture fails or a graph is produced — this is the measurement that resolves the "silent broken graph vs. loud error" question above.
3. Unit-level: the harness pattern already exists — assert compile_cache_info()["compile_misses"] is unchanged across a capture, as validation/cutlass_migration/acceptance/w4a16_serving.py:405-418 does; extend it to the fused_launch=None bind path.

Provenance of the observation

First noted (unfiled) as "Latent capture hazard" in a review comment on the unrelated perf issue #96: #96 (comment) — parent issue #96 (W4A16 launch-path per-call cost / unreachable prewarm) is linked for context only; this guard gap is not covered by #96's body.

[malaiwah]

Implemented as PR #102: #102

The fix is deliberately at the runtime resolve boundary, not the compile API: capture may use an already-resolved _FUSED_CACHE entry, but a genuine miss raises before small-M resolution, CuTe JIT, persistent-cache loading, or CUDA module loading. Explicit planner/prewarm calls retain the supported compile behavior.

Verification:

• CPU-isolated r12 W4A16 module: 7 passed, 87 skipped
• regression proves an empty fused cache with capture-required lookup cannot enter the compile path
• Ruff lint: pass
• independent adversarial review: APPROVE (0.98), no code finding
• CodeRabbit: pass

The issue's reachability classification remains unchanged: defensive hardening; no live failure is claimed for the shipped r12 profiles because their capture keys are eagerly warmed. The active production container was not rebuilt or restarted and remains healthy.

[malaiwah]

Completion update: PR #102 now covers every surviving capture-resolution finding: #102

• Runtime capture misses fail before small-M dispatch, CuTe JIT, persistent disk lookup, filesystem/module load, or compilation; in-memory hits remain valid.
• Compiler freeze applies before the persistent-cache path, closing the disk-hit bypass.
• Capture detection follows an explicitly selected non-current CUDA stream while preserving the current-stream fast path.
• Planner prewarm, route packing, preplanned slicing, and runtime resolution share one capacity/key derivation: generic 3072 resolves the 4096/670 bucket everywhere, while Trellis fixed capacity stays exact.
• CPU-isolated final affected suites: 29 passed, 87 skipped; changed-file Ruff passes.

PR #102 is now based on #92's fixed-capacity Trellis branch and is mergeable. No live-GPU result is claimed because the unchanged production service occupies all four GPUs.

[malaiwah]

The capture-resolution acceptance tests pass on final #102 head 2bed880a7e9edbd9f2d976ba1a8ee88c9ba6e338:

• 29 CPU tests passed / 87 skipped; Ruff clean;
• 108 GPU tests passed / 8 skipped, cold and warm;
• capture hit/miss guards, non-current stream handling, generic 3072→4096 resolution, and exact fixed Trellis 3072 passed.

The original handoff head had a relu2 inactive-helper failure; validation added fixes 2c2e220 and 2bed880.

The separate #100 + #102 ordered scratch-reuse blocker is now also resolved. Exact composition cae3aad137bd739881833c064e1912177698156f passes 220 tests, 16 skipped cold/warm, plus 24/24 Compute Sanitizer cases with 0 errors. The repaired composition received independent approval after two earlier candidates were rejected.

Full chronology and evidence: https://github.com/malaiwah/glm52-exl3-vast/blob/d4b0a90/docs/field-review-results/2026-07-30-vast-46335896/UPSTREAM-REPAIR-CAMPAIGN.md#b-sparkinfer-w4a16-scratch-lifetime

Exact r14 deployable-source replay

The immutable r14 SparkInfer release tree was independently reconstructed from blackwell-llm-docker@2464cc03: base 9b852b281250123fe323f63ccb1df3cac0f3bbca plus the release lock produced the published tree 8110e3ea417794bfb08aff1fba20135102e5536b exactly. Applying the accepted #100/#102 repairs produced candidate e2205cba8d78db03427a3945a75a1f647b51ef5a (tree 1d378d599538daecfc1e8c11a44010b27d3ddfe5).

On physical GPU 2 with a fresh candidate-specific extension cache, the complete seven-file union passed 332 passed / 16 intentional skips cold in 237.03 s and identically warm in 10.03 s. This closes the image-source composition boundary for the W4 repair; no new numerical, compile-cache, capture, or mixed-K regression appeared.

Exact source provenance and immutable raw logs: r14 reconstruction ledger, cold log, warm log.

[malaiwah]

The capture-resolution fix is in #102 at 2bed880. Exact-head CPU/GPU and composed sanitizer gates pass, and the final appliance gate found no warm serving regression. The capture-miss safety claim comes from the targeted negative tests, not from the already-warmed serving run. Evidence: https://github.com/malaiwah/glm52-exl3-vast/blob/5c76a2536e7fc9a5f1cb6bf182531889f5385e65/docs/field-review-results/2026-07-30-vast-46335896/UPSTREAM-REPAIR-CAMPAIGN.md. Keeping this issue open until #102 merges.