[RFC] Experimental Qwen3.5 MoE AFD support: CUDA correctness baseline

[shanyulu]

Summary

This RFC proposes an experimental, CUDA-only Qwen3.5 MoE AFD validation lane for the AFD plugin.

The purpose is to establish a reviewed model/architecture baseline before continuing the MUSA work proposed in #173. It is not a request to change the project-wide vLLM alignment policy in #155, and it does not claim production readiness or performance improvement.

A local CUDA implementation has completed deterministic correctness validation for Qwen/Qwen3.5-35B-A3B on vLLM 0.23.0. The implementation follows native vLLM model/decoder/weight-loading lifecycles and disaggregates only at the MoE experts boundary.

Why an RFC before an implementation PR?

Qwen3.5 is a hybrid full-attention + linear-attention/Mamba MoE model. AFD must keep attention/KV and hybrid state on the Attention role while moving only routed/shared expert execution to the FFN role.

Maintainer feedback is requested now on:

1. whether Qwen3.5 belongs in the AFD model-support scope;
2. whether an eager, text-only correctness-first contribution is an acceptable first step;
3. whether the Attention-side router boundary is the preferred design; and
4. whether this experimental vLLM 0.23 validation should be treated solely as an architecture/evidence lane while the project follows the alignment policy in [RFC]: afd-plugin project roadmap #155.

Proposed initial scope

• Model: Qwen/Qwen3.5-35B-A3B, original BF16 checkpoint.
• Backend: NVIDIA CUDA only.
• Topology: single host, Attention GPUs 0–1 / TP=2; FFN GPUs 2–3 / TP=2.
• Runtime: vLLM V1, BF16, eager execution, synchronous NCCL/P2P.
• Routing: gate on Attention; hidden states plus router logits transferred to FFN.
• Correctness only: no performance claim.

Explicit non-goals

• no CUDA Graph, DBO, async communication, microbatching, EP/SP/PP, multi-node, quantisation, or performance tuning;
• no multimodal image/video support;
• no claim of broad Qwen or backend support;
• no MUSA implementation in this contribution.

MUSA remains a separate follow-up track under #173, where the CUDA result can serve as the model-boundary and correctness reference.

Proposed plugin boundary

The proposed adapter keeps native vLLM Qwen control flow and loading authoritative:

native Qwen decoder
  -> native Attention / linear-attention state
  -> native Qwen router on Attention
  -> parameter-free remote-experts proxy
  -> synchronous P2P(hidden_states, 256-wide router logits, layer/stage correlation)
  -> native FusedMoE on FFN (routed experts + shared expert)
  -> synchronous P2P(expert output)
  -> native decoder residual path

Ownership is intentionally strict:

Role	Owns	Must not own
Attention	embeddings, norms, full-attention KV cache, linear-attention/Mamba state, router	routed/shared expert weights or local expert execution
FFN	native FusedMoE, routed experts, shared expert, shared-expert gate	attention backend, KV cache, or hybrid/Mamba state

The adapter is model-specific rather than a copy of Model.forward, DecoderLayer.forward, or the native loader.

CUDA correctness evidence

Environment:

• base commit: 987301208dc0e3597d01e700880094a3603c5bea;
• Python 3.12.3, PyTorch 2.11.0+cu130, vLLM 0.23.0;
• 4 × NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GiB each, driver 580.82.09;
• model integrity verified: 14/14 shards.

Every comparison requires:

• exact generated token-ID equality;
• exact top-5 token-set equality at every decode step; and
• maximum absolute top-5 logprob error of 0.0.

The following gates passed against an equivalently configured colocated TP2 baseline:

Gate	Result
AFD batch 1	PASS, 32/32 decode steps, max error 0.0
Repeated AFD requests	PASS, 32/32 steps, max error 0.0
Alternate prompts	PASS, max error 0.0
Concurrent batch 2, prompt A	PASS, 32/32 steps, max error 0.0
Concurrent batch 2, prompt B	PASS, 32/32 steps, max error 0.0
CPU model-role/package/FFN-runner contracts	PASS, 37 tests
Shutdown cleanup	PASS; no vLLM workers and all four GPUs at 0 MiB

The implementation also asserts that the Attention proxy has no expert parameters and that FFN construction does not instantiate KV-cache, attention, or hybrid/Mamba state.

Known limitations

• Text prompts only. Qwen3.5's checkpoint architecture is multimodal; visual modules are deliberately not constructed or loaded.
• vLLM emits a warning when chunked prefill is disabled for this model. This validation pins eager/no-DBO behavior and records that warning; it is not a claim of general chunked-prefill support.
• The FlashInfer sampler was disabled with VLLM_USE_FLASHINFER_SAMPLER=0 because the local CUDA runtime cannot JIT it for SM120. This does not change model weights, MoE computation, or AFD communication.

Proposed follow-up, subject to feedback

If accepted, the implementation can be proposed in small reviewable units:

1. Qwen3.5 role-aware adapter and typed experts-boundary payload;
2. targeted CPU contracts plus reproducible CUDA correctness recipe/evidence;
3. separately, MUSA eager/synchronous adaptation under [RFC]: Add Moore Threads MUSA backend support for AFD #173.

Questions for maintainers

1. Is this narrow Qwen3.5 CUDA correctness lane in scope for AFD?
2. Is Attention-side router ownership and FFN-side native FusedMoE execution the preferred first boundary?
3. Is text-only, eager/synchronous support acceptable as the initial model contribution?
4. Should the CUDA proof remain an explicitly experimental vLLM 0.23 architecture lane, with any durable support PR rebased onto the project-selected vLLM line?
5. Would you prefer the model adapter and typed payload support in one PR, or split them before the CUDA recipe/tests?

Relationship to existing RFCs

• [RFC]: afd-plugin project roadmap #155 defines the project roadmap and layered support/evidence policy. This RFC proposes a model/backend-specific experimental evidence lane under that policy.
• [RFC]: Add Moore Threads MUSA backend support for AFD #173 proposes MUSA support. The CUDA result here is intended to separate Qwen3.5 model-boundary questions from backend-specific MUSA work before that follow-up begins.

[shanyulu]

Status update: the Qwen3.5 / vLLM 0.23 work remains historical architecture evidence. The active path has moved to Qwen3.6-35B-A3B on exact vLLM 0.26.0, following the direction in #173.

CUDA native TP2 and AFD 2A2F TP2 correctness are complete in #181, stacked on #176. The result is 8/8 with exact token IDs, exact per-step top-5 token sets, and maximum logprob error 0.0.

No v0.23 implementation is being proposed for upstream; this issue is left open as historical context.

[shanyulu]

Qwen3.5/vLLM 0.23 remains historical architecture evidence only. The active CUDA line is Qwen3.6-35B-A3B on vLLM 0.26.0 in #181, rebased on update/v0.26.0. This issue stays open for context; it is not the implementation target and is not closed by #181.

[hsliuustc0106]

#186 PTAL