feat(model): DeepSeek-V4-Flash W6 — sqrtsoftplus + hash-routed MoE host ref + unit gate

The three genuinely-new-vs-V2/V3 MoE pieces of DeepseekV4ForCausalLM, as
portable host (CPU) references + unit gate (additive, SACRED-inert). Only the
NEW scoring + hash-route + clamp are net-new; the shared DeepSeek grouped-GEMM /
256-expert / shared-expert / NVFP4 machinery is REUSED, not re-ported.

New TUs deepseek_v4_moe.{h,cpp} (ported 1:1 with file:line, vLLM primary +
SGLang v0.5.15 cross-ref):
- SqrtSoftplus <- fused_topk_bias_router.py:88 (sqrt(softplus(x)) router score,
  distinct from V2/V3 sigmoid/softmax noaux_tc).
- SqrtSoftplusRouteTopk <- fused_topk_bias_router.py:75-118
  (_topk_softplus_sqrt_torch) + hash branch :100-106 + hash-table wiring
  nvidia/model.py:562-578,:686,:696-717: bias for SELECTION ONLY, weights
  gathered from the UNBIASED scores, the tid2eid token-id->expert HASH lookup
  that BYPASSES top-k, renormalize, routed_scaling_factor.
- ClampedSwiGLU <- activation.py:197-201 (SiluAndMulWithClamp): gate max-only,
  up both-sided, gate*sigmoid(a*gate)*(up+b).

MegaMoE is SM100-only (major==10) so this mirrors the FusedMoE-fallback router
GB10 runs; the device kernels reuse the existing grouped-GEMM and the forward
assembly is a W7 residual.

Gate: test_deepseek_v4_moe 12/12 cases / 716 assertions GREEN (CPU Debug;
new TUs -Wall -Werror -Wextra-clean). Hand-derived literals (sqrt-softplus
composition, bias-flips-selection-but-weight-stays-unbiased, hash bypass,
asymmetric clamp, clamp boundaries, alpha/beta) + from-first-principles
double-precision references. RED-first PROVEN all three levers: drop sqrt
fails 8/493, biased-weight-gather fails 2/181, symmetric gate-clamp fails 2/6;
revert restores 12/12. Honest gate form: host-reference + hand-case +
structural review (fixed-config 167B not constructible at a tiny shape).
Full-model gate stays multi-Spark-blocked (156.7 GiB).

SACRED-inert: no existing forward touched; test_deepseek_v4_mhc 14/14,
test_deepseek_v4_compressor 12/12, test_deepseek_v4_dsa 13/13,
test_deepseek_v4_scaffold 4/4 unchanged. New kernel row
KERNEL-MOE-SQRTSOFTPLUS-HASH (SPIKE); checker count 41->42. Records updated:
model-matrix, kernel-matrix, feature-matrix, roadmap_v1, coordination
(CLAIM-DEEPSEEK-V4-W6), parity-ledger, state, docs/STATUS, docs/BENCHMARKS,
spec §W6. All record checkers rc=0.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

Author: mudler

.agents/coordination.md

@@ -210,7 +210,7 @@ is configured, exactly as upstream loads its draft model on demand.
 | GDN spec segments (metadata + slot-snapshot rollback) | `v1/attention/backends/gdn_attn.py`, `fla/ops/fused_sigmoid_gating.py` | ✅ landed (`SPEC-GDN-SEGMENTS` ACTIVE, I4/I5a/I7) | metadata split + reclassification, `T>1`/IS_SPEC recurrence, conv rollback, k+1 slots, mixed split/merge; bit-exact; consumed by MTP DONE | [specs/mtp-spec-decode.md](specs/mtp-spec-decode.md) (3) |
 | DFlash (block-diffusion drafter) | in-pin + published drafts for our models | 🚧 **spec written** (after MTP) | DGX-Spark community container exists; GDN slot memory at k=15 flagged | [specs/dflash-spec-decode.md](specs/dflash-spec-decode.md) |
 | DeepSeek-V4 MTP (`DeepSeekV4MTP`) | `models/deepseek_v4/nvidia/mtp.py:72`, `registry.py:630` | 📋 **W0-scoped (`CLAIM-DEEPSEEK-V4-SCOPE`, 2026-07-28)** | V4 MTP head = separate `e_proj`/`h_proj` (fp8) + `enorm`/`hnorm` + `hc_head` collapse, `mtp_block` a full V4 decoder (inherits MHC + DSA + sqrtsoftplus/hash MoE). REUSES the landed MTP spec-decode engine loop; only the draft forward is new, gated behind the V4 target model. HW-fit follows the target — CORRECTED 2026-07-28: the `nvidia/DeepSeek-V4-Flash-NVFP4` target is **156.7 GiB** (not ~83 GiB) and does NOT fit ONE GB10, so the V4 target's W1 oracle run is memory-infeasible on a single Spark (see deepseek-v4-flash.md §8) | [specs/deepseek-v4-flash.md](specs/deepseek-v4-flash.md) |
-| DeepSeek-V4 DSA attention + MHC primitives (`DeepseekV4ForCausalLM`) | `models/deepseek_v4/attention.py`, `v1/attention/ops/triton_fp8_mqa_logits.py`, `deepseek_v4/nvidia/ops/o_proj.py`, `deepseek_v4/common/ops/fused_compress_quant_cache.py`, `deepseek_v4/compressor.py`, `model_executor/kernels/mhc/torch.py` | 📋 **W3+W4+W5 primitives LANDED (`CLAIM-DEEPSEEK-V4-W3`/`-W4`/`-W5`, 2026-07-28/29)** | The genuinely-NEW-vs-V2/V3 attention + residual-topology math ported as portable host references + unit-gated. **W3** (`deepseek_v4_dsa.{h,cpp}`, `test_deepseek_v4_dsa` 13/13·38): the DSA "Lightning Indexer" sparse SELECTION (weighted-MQA logit `Σ_h w·ReLU(q·k)` — the per-head ReLU is load-bearing — + causal top-k `index_topk=512` + short-context all-select), per-head attention-**sink** softmax, and grouped output-LoRA (`wo_a` bmm→`wo_b`). **W4** (`deepseek_v4_compressor.{h,cpp}`, `test_deepseek_v4_compressor` 12/12·164; kernel row `KERNEL-ATTN-DSA-COMPRESSOR`): the DSA COMPRESSOR forward (softmax-weighted window POOL, per head-dim column, + save-time APE add + RMSNorm) and the **fp8_ds_mla** KV-cache state layout (448 fp8 NoPE with per-64 UE8M0 power-of-two block scales + 64 bf16 RoPE, 576B stride, 7+1 scale region, + the dequant read). **W5** (`deepseek_v4_mhc.{h,cpp}`, `test_deepseek_v4_mhc` 14/14·125; new kernel row `KERNEL-MHC-SINKHORN`): the Manifold/Markov Hyper-Connections topology — the 20-iteration Sinkhorn-normalized doubly-stochastic mixing matrix + the mHC pre/post mixes (folded attn/ffn RMSNorms) + the hc_head collapse; the W0 "no eager reference" premise is CORRECTED (vLLM ships `mhc/torch.py`; four upstream impls agree on the Sinkhorn), ported 1:1 + gated vs an independent double-precision Sinkhorn derivation, RED-first on iteration-count + axis-swap. Honest gate: hand-derived literals + double-precision references (rel-L2 < 1e-6..1e-4) + RED-first; NOT a dumped-oracle gate (fixed-config 167B, not constructible tiny). SGLang v0.5.15 registers V4 (viable 2nd reference). Residuals: sqrtsoftplus/hash MoE (W6), device kernel + forward integration + strict gate (W7-W8 = multi-Spark) | [specs/deepseek-v4-flash.md](specs/deepseek-v4-flash.md) |
+| DeepSeek-V4 DSA attention + MHC + MoE primitives (`DeepseekV4ForCausalLM`) | `models/deepseek_v4/attention.py`, `v1/attention/ops/triton_fp8_mqa_logits.py`, `deepseek_v4/nvidia/ops/o_proj.py`, `deepseek_v4/common/ops/fused_compress_quant_cache.py`, `deepseek_v4/compressor.py`, `model_executor/kernels/mhc/torch.py`, `fused_moe/router/fused_topk_bias_router.py`, `layers/activation.py` | 📋 **W3+W4+W5+W6 primitives LANDED (`CLAIM-DEEPSEEK-V4-W3`/`-W4`/`-W5`/`-W6`, 2026-07-28/29)** | The genuinely-NEW-vs-V2/V3 attention + residual-topology math ported as portable host references + unit-gated. **W3** (`deepseek_v4_dsa.{h,cpp}`, `test_deepseek_v4_dsa` 13/13·38): the DSA "Lightning Indexer" sparse SELECTION (weighted-MQA logit `Σ_h w·ReLU(q·k)` — the per-head ReLU is load-bearing — + causal top-k `index_topk=512` + short-context all-select), per-head attention-**sink** softmax, and grouped output-LoRA (`wo_a` bmm→`wo_b`). **W4** (`deepseek_v4_compressor.{h,cpp}`, `test_deepseek_v4_compressor` 12/12·164; kernel row `KERNEL-ATTN-DSA-COMPRESSOR`): the DSA COMPRESSOR forward (softmax-weighted window POOL, per head-dim column, + save-time APE add + RMSNorm) and the **fp8_ds_mla** KV-cache state layout (448 fp8 NoPE with per-64 UE8M0 power-of-two block scales + 64 bf16 RoPE, 576B stride, 7+1 scale region, + the dequant read). **W5** (`deepseek_v4_mhc.{h,cpp}`, `test_deepseek_v4_mhc` 14/14·125; new kernel row `KERNEL-MHC-SINKHORN`): the Manifold/Markov Hyper-Connections topology — the 20-iteration Sinkhorn-normalized doubly-stochastic mixing matrix + the mHC pre/post mixes (folded attn/ffn RMSNorms) + the hc_head collapse; the W0 "no eager reference" premise is CORRECTED (vLLM ships `mhc/torch.py`; four upstream impls agree on the Sinkhorn), ported 1:1 + gated vs an independent double-precision Sinkhorn derivation, RED-first on iteration-count + axis-swap. Honest gate: hand-derived literals + double-precision references (rel-L2 < 1e-6..1e-4) + RED-first; NOT a dumped-oracle gate (fixed-config 167B, not constructible tiny). **W6** (`deepseek_v4_moe.{h,cpp}`, `test_deepseek_v4_moe` 12/12·716; new kernel row `KERNEL-MOE-SQRTSOFTPLUS-HASH`): the three genuinely-new MoE pieces — the router score `sqrt(softplus(x))` (distinct from V2/V3 sigmoid/softmax), the router (bias-for-SELECTION-only top-k OR the `tid2eid` token-id→expert HASH lookup that BYPASSES top-k, weights GATHERED from the UNBIASED scores, renormalize, ×routed_scaling_factor), and the asymmetric clamped SwiGLU (`SiluAndMulWithClamp`: gate max-only, up both-sided). REUSE not re-port: the shared grouped-GEMM / 256-expert / shared-expert / NVFP4 machinery is untouched; MegaMoE is SM100-only so GB10 mirrors the FusedMoE fallback. RED-first proven all three levers (drop sqrt 8/493, biased-weight-gather 2/181, symmetric gate-clamp 2/6). Honest gate: hand-derived literals + double-precision references (rel-L2 < 1e-6..1e-4) + RED-first; NOT a dumped-oracle gate (fixed-config 167B, not constructible tiny). SGLang v0.5.15 registers V4 (viable 2nd reference). Residuals: device kernels (reuse the existing grouped-GEMM) + forward integration + strict gate (W7-W8 = multi-Spark) | [specs/deepseek-v4-flash.md](specs/deepseek-v4-flash.md) |
 | DSpark (semi-autoregressive block drafter) | `v1/worker/gpu/spec_decode/dspark/`, `models/{qwen3_dspark,deepseek_v4/nvidia/dspark}.py` | ☐ T1 (**user-promoted**) | DeepSeek-V4 and Qwen3 draft layouts, reduced-vocab mapping, Markov sampling and full-CUDA-graph behavior inventoried as `SPEC-DSPARK`; dedicated spike follows parity/MTP | `planned: specs/dspark-spec-decode.md` |
 | TLI heterogeneous-vocabulary spec decode | `v1/spec_decode/vocab_mapping.py`, `config/speculative.py` | ☐ T1 | target↔draft ID mapping and shared-token constrained logits; current upstream validation is greedy draft only; inventory row `SPEC-TLI` | `planned: specs/tli-spec-decode.md` |
 | ngram (draft-free proposer) | `v1/spec_decode/ngram_proposer.py` | ✅ **DONE (`SPEC-NGRAM`, 2026-07-27)** | Draft-FREE suffix-ngram matcher (KMP-LPS, 1:1 port) wired as a third method reusing the MTP/DFlash verify/reject/`take_draft_token_ids` loop. 27B gate: 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted; unit 19/19; spec-OFF byte-identical (SACRED 235/235 + MTP 9/9 + DFlash 27/27); no new kernel | [specs/spec-decode-breadth-d3.md](specs/spec-decode-breadth-d3.md) |