diff --git a/.claude/skills/run-model-cases/SKILL.md b/.claude/skills/run-model-cases/SKILL.md index f75ca5a5c..85e3c626e 100644 --- a/.claude/skills/run-model-cases/SKILL.md +++ b/.claude/skills/run-model-cases/SKILL.md @@ -12,7 +12,7 @@ model documentation, which explains what each program computes: - [Models](../../../docs/models/index.md) - [Qwen3-14B](../../../docs/models/qwen3_14b/index.md) -- [DeepSeek V4-Flash](../../../docs/models/deepseek_v4_flash_mtp.md) +- [DeepSeek V4-Flash](../../../docs/models/deepseek_v4_flash_mtp/index.md) - [Platforms and Devices](../../../docs/get-started/platforms.md) - [Compile and Runtime Workflow](../../../docs/run-and-validate/compile-runtime-workflow.md) diff --git a/.github/workflows/daily_ci.yml b/.github/workflows/daily_ci.yml index 44b4ced1d..b5231fe41 100644 --- a/.github/workflows/daily_ci.yml +++ b/.github/workflows/daily_ci.yml @@ -484,7 +484,7 @@ jobs: # Real-weight DeepSeek-V4-Flash token generation on 8 Ascend 950 cards: the # models/deepseek_v4_pro drivers with the Flash config (--variant flash), # EP8/TP2, one prefill plus 32 greedy decode steps, driven by - # synthetic_token_loop.py exactly as docs/models/deepseek_v4_pro.md + # synthetic_token_loop.py exactly as docs/models/deepseek_v4_pro/index.md # ("End-to-end token generation") documents. The prefill has a fixed # 128-row capacity while active rows follow the encoded prompt length. It is the only # nightly that exercises the full prompt -> tokens -> text path on real diff --git a/docs/debug-and-tune/index.md b/docs/debug-and-tune/index.md index b693952f4..51c4faf8e 100644 --- a/docs/debug-and-tune/index.md +++ b/docs/debug-and-tune/index.md @@ -16,8 +16,8 @@ then move from the broadest evidence to the narrowest: | Diagnose compile errors, runtime failures, hangs, or missing dependencies | [Debugging](debugging.md) | | Diagnose numerical drift and choose comparison thresholds | [Precision Tuning](precision-tuning.md) | | Measure end-to-end time and inspect the task schedule | [Performance Tuning](performance-tuning.md) | -| See how a full decode path was optimized, in order, with the limit found at each step | [DeepSeek V4 Decode Optimization](deepseek-v4-decode-optimization.md) | -| See how a single-card dense model's kernels were tuned, in the order the work happened | [Qwen3-14B Optimization](qwen3-14b-optimization.md) | +| See how a full decode path was optimized, in order, with the limit found at each step | [DeepSeek V4 Decode Optimization](../models/deepseek_v4_flash_mtp/decode_optimization.md) | +| See how a single-card dense model's kernels were tuned, in the order the work happened | [Qwen3-14B Optimization](../models/qwen3_14b/optimization.md) | | Understand how task edges are formed and when the scheduler issues them | [Dependencies and Scheduling](dependency-and-scheduling.md) | | Fit intermediate tensors in the runtime's ring heaps and measure per-scope peaks | [Ring Heap and Scope Stats](ring-heap-and-scope-stats.md) | | Choose matmul row, N, and K tiles | [Cube Tile Tuning](cube-tile-tuning.md) | diff --git a/docs/index.md b/docs/index.md index 05a405936..0a9237e22 100644 --- a/docs/index.md +++ b/docs/index.md @@ -43,9 +43,9 @@ Begin with the [debugging playbook](debug-and-tune/debugging.md), then choose the [precision](debug-and-tune/precision-tuning.md) or [performance](debug-and-tune/performance-tuning.md) workflow for the problem at hand. -[DeepSeek V4 decode optimization](debug-and-tune/deepseek-v4-decode-optimization.md) +[DeepSeek V4 decode optimization](models/deepseek_v4_flash_mtp/decode_optimization.md) follows one model's decode path end to end and records which levers paid. -[Qwen3-14B optimization](debug-and-tune/qwen3-14b-optimization.md) records how a +[Qwen3-14B optimization](models/qwen3_14b/optimization.md) records how a single-card dense model's kernels were tuned, in the order the work happened. ## Ecosystem diff --git a/docs/models/deepseek_v4_flash_dspark/index.md b/docs/models/deepseek_v4_flash_dspark/index.md new file mode 100644 index 000000000..820eb8bd8 --- /dev/null +++ b/docs/models/deepseek_v4_flash_dspark/index.md @@ -0,0 +1,198 @@ +# DeepSeek V4-Flash, DSpark point + +`models/deepseek_v4_flash_dspark/` is the second deployment point of the same +V4-Flash checkpoint: a wide-batch serving configuration whose speculation comes +from a **DSpark drafter** instead of one MTP layer, and whose attention is +tensor-parallel and context-parallel instead of purely data-parallel. + +The model body — 43 layers, the three attention paths, the 256-expert MoE, the +hyper-connection stack — is the one described on the +[V4-Flash MTP page](../deepseek_v4_flash_mtp/index.md), and both trees read the +same `FLASH` preset. This page covers what the DSpark point changes. + +## Deployment configuration + +[config.py](../../../models/deepseek_v4_flash_dspark/config.py) carries the same +`DeepSeekV4Config` presets as the MTP tree and its own deployment constants +below the presets. + +| Deployment property | Value | +| --- | --- | +| Speculative decoding | DSpark — a three-layer drafter proposes `DSPARK_SPEC_TOKENS = 7` drafts per request, so the target model verifies `S = 8` token rows per step (`DECODE_SEQ`) | +| Decode batch per card | 64 requests → 512 token rows per step (`DECODE_BATCH`, `DECODE_TOKENS`) | +| Decode context length | up to 1,048,576 positions, paged in **32-token** pages (`max_position_embeddings`, `BLOCK_SIZE`) | +| Prefill shape | one packed request stream per CP group, `PREFILL_SEQ = 512` tokens per dispatch; longer prompts arrive as chunks against a resident prefix | +| Platform | Ascend A2/A3, single node | +| Tensor parallelism | `--tp 1/2/4`; the deployment point is TP 4 — the grouped output projection and the LM head are vocab/group-sharded over it | +| Context parallelism | DSA-CP reuses that same physical TP group: each rank owns a slice of the step's token rows, and the attention KV stream is replicated rank-major across the group | +| Expert parallelism | `--ep 2/4/8/16`; the deployment point is EP 16, and each rank holds `256 / ep` routed experts | +| Data parallelism | `DP = 4` groups per node, so the deployment point is 16 cards (`TP * DP`) | +| Quantization | W8A8 INT8, identical to the MTP tree — INT8 weights with FP32 dequant scales, activations quantized per token at the INT8 matmuls | + +`BLOCK_SIZE = 32` sets four page sizes at once: the paged KV cache, the +compressed KV cache, the indexer cache, and — through +`C4A_COMPRESSOR_BLOCK_SIZE = 2` and `C128_COMPRESSOR_BLOCK_SIZE = 8` — both +compressor-state pools. + +### What DSpark changes + +| | MTP point | DSpark point | +| --- | --- | --- | +| Drafting | one MTP layer, 1 draft token | a 3-layer drafter, 7 draft tokens, plus a Markov head | +| Rows per decode step, per card | 4 requests × 2 = 8 | 64 requests × 8 = 512 | +| Attention parallelism | data-parallel; each rank owns its own micro-batch | TP-sharded output projection over a DSA-CP token split | +| Page size | 128 | 32 | +| Context ceiling | `max_position_embeddings` truncated to 16,384 | the checkpoint's own 1,048,576, and the cache capacities are sized from it | + +The wider verify window is the reason for the rest of the table: 512 rows per +step is too much attention work for one card, so the token axis is split across +the CP group and the output projection is sharded along with it. + +## Model structure, top down + +### `decode_fwd` + +[decode_fwd.py](../../../models/deepseek_v4_flash_dspark/decode_fwd.py) hand-unrolls +the 43-layer schedule inside one rank-generic `@pl.jit` kernel, launched per +rank from an `@pl.jit.host` driver — the same shape as the MTP tree's forward, +with each attention and MoE stage in its own `pl.scope()` under +`auto_scope=False`: + +``` +decode_fwd +├── preamble embedding lookup, metadata lowering, CP token all-gather +├── layers 0, 1 decode_swa → moe +├── loop ×20 decode_csa → moe (layers 2, 4, …, 40) +│ decode_hca → moe (layers 3, 5, …, 41) +├── layer 42 decode_csa → moe +└── tail hc_head → rms_norm → lm_head (TP vocab shard) +``` + +Every `decode_{swa,csa,hca}` entry has a `_tp1` twin: the single-rank form runs +the layer without the CP gather and the TP publish, and is what the golden +compares against. The batch is dynamic: `--start-pos` takes one position per +request and its length is the batch, defaulting to the 16 per-rank requests the +MoE token budget is sized for. + +### `prefill_fwd` + +[prefill_fwd.py](../../../models/deepseek_v4_flash_dspark/prefill_fwd.py) mirrors +that structure for a packed prompt: the same per-rank kernel shape, the same +per-stage scopes, `prefill_{swa,hca,csa}` in place of the decode +orchestrations, and the same `hc_head → rms_norm → lm_head` tail. Prompt +sequence lengths that do not divide the CP group are padded rather than +rejected, so one program serves an arbitrary chunk against a resident prefix. + +### Attention under DSA-CP + +The three attention paths are the MTP tree's, re-cut along the token axis: + +``` +decode_swa hc_pre → rmsnorm → qkv_proj_rope → decode_sparse_attn_swa + → decode_o_proj (TP publish) → hc_post +decode_hca … → decode_compressor_ratio128 → decode_sparse_attn_hca → … +decode_csa … → decode_compressor_ratio4 (main, inner) + → decode_indexer → decode_indexer_compressor + → decode_sparse_attn_csa → … +``` + +- [decode_cp_token_allgather.py](../../../models/deepseek_v4_flash_dspark/decode_cp_token_allgather.py) + gathers the CP group's token rows into rank-major order on **every** rank. + Each rank then writes the group's whole KV stream into its own replicated + cache, so a compressor or indexer sees the full context while its queries stay + on their token owner. +- [decode_o_proj.py](../../../models/deepseek_v4_flash_dspark/decode_o_proj.py) + owns the grouped output projection and its TP communication: each rank + dequantizes and projects its own `o_groups` shard, then publishes the result + to the group so every rank leaves the stage with the complete rows. +- The prefill side is the same decomposition over + [prefill_cp_token_allgather.py](../../../models/deepseek_v4_flash_dspark/prefill_cp_token_allgather.py) + and [prefill_o_proj.py](../../../models/deepseek_v4_flash_dspark/prefill_o_proj.py). + +### MoE and output stages + +[moe.py](../../../models/deepseek_v4_flash_dspark/moe.py) is unchanged in shape +from the MTP tree — `gate` produces the top-6 routing and the per-token INT8 +view, `dispatch` / `combine` are the EP collectives, `expert_shared` and +`expert_routed` are the two FFN paths — but it carries `DP * DECODE_TOKENS` +worth of receive capacity, because a DSpark step dispatches 512 rows per card +rather than 8. + +`hc_head` folds the hyper-connection stack back to one hidden row, the final +`rms_norm` normalizes it, and [lm_head.py](../../../models/deepseek_v4_flash_dspark/lm_head.py) +all-gathers the group's hidden rows, projects them against this card's +`vocab / tp` shard, and all-to-alls the logits back to their row owners. + +### The DSpark drafter + +The drafter is a small model of its own, run after the target step over the +target's hidden states: + +``` +dspark_proj main_proj(concat of 3 target layers' hidden) → RMSNorm +dspark_context_kv project the target's token stream into each draft layer's + paged SWA cache (per proposal, decode rows or prompt chunk) +dspark_drafter ×3 hc_pre → rmsnorm → qkv_proj_rope + → dspark_attention → o_proj publish → hc_post + → moe +markov_head low-rank (256) Markov embedding + full-vocabulary logits +dspark_markov lm_head → sequential Markov sampling → confidence head +``` + +- [dspark_proj.py](../../../models/deepseek_v4_flash_dspark/dspark_proj.py) + collapses three target layers' hidden states (`dspark_target_layer_ids`) into + one drafter hidden row. `main_proj` stays BF16: the W8A8 checkpoint quantizes + it only under an FP8 quant method. +- [dspark_attention.py](../../../models/deepseek_v4_flash_dspark/dspark_attention.py) + runs one anchor-first draft query block of 7 rows per request against the + paged sliding window. Every draft row sees the trailing window plus the whole + block through one index list, so there is no causal mask inside the block. +- [dspark_markov.py](../../../models/deepseek_v4_flash_dspark/dspark_markov.py) + emits the 7 drafts sequentially — each step's sampled id feeds the next + through a rank-256 Markov transition — and a sigmoid confidence head scores + the block for the acceptance policy. +- [dspark_prefill.py](../../../models/deepseek_v4_flash_dspark/dspark_prefill.py) + is the drafter's prefill entry: prompt-context KV insertion followed by the + same seven-query proposal. + +The drafter and the target forward are compiled and validated as separate +programs; there is no single entry composing a full target-plus-draft serving +step yet. + +## Status + +Under development, and not wired into `pypto-serving`. Every executable file +carries its own Golden Harness fixture and CI markers, and the daily model +workflow sweeps the directory like any other model tree; the +`decode_fwd` / `prefill_fwd` / `decode_layer` / `prefill_layer` compositions and +the distributed communication oracles are device-only (`ci: no-sim`). + +```bash +python models/deepseek_v4_flash_dspark/decode_layer.py -p a2a3 --tp 2 --ep 2 -d 0,1 +python models/deepseek_v4_flash_dspark/decode_fwd.py -p a2a3 --tp 2 --ep 2 -d 0,1 +python models/deepseek_v4_flash_dspark/dspark_drafter.py -p a2a3 --tp 4 --ep 4 -d 0,1,2,3 +``` + +`--tp` and `--ep` are read at import time, because the shapes they derive +freeze before the kernels are traced; passing a value the module did not import +with is rejected rather than silently ignored. + +## Files + +| Group | Files | +| --- | --- | +| Full forward | [decode_fwd.py](../../../models/deepseek_v4_flash_dspark/decode_fwd.py), [prefill_fwd.py](../../../models/deepseek_v4_flash_dspark/prefill_fwd.py) | +| Layer composition | [decode_layer.py](../../../models/deepseek_v4_flash_dspark/decode_layer.py), [prefill_layer.py](../../../models/deepseek_v4_flash_dspark/prefill_layer.py) | +| DSpark drafter | [dspark_drafter.py](../../../models/deepseek_v4_flash_dspark/dspark_drafter.py), [dspark_prefill.py](../../../models/deepseek_v4_flash_dspark/dspark_prefill.py), [dspark_proj.py](../../../models/deepseek_v4_flash_dspark/dspark_proj.py), [dspark_attention.py](../../../models/deepseek_v4_flash_dspark/dspark_attention.py), [dspark_context_kv.py](../../../models/deepseek_v4_flash_dspark/dspark_context_kv.py) | +| DSpark sampling | [dspark_markov.py](../../../models/deepseek_v4_flash_dspark/dspark_markov.py), [markov_head.py](../../../models/deepseek_v4_flash_dspark/markov_head.py) | +| Decode attention orchestration | [decode_swa.py](../../../models/deepseek_v4_flash_dspark/decode_swa.py), [decode_csa.py](../../../models/deepseek_v4_flash_dspark/decode_csa.py), [decode_hca.py](../../../models/deepseek_v4_flash_dspark/decode_hca.py) | +| Decode sparse attention | [decode_sparse_attn_swa.py](../../../models/deepseek_v4_flash_dspark/decode_sparse_attn_swa.py), [decode_sparse_attn_csa.py](../../../models/deepseek_v4_flash_dspark/decode_sparse_attn_csa.py), [decode_sparse_attn_hca.py](../../../models/deepseek_v4_flash_dspark/decode_sparse_attn_hca.py) | +| Decode compressors and indexer | [decode_compressor_ratio4.py](../../../models/deepseek_v4_flash_dspark/decode_compressor_ratio4.py), [decode_compressor_ratio128.py](../../../models/deepseek_v4_flash_dspark/decode_compressor_ratio128.py), [decode_indexer.py](../../../models/deepseek_v4_flash_dspark/decode_indexer.py), [decode_indexer_compressor.py](../../../models/deepseek_v4_flash_dspark/decode_indexer_compressor.py) | +| Prefill attention and cache | [prefill_swa.py](../../../models/deepseek_v4_flash_dspark/prefill_swa.py), [prefill_csa.py](../../../models/deepseek_v4_flash_dspark/prefill_csa.py), [prefill_hca.py](../../../models/deepseek_v4_flash_dspark/prefill_hca.py), [prefill_sparse_attn.py](../../../models/deepseek_v4_flash_dspark/prefill_sparse_attn.py), [prefill_compressor_ratio4.py](../../../models/deepseek_v4_flash_dspark/prefill_compressor_ratio4.py), [prefill_compressor_ratio128.py](../../../models/deepseek_v4_flash_dspark/prefill_compressor_ratio128.py), [prefill_indexer.py](../../../models/deepseek_v4_flash_dspark/prefill_indexer.py), [prefill_indexer_compressor.py](../../../models/deepseek_v4_flash_dspark/prefill_indexer_compressor.py) | +| Output projection and CP transport | [decode_o_proj.py](../../../models/deepseek_v4_flash_dspark/decode_o_proj.py), [prefill_o_proj.py](../../../models/deepseek_v4_flash_dspark/prefill_o_proj.py), [decode_cp_token_allgather.py](../../../models/deepseek_v4_flash_dspark/decode_cp_token_allgather.py), [prefill_cp_token_allgather.py](../../../models/deepseek_v4_flash_dspark/prefill_cp_token_allgather.py) | +| Shared transforms | [rmsnorm.py](../../../models/deepseek_v4_flash_dspark/rmsnorm.py), [qkv_proj_rope.py](../../../models/deepseek_v4_flash_dspark/qkv_proj_rope.py), [hc_pre.py](../../../models/deepseek_v4_flash_dspark/hc_pre.py), [hc_post.py](../../../models/deepseek_v4_flash_dspark/hc_post.py), [hc_head.py](../../../models/deepseek_v4_flash_dspark/hc_head.py), [rope_interleave.py](../../../models/deepseek_v4_flash_dspark/rope_interleave.py), [lookup_embedding.py](../../../models/deepseek_v4_flash_dspark/lookup_embedding.py) | +| MoE and output | [moe.py](../../../models/deepseek_v4_flash_dspark/moe.py), [gate.py](../../../models/deepseek_v4_flash_dspark/gate.py), [expert_shared.py](../../../models/deepseek_v4_flash_dspark/expert_shared.py), [expert_routed.py](../../../models/deepseek_v4_flash_dspark/expert_routed.py), [lm_head.py](../../../models/deepseek_v4_flash_dspark/lm_head.py) | +| Metadata and host helpers | [decode_metadata.py](../../../models/deepseek_v4_flash_dspark/decode_metadata.py), [prefill_metadata.py](../../../models/deepseek_v4_flash_dspark/prefill_metadata.py), [config.py](../../../models/deepseek_v4_flash_dspark/config.py), [utils.py](../../../models/deepseek_v4_flash_dspark/utils.py) | + +`config.py`, `utils.py`, `rope_interleave.py`, and `prefill_o_proj.py` have no +`__main__` block: they are imported rather than run. diff --git a/docs/debug-and-tune/deepseek-v4-decode-optimization.md b/docs/models/deepseek_v4_flash_mtp/decode_optimization.md similarity index 94% rename from docs/debug-and-tune/deepseek-v4-decode-optimization.md rename to docs/models/deepseek_v4_flash_mtp/decode_optimization.md index 351ab8b95..c9a577246 100644 --- a/docs/debug-and-tune/deepseek-v4-decode-optimization.md +++ b/docs/models/deepseek_v4_flash_mtp/decode_optimization.md @@ -1,17 +1,17 @@ # DeepSeek V4 Decode Optimization This page is a case study rather than a reference. It follows -[`models/deepseek_v4_flash_mtp/`](../../models/deepseek_v4_flash_mtp/) — a +[`models/deepseek_v4_flash_mtp/`](../../../models/deepseek_v4_flash_mtp/) — a 43-layer DeepSeek V4-Flash build with MTP speculative decoding, W8A8 quantization, three attention paths, and a 256-expert MoE — from its first kernels to its current state, and records which levers moved the number, which did not, and what each one cost. The mechanisms themselves live elsewhere: -[Performance Tuning](performance-tuning.md) for how to measure and capture, -[Cube Tile Tuning](cube-tile-tuning.md) for choosing tiles, -[Dependencies and Scheduling](dependency-and-scheduling.md) for the task graph -and the scheduler, and [Precision Tuning](precision-tuning.md) for thresholds +[Performance Tuning](../../debug-and-tune/performance-tuning.md) for how to measure and capture, +[Cube Tile Tuning](../../debug-and-tune/cube-tile-tuning.md) for choosing tiles, +[Dependencies and Scheduling](../../debug-and-tune/dependency-and-scheduling.md) for the task graph +and the scheduler, and [Precision Tuning](../../debug-and-tune/precision-tuning.md) for thresholds and rounding. Read those for *how*; read this for *in what order, and what to expect*. @@ -50,7 +50,7 @@ The kernel boundaries are not invented. They are read off the official HuggingFace **DeepSeek-V4-Flash** torch implementation — the modeling code that ships with the checkpoint is the specification, and each kernel entry corresponds to a span of it worth scheduling as one unit. -[config.py](../../models/deepseek_v4_flash_mtp/config.py)'s `FLASH` preset +[config.py](../../../models/deepseek_v4_flash_mtp/config.py)'s `FLASH` preset mirrors that checkpoint's `config.json` field for field, so a shape or hyper-parameter question is answered by the reference rather than guessed. @@ -67,7 +67,7 @@ Three things follow from partitioning along the reference's own structure: from that checkpoint — see the fixture comments in the compressor, indexer and expert modules — so a kernel is exercised at the distribution it will actually see. See - [Precision Tuning](precision-tuning.md#8-test-with-real-weights-and-matched-data-distribution). + [Precision Tuning](../../debug-and-tune/precision-tuning.md#8-test-with-real-weights-and-matched-data-distribution). ### Freeze the deployment contract first @@ -106,7 +106,7 @@ algebraically equivalent reorder still changes the last bits. A golden that follows torch's natural order while the kernel accumulates in tile order does not report a tolerance; it reports noise, and that noise hides the real error the moment one appears. The mechanics are in -[Precision Tuning](precision-tuning.md#2-make-the-kernel-and-golden-implementations-identical). +[Precision Tuning](../../debug-and-tune/precision-tuning.md#2-make-the-kernel-and-golden-implementations-identical). The consequence for optimization work is the rule this section exists to establish: @@ -171,7 +171,7 @@ non-degenerate `stage=2` pipeline, lifting L0C occupancy from 25 % to 50 %. `qproj_matmul` went 56.3 µs → 36.0 µs **with no change in task count** (#718). The walls that bound this, all hit repeatedly (see -[Cube Tile Tuning](cube-tile-tuning.md#model-the-three-practical-constraints)): +[Cube Tile Tuning](../../debug-and-tune/cube-tile-tuning.md#model-the-three-practical-constraints)): | Wall | Value | What it forced here | |---|---|---| @@ -255,7 +255,7 @@ weight L1 tile, freeing Mat for an N fragment of 256. - **Check PMU before pipelining.** The gate/up K loop is MTE2-bound (~80 %) and gains from a pipeline; the w2 K loop is scalar-bound (~97 %) and was deliberately left serial (#473). See - [Performance Tuning](performance-tuning.md#4-read-pmu-utilization). + [Performance Tuning](../../debug-and-tune/performance-tuning.md#4-read-pmu-utilization). - `pl.split(UP_DOWN)` fixes vector stragglers in mixed regions. In `proj_b` the INT8 GEMM finished early and the whole region's wall was set by whichever of the two vector lanes got the larger share of the dequant epilogue; splitting @@ -424,7 +424,7 @@ are gone (#985). Once the arithmetic is mined out, dispatch count and graph shape become first-order. The mechanisms are documented in -[Dependencies and Scheduling](dependency-and-scheduling.md); what follows is +[Dependencies and Scheduling](../../debug-and-tune/dependency-and-scheduling.md); what follows is what they bought here. ### 3.1 Remove tasks and barriers @@ -466,7 +466,7 @@ It costs a dispatch hop, and it costs a source change: the producer must switch to the `with pl.spmd(...) as tid` capture form, because the `for ... in` form yields no TaskId. Standalone entries with no producer pass an empty `task_dummy(deps=[])`. Both idioms are catalogued in -[Deliberately delaying a task](dependency-and-scheduling.md#deliberately-delaying-a-task). +[Deliberately delaying a task](../../debug-and-tune/dependency-and-scheduling.md#deliberately-delaying-a-task). ### 3.4 Delete redundant edges, and anchor waits correctly @@ -496,7 +496,7 @@ per wave, so in a multi-layer forward a faster rank's next-epoch rows could satisfy a slower rank's current expectation. Withdrawn in full (#975, #978). Two lessons, both general: **core-time gains that do not become wall-time gains -are not gains** (see [Performance Tuning](performance-tuning.md)), and **any +are not gains** (see [Performance Tuning](../../debug-and-tune/performance-tuning.md)), and **any counter-based handshake reused across waves needs an epoch dimension.** --- @@ -588,13 +588,13 @@ value. That is what makes it safe to tune aggressively. ## See also -- [Performance Tuning](performance-tuning.md) — measurement, capture, and the +- [Performance Tuning](../../debug-and-tune/performance-tuning.md) — measurement, capture, and the L2 / L1 / L0 tuning rules -- [Cube Tile Tuning](cube-tile-tuning.md) — choosing row, N and K tiles against +- [Cube Tile Tuning](../../debug-and-tune/cube-tile-tuning.md) — choosing row, N and K tiles against the compiler's memory report -- [Dependencies and Scheduling](dependency-and-scheduling.md) — how edges form, +- [Dependencies and Scheduling](../../debug-and-tune/dependency-and-scheduling.md) — how edges form, when the scheduler issues, early dispatch, and dummy-task idioms -- [Precision Tuning](precision-tuning.md) — rounding modes, dtype alignment, and +- [Precision Tuning](../../debug-and-tune/precision-tuning.md) — rounding modes, dtype alignment, and threshold selection -- [DeepSeek V4-Flash (MTP)](../models/deepseek_v4_flash_mtp.md) — the model this +- [DeepSeek V4-Flash (MTP)](index.md) — the model this page follows, top down diff --git a/docs/models/deepseek_v4_flash_mtp.md b/docs/models/deepseek_v4_flash_mtp/index.md similarity index 70% rename from docs/models/deepseek_v4_flash_mtp.md rename to docs/models/deepseek_v4_flash_mtp/index.md index dc98df6ab..439fd257e 100644 --- a/docs/models/deepseek_v4_flash_mtp.md +++ b/docs/models/deepseek_v4_flash_mtp/index.md @@ -2,12 +2,15 @@ `models/deepseek_v4_flash_mtp/` is the reference V4-Flash tree: the operators, the single-layer compositions, and the prefill/decode full forwards. +[V4-Flash DSpark](../deepseek_v4_flash_dspark/index.md) is the same checkpoint at a +second deployment point — wider batch, seven draft tokens, context-parallel +attention. ## Deployment configuration The tree implements the HuggingFace **DeepSeek-V4-Flash** checkpoint — the `FLASH` preset in -[config.py](../../models/deepseek_v4_flash_mtp/config.py) mirrors that model's +[config.py](../../../models/deepseek_v4_flash_mtp/config.py) mirrors that model's `config.json` field for field, and `config.py` is a per-directory singleton that every kernel imports as a bare sibling module. @@ -65,7 +68,7 @@ hyper-connection stack is 4 streams wide (`hc_mult = 4`). ### `decode_fwd` -[decode_fwd.py](../../models/deepseek_v4_flash_mtp/decode_fwd.py) hand-unrolls +[decode_fwd.py](../../../models/deepseek_v4_flash_mtp/decode_fwd.py) hand-unrolls the layer schedule inside one rank-generic `@pl.jit` kernel, launched once per EP rank from an `@pl.jit.host` driver: @@ -84,7 +87,7 @@ three compressor states) are passed in flat and sliced per layer. ### `prefill_fwd` -[prefill_fwd.py](../../models/deepseek_v4_flash_mtp/prefill_fwd.py) mirrors +[prefill_fwd.py](../../../models/deepseek_v4_flash_mtp/prefill_fwd.py) mirrors that structure for a packed prompt: the same per-rank kernel shape, the same per-stage scopes, `prefill_{swa,hca,csa}` in place of the decode orchestrations, and the same `hc_head → rms_norm → lm_head` tail over selected @@ -92,7 +95,7 @@ hidden rows. ### `decode_fwd_mtp` -[decode_fwd_mtp.py](../../models/deepseek_v4_flash_mtp/decode_fwd_mtp.py) is +[decode_fwd_mtp.py](../../../models/deepseek_v4_flash_mtp/decode_fwd_mtp.py) is the third top-level composition: it chains the main decode forward, the draft verification, and the MTP decode layer into one serving step. Its device-only CLI fixture composes the standalone forward and MTP tensor fixtures with a @@ -142,7 +145,7 @@ decode_csa hc_pre → rmsnorm → qkv_proj_rope ### MoE stage -[moe.py](../../models/deepseek_v4_flash_mtp/moe.py) is one distributed +[moe.py](../../../models/deepseek_v4_flash_mtp/moe.py) is one distributed single-layer program that `decode_fwd`, `prefill_fwd`, the layer harnesses, and the MTP entries all call. `gate` is RMSNorm + router + top-k + normalize and also produces the per-token INT8 view; `dispatch` and `combine` are the EP @@ -180,7 +183,7 @@ counterpart used by the test fixtures. ## How it was optimized -[DeepSeek V4 decode optimization](../debug-and-tune/deepseek-v4-decode-optimization.md) +[DeepSeek V4 decode optimization](decode_optimization.md) follows the tuning of this decode path in order — contracts and golden first, then the general tiling / parallelism / fusion levers, the attention, hyper-connection, MoE and router rewrites, scheduling, and finally @@ -190,18 +193,18 @@ serving-level residency and lowering — with the limit measured at each step. | Group | Files | | --- | --- | -| Full forward | [decode_fwd.py](../../models/deepseek_v4_flash_mtp/decode_fwd.py), [prefill_fwd.py](../../models/deepseek_v4_flash_mtp/prefill_fwd.py), [decode_fwd_mtp.py](../../models/deepseek_v4_flash_mtp/decode_fwd_mtp.py) | -| Layer composition | [decode_layer.py](../../models/deepseek_v4_flash_mtp/decode_layer.py), [prefill_layer.py](../../models/deepseek_v4_flash_mtp/prefill_layer.py) | -| MTP | [decode_mtp.py](../../models/deepseek_v4_flash_mtp/decode_mtp.py), [prefill_mtp.py](../../models/deepseek_v4_flash_mtp/prefill_mtp.py), [mtp_projection.py](../../models/deepseek_v4_flash_mtp/mtp_projection.py) | -| Decode attention orchestration | [decode_swa.py](../../models/deepseek_v4_flash_mtp/decode_swa.py), [decode_csa.py](../../models/deepseek_v4_flash_mtp/decode_csa.py), [decode_hca.py](../../models/deepseek_v4_flash_mtp/decode_hca.py) | -| Decode sparse attention (fused o-proj) | [decode_sparse_attn_swa.py](../../models/deepseek_v4_flash_mtp/decode_sparse_attn_swa.py), [decode_sparse_attn_csa.py](../../models/deepseek_v4_flash_mtp/decode_sparse_attn_csa.py), [decode_sparse_attn_hca.py](../../models/deepseek_v4_flash_mtp/decode_sparse_attn_hca.py) | -| Decode compressors and indexer | [decode_compressor_ratio4.py](../../models/deepseek_v4_flash_mtp/decode_compressor_ratio4.py), [decode_compressor_ratio128.py](../../models/deepseek_v4_flash_mtp/decode_compressor_ratio128.py), [decode_indexer.py](../../models/deepseek_v4_flash_mtp/decode_indexer.py), [decode_indexer_compressor.py](../../models/deepseek_v4_flash_mtp/decode_indexer_compressor.py) | -| Prefill attention and cache | [prefill_swa.py](../../models/deepseek_v4_flash_mtp/prefill_swa.py), [prefill_csa.py](../../models/deepseek_v4_flash_mtp/prefill_csa.py), [prefill_hca.py](../../models/deepseek_v4_flash_mtp/prefill_hca.py), [prefill_sparse_attn.py](../../models/deepseek_v4_flash_mtp/prefill_sparse_attn.py), [prefill_compressor_ratio4.py](../../models/deepseek_v4_flash_mtp/prefill_compressor_ratio4.py), [prefill_compressor_ratio128.py](../../models/deepseek_v4_flash_mtp/prefill_compressor_ratio128.py), [prefill_indexer.py](../../models/deepseek_v4_flash_mtp/prefill_indexer.py), [prefill_indexer_compressor.py](../../models/deepseek_v4_flash_mtp/prefill_indexer_compressor.py) | -| Shared transforms | [rmsnorm.py](../../models/deepseek_v4_flash_mtp/rmsnorm.py), [qkv_proj_rope.py](../../models/deepseek_v4_flash_mtp/qkv_proj_rope.py), [hc_pre.py](../../models/deepseek_v4_flash_mtp/hc_pre.py), [hc_post.py](../../models/deepseek_v4_flash_mtp/hc_post.py), [hc_head.py](../../models/deepseek_v4_flash_mtp/hc_head.py), [rope_interleave.py](../../models/deepseek_v4_flash_mtp/rope_interleave.py), [lookup_embedding.py](../../models/deepseek_v4_flash_mtp/lookup_embedding.py) | -| MoE and output | [moe.py](../../models/deepseek_v4_flash_mtp/moe.py), [gate.py](../../models/deepseek_v4_flash_mtp/gate.py), [expert_shared.py](../../models/deepseek_v4_flash_mtp/expert_shared.py), [expert_routed.py](../../models/deepseek_v4_flash_mtp/expert_routed.py), [lm_head.py](../../models/deepseek_v4_flash_mtp/lm_head.py) | -| Metadata and host helpers | [decode_prepare.py](../../models/deepseek_v4_flash_mtp/decode_prepare.py), [config.py](../../models/deepseek_v4_flash_mtp/config.py), [utils.py](../../models/deepseek_v4_flash_mtp/utils.py) | +| Full forward | [decode_fwd.py](../../../models/deepseek_v4_flash_mtp/decode_fwd.py), [prefill_fwd.py](../../../models/deepseek_v4_flash_mtp/prefill_fwd.py), [decode_fwd_mtp.py](../../../models/deepseek_v4_flash_mtp/decode_fwd_mtp.py) | +| Layer composition | [decode_layer.py](../../../models/deepseek_v4_flash_mtp/decode_layer.py), [prefill_layer.py](../../../models/deepseek_v4_flash_mtp/prefill_layer.py) | +| MTP | [decode_mtp.py](../../../models/deepseek_v4_flash_mtp/decode_mtp.py), [prefill_mtp.py](../../../models/deepseek_v4_flash_mtp/prefill_mtp.py), [mtp_projection.py](../../../models/deepseek_v4_flash_mtp/mtp_projection.py) | +| Decode attention orchestration | [decode_swa.py](../../../models/deepseek_v4_flash_mtp/decode_swa.py), [decode_csa.py](../../../models/deepseek_v4_flash_mtp/decode_csa.py), [decode_hca.py](../../../models/deepseek_v4_flash_mtp/decode_hca.py) | +| Decode sparse attention (fused o-proj) | [decode_sparse_attn_swa.py](../../../models/deepseek_v4_flash_mtp/decode_sparse_attn_swa.py), [decode_sparse_attn_csa.py](../../../models/deepseek_v4_flash_mtp/decode_sparse_attn_csa.py), [decode_sparse_attn_hca.py](../../../models/deepseek_v4_flash_mtp/decode_sparse_attn_hca.py) | +| Decode compressors and indexer | [decode_compressor_ratio4.py](../../../models/deepseek_v4_flash_mtp/decode_compressor_ratio4.py), [decode_compressor_ratio128.py](../../../models/deepseek_v4_flash_mtp/decode_compressor_ratio128.py), [decode_indexer.py](../../../models/deepseek_v4_flash_mtp/decode_indexer.py), [decode_indexer_compressor.py](../../../models/deepseek_v4_flash_mtp/decode_indexer_compressor.py) | +| Prefill attention and cache | [prefill_swa.py](../../../models/deepseek_v4_flash_mtp/prefill_swa.py), [prefill_csa.py](../../../models/deepseek_v4_flash_mtp/prefill_csa.py), [prefill_hca.py](../../../models/deepseek_v4_flash_mtp/prefill_hca.py), [prefill_sparse_attn.py](../../../models/deepseek_v4_flash_mtp/prefill_sparse_attn.py), [prefill_compressor_ratio4.py](../../../models/deepseek_v4_flash_mtp/prefill_compressor_ratio4.py), [prefill_compressor_ratio128.py](../../../models/deepseek_v4_flash_mtp/prefill_compressor_ratio128.py), [prefill_indexer.py](../../../models/deepseek_v4_flash_mtp/prefill_indexer.py), [prefill_indexer_compressor.py](../../../models/deepseek_v4_flash_mtp/prefill_indexer_compressor.py) | +| Shared transforms | [rmsnorm.py](../../../models/deepseek_v4_flash_mtp/rmsnorm.py), [qkv_proj_rope.py](../../../models/deepseek_v4_flash_mtp/qkv_proj_rope.py), [hc_pre.py](../../../models/deepseek_v4_flash_mtp/hc_pre.py), [hc_post.py](../../../models/deepseek_v4_flash_mtp/hc_post.py), [hc_head.py](../../../models/deepseek_v4_flash_mtp/hc_head.py), [rope_interleave.py](../../../models/deepseek_v4_flash_mtp/rope_interleave.py), [lookup_embedding.py](../../../models/deepseek_v4_flash_mtp/lookup_embedding.py) | +| MoE and output | [moe.py](../../../models/deepseek_v4_flash_mtp/moe.py), [gate.py](../../../models/deepseek_v4_flash_mtp/gate.py), [expert_shared.py](../../../models/deepseek_v4_flash_mtp/expert_shared.py), [expert_routed.py](../../../models/deepseek_v4_flash_mtp/expert_routed.py), [lm_head.py](../../../models/deepseek_v4_flash_mtp/lm_head.py) | +| Metadata and host helpers | [decode_prepare.py](../../../models/deepseek_v4_flash_mtp/decode_prepare.py), [config.py](../../../models/deepseek_v4_flash_mtp/config.py), [utils.py](../../../models/deepseek_v4_flash_mtp/utils.py) | `config.py`, `utils.py`, `rope_interleave.py`, and `decode_prepare.py` have no `__main__` block: they are imported rather than run. Executable compositions, including `decode_fwd_mtp.py`, are scheduled by the -[daily model workflow](../../.github/workflows/daily_ci.yml). +[daily model workflow](../../../.github/workflows/daily_ci.yml). diff --git a/docs/models/deepseek_v4_pro.md b/docs/models/deepseek_v4_pro/index.md similarity index 78% rename from docs/models/deepseek_v4_pro.md rename to docs/models/deepseek_v4_pro/index.md index c55ad63a3..5d0f8ca28 100644 --- a/docs/models/deepseek_v4_pro.md +++ b/docs/models/deepseek_v4_pro/index.md @@ -6,7 +6,7 @@ and compile time with `DEEPSEEK_V4_VARIANT=pro|flash` or `--variant pro|flash`. ## Deployment configuration -The `PRO` and `FLASH` presets in [config.py](../../models/deepseek_v4_pro/config.py) +The `PRO` and `FLASH` presets in [config.py](../../../models/deepseek_v4_pro/config.py) define the architecture-specific shapes and layer schedules. Pro remains the default so existing operator entry points and DailyCI keep their prior behavior. @@ -32,8 +32,8 @@ constant if a case needs a longer context. Native MXFP8-MXFP4 is not implemented yet. The tracked kernels run an INT8 stand-in with the same tensor split as -[V4-Flash](deepseek_v4_flash_mtp.md#what-is-quantized): `gen_routed_weight` in -[expert_routed.py](../../models/deepseek_v4_pro/expert_routed.py) re-quantizes +[V4-Flash](../deepseek_v4_flash_mtp/index.md#what-is-quantized): `gen_routed_weight` in +[expert_routed.py](../../../models/deepseek_v4_pro/expert_routed.py) re-quantizes off the MXFP4 grid into INT8 rather than feeding the cube MXFP4 weights. ### Model shape and layer schedule @@ -75,7 +75,7 @@ prefill_fwd same schedule with prefill_attention_{hca,csa} → moe, ``` Both forwards finish with the final norm and LM-head sampling. The standalone -[lm_head.py](../../models/deepseek_v4_pro/lm_head.py) entry point validates that +[lm_head.py](../../../models/deepseek_v4_pro/lm_head.py) entry point validates that distributed tail separately. ### One layer @@ -174,7 +174,7 @@ follows the prompt length). ### End-to-end token generation -[synthetic_token_loop.py](../../models/deepseek_v4_pro/synthetic_token_loop.py) +[synthetic_token_loop.py](../../../models/deepseek_v4_pro/synthetic_token_loop.py) drives the full prompt-to-text path on real weights: the prompt is encoded with the checkpoint's `tokenizer.json` (BOS prepended unless `--no-bos`), the resident session runs one prefill plus `--decode-steps` greedy decode @@ -226,7 +226,7 @@ The prefill and decode RoPE paths use fixed even/odd lane gather and scatter operations for adjacent-lane permutations instead of synthesizing tile-local index tensors. -The [daily model workflow](../../.github/workflows/daily_ci.yml) runs this +The [daily model workflow](../../../.github/workflows/daily_ci.yml) runs this EP8 loop nightly on the A5 runner (job `e2e-flash-a5`: real DeepSeek-V4-Flash weights, fixed 128-row prefill capacity with active rows set from the prompt, and 32 greedy decode steps from "The capital of France is") and @@ -255,19 +255,19 @@ as exactly what it was. | Group | Files | | --- | --- | -| Full forward | [decode_fwd.py](../../models/deepseek_v4_pro/decode_fwd.py), [prefill_fwd.py](../../models/deepseek_v4_pro/prefill_fwd.py) | -| Layer composition | [decode_layer.py](../../models/deepseek_v4_pro/decode_layer.py), [prefill_layer.py](../../models/deepseek_v4_pro/prefill_layer.py) | -| MTP | [decode_mtp.py](../../models/deepseek_v4_pro/decode_mtp.py), [prefill_mtp.py](../../models/deepseek_v4_pro/prefill_mtp.py), [mtp_projection.py](../../models/deepseek_v4_pro/mtp_projection.py) | -| Decode attention orchestration | [decode_attention_swa.py](../../models/deepseek_v4_pro/decode_attention_swa.py), [decode_attention_csa.py](../../models/deepseek_v4_pro/decode_attention_csa.py), [decode_attention_hca.py](../../models/deepseek_v4_pro/decode_attention_hca.py) | -| Decode sparse attention (fused o-proj) | [decode_sparse_attn.py](../../models/deepseek_v4_pro/decode_sparse_attn.py), [decode_sparse_attn_swa.py](../../models/deepseek_v4_pro/decode_sparse_attn_swa.py), [decode_sparse_attn_hca.py](../../models/deepseek_v4_pro/decode_sparse_attn_hca.py) | -| Decode compressors and indexer | [decode_compressor_ratio4.py](../../models/deepseek_v4_pro/decode_compressor_ratio4.py), [decode_compressor_ratio128.py](../../models/deepseek_v4_pro/decode_compressor_ratio128.py), [decode_indexer.py](../../models/deepseek_v4_pro/decode_indexer.py), [decode_indexer_compressor.py](../../models/deepseek_v4_pro/decode_indexer_compressor.py) | -| Prefill attention and cache | [prefill_attention_swa.py](../../models/deepseek_v4_pro/prefill_attention_swa.py), [prefill_attention_csa.py](../../models/deepseek_v4_pro/prefill_attention_csa.py), [prefill_attention_hca.py](../../models/deepseek_v4_pro/prefill_attention_hca.py), [prefill_sparse_attn.py](../../models/deepseek_v4_pro/prefill_sparse_attn.py), [prefill_compressor_ratio4.py](../../models/deepseek_v4_pro/prefill_compressor_ratio4.py), [prefill_compressor_ratio128.py](../../models/deepseek_v4_pro/prefill_compressor_ratio128.py), [prefill_indexer.py](../../models/deepseek_v4_pro/prefill_indexer.py), [prefill_indexer_compressor.py](../../models/deepseek_v4_pro/prefill_indexer_compressor.py) | -| Shared transforms | [rmsnorm.py](../../models/deepseek_v4_pro/rmsnorm.py), [qkv_proj_rope.py](../../models/deepseek_v4_pro/qkv_proj_rope.py), [hc_pre.py](../../models/deepseek_v4_pro/hc_pre.py), [hc_post.py](../../models/deepseek_v4_pro/hc_post.py), [hc_head.py](../../models/deepseek_v4_pro/hc_head.py) | -| MoE and output | [moe.py](../../models/deepseek_v4_pro/moe.py), [gate.py](../../models/deepseek_v4_pro/gate.py), [expert_shared.py](../../models/deepseek_v4_pro/expert_shared.py), [expert_routed.py](../../models/deepseek_v4_pro/expert_routed.py), [lm_head.py](../../models/deepseek_v4_pro/lm_head.py) | -| Metadata and host helpers | [config.py](../../models/deepseek_v4_pro/config.py), [decode_metadata.py](../../models/deepseek_v4_pro/decode_metadata.py), [rope_tables.py](../../models/deepseek_v4_pro/rope_tables.py) | -| Real-weight loading | [utils.py](../../models/deepseek_v4_pro/utils.py) | -| Token loop | [synthetic_token_loop.py](../../models/deepseek_v4_pro/synthetic_token_loop.py) | +| Full forward | [decode_fwd.py](../../../models/deepseek_v4_pro/decode_fwd.py), [prefill_fwd.py](../../../models/deepseek_v4_pro/prefill_fwd.py) | +| Layer composition | [decode_layer.py](../../../models/deepseek_v4_pro/decode_layer.py), [prefill_layer.py](../../../models/deepseek_v4_pro/prefill_layer.py) | +| MTP | [decode_mtp.py](../../../models/deepseek_v4_pro/decode_mtp.py), [prefill_mtp.py](../../../models/deepseek_v4_pro/prefill_mtp.py), [mtp_projection.py](../../../models/deepseek_v4_pro/mtp_projection.py) | +| Decode attention orchestration | [decode_attention_swa.py](../../../models/deepseek_v4_pro/decode_attention_swa.py), [decode_attention_csa.py](../../../models/deepseek_v4_pro/decode_attention_csa.py), [decode_attention_hca.py](../../../models/deepseek_v4_pro/decode_attention_hca.py) | +| Decode sparse attention (fused o-proj) | [decode_sparse_attn.py](../../../models/deepseek_v4_pro/decode_sparse_attn.py), [decode_sparse_attn_swa.py](../../../models/deepseek_v4_pro/decode_sparse_attn_swa.py), [decode_sparse_attn_hca.py](../../../models/deepseek_v4_pro/decode_sparse_attn_hca.py) | +| Decode compressors and indexer | [decode_compressor_ratio4.py](../../../models/deepseek_v4_pro/decode_compressor_ratio4.py), [decode_compressor_ratio128.py](../../../models/deepseek_v4_pro/decode_compressor_ratio128.py), [decode_indexer.py](../../../models/deepseek_v4_pro/decode_indexer.py), [decode_indexer_compressor.py](../../../models/deepseek_v4_pro/decode_indexer_compressor.py) | +| Prefill attention and cache | [prefill_attention_swa.py](../../../models/deepseek_v4_pro/prefill_attention_swa.py), [prefill_attention_csa.py](../../../models/deepseek_v4_pro/prefill_attention_csa.py), [prefill_attention_hca.py](../../../models/deepseek_v4_pro/prefill_attention_hca.py), [prefill_sparse_attn.py](../../../models/deepseek_v4_pro/prefill_sparse_attn.py), [prefill_compressor_ratio4.py](../../../models/deepseek_v4_pro/prefill_compressor_ratio4.py), [prefill_compressor_ratio128.py](../../../models/deepseek_v4_pro/prefill_compressor_ratio128.py), [prefill_indexer.py](../../../models/deepseek_v4_pro/prefill_indexer.py), [prefill_indexer_compressor.py](../../../models/deepseek_v4_pro/prefill_indexer_compressor.py) | +| Shared transforms | [rmsnorm.py](../../../models/deepseek_v4_pro/rmsnorm.py), [qkv_proj_rope.py](../../../models/deepseek_v4_pro/qkv_proj_rope.py), [hc_pre.py](../../../models/deepseek_v4_pro/hc_pre.py), [hc_post.py](../../../models/deepseek_v4_pro/hc_post.py), [hc_head.py](../../../models/deepseek_v4_pro/hc_head.py) | +| MoE and output | [moe.py](../../../models/deepseek_v4_pro/moe.py), [gate.py](../../../models/deepseek_v4_pro/gate.py), [expert_shared.py](../../../models/deepseek_v4_pro/expert_shared.py), [expert_routed.py](../../../models/deepseek_v4_pro/expert_routed.py), [lm_head.py](../../../models/deepseek_v4_pro/lm_head.py) | +| Metadata and host helpers | [config.py](../../../models/deepseek_v4_pro/config.py), [decode_metadata.py](../../../models/deepseek_v4_pro/decode_metadata.py), [rope_tables.py](../../../models/deepseek_v4_pro/rope_tables.py) | +| Real-weight loading | [utils.py](../../../models/deepseek_v4_pro/utils.py) | +| Token loop | [synthetic_token_loop.py](../../../models/deepseek_v4_pro/synthetic_token_loop.py) | `config.py`, `decode_metadata.py`, and `rope_tables.py` have no `__main__` block and are imported rather than run. Which entry points CI schedules is -defined by the [daily model workflow](../../.github/workflows/daily_ci.yml). +defined by the [daily model workflow](../../../.github/workflows/daily_ci.yml). diff --git a/docs/models/index.md b/docs/models/index.md index 2789a1df2..13351012f 100644 --- a/docs/models/index.md +++ b/docs/models/index.md @@ -6,13 +6,16 @@ Files ending in `_draft.py` are work in progress and excluded from CI. | Directory | What it implements | `pypto-serving` | | --- | --- | --- | | [qwen3_14b](qwen3_14b/index.md) | Qwen3-14B BF16 prefill and decode with the serving contract, plus A8W8 and TurboQuant variants and the sampling components | Supported — one-card A2/A3 accuracy job on relevant PRs | -| [deepseek_v4_flash_mtp](deepseek_v4_flash_mtp.md) | DeepSeek V4-Flash at MTP = 1, batch 4 per card: operators, layer and MTP compositions, prefill/decode full forwards | Supported — eight-card accuracy job on relevant PRs | -| [deepseek_v4_pro](deepseek_v4_pro.md) | Ascend A5 DeepSeek V4-Pro with an optional Flash preset, quantized Hybrid MXFP8-MXFP4 | Not supported | -| `deepseek_v4_flash_dspark` | The V4-Flash operators re-sized to batch 64 per card and S = 8 DSpark speculation; under development, operators only | Not supported | +| [deepseek_v4_flash_mtp](deepseek_v4_flash_mtp/index.md) | DeepSeek V4-Flash at MTP = 1, batch 4 per card: operators, layer and MTP compositions, prefill/decode full forwards | Supported — eight-card accuracy job on relevant PRs | +| [deepseek_v4_flash_dspark](deepseek_v4_flash_dspark/index.md) | The same V4-Flash checkpoint at batch 64 per card and S = 8 DSpark speculation, with TP-sharded DSA-CP attention; under development | Not supported | +| [deepseek_v4_pro](deepseek_v4_pro/index.md) | Ascend A5 DeepSeek V4-Pro with an optional Flash preset, quantized Hybrid MXFP8-MXFP4 | Not supported | -The linked directories have a page covering their deployment configuration and -how their files compose. The rest are kernel harnesses validated against the -Golden Harness only. +Each page covers that directory's deployment configuration and how its files +compose. The Qwen3-14B and V4-Flash MTP pages also carry the optimization +history of their tuned path — +[Qwen3-14B optimization](qwen3_14b/optimization.md) and +[V4-Flash decode optimization](deepseek_v4_flash_mtp/decode_optimization.md) — +which record which levers moved the number and what each one cost. Entry points take script-specific platform and device arguments; inspect `--help`, the [platform guide](../get-started/platforms.md), and the diff --git a/docs/models/qwen3_14b/index.md b/docs/models/qwen3_14b/index.md index 1f3b2e10c..bd6f638eb 100644 --- a/docs/models/qwen3_14b/index.md +++ b/docs/models/qwen3_14b/index.md @@ -3,7 +3,7 @@ `models/qwen3_14b/` implements the HuggingFace **Qwen3-14B** checkpoint: a BF16 prefill and decode pair with the serving contract, plus A8W8 and TurboQuant variants and the sampling components. Together with -[V4-Flash MTP](../deepseek_v4_flash_mtp.md) it is one of the two trees wired up +[V4-Flash MTP](../deepseek_v4_flash_mtp/index.md) it is one of the two trees wired up for full `pypto-serving` integration. ## Deployment configuration @@ -124,7 +124,7 @@ constants an external runtime needs. ## How it was optimized -[Qwen3-14B optimization](../../debug-and-tune/qwen3-14b-optimization.md) follows the +[Qwen3-14B optimization](optimization.md) follows the tuning of this tree in order — the shape constraint that forbids the obvious attention fusion, the measurement surface and the pass-rate golden, the general task-count and tiling levers, the attention, sampling and KV-cache rewrites, and diff --git a/docs/debug-and-tune/qwen3-14b-optimization.md b/docs/models/qwen3_14b/optimization.md similarity index 94% rename from docs/debug-and-tune/qwen3-14b-optimization.md rename to docs/models/qwen3_14b/optimization.md index d3afb5e5b..bd9fa81ca 100644 --- a/docs/debug-and-tune/qwen3-14b-optimization.md +++ b/docs/models/qwen3_14b/optimization.md @@ -1,7 +1,7 @@ # Qwen3-14B Optimization This page is a case study rather than a reference. It follows -[`models/qwen3_14b/`](../../models/qwen3_14b/) — a 40-layer dense GQA model that +[`models/qwen3_14b/`](../../../models/qwen3_14b/) — a 40-layer dense GQA model that runs prefill, decode, and sampling on a **single card** — from its first kernels to their current state, and records which levers moved the number, which did not, and what each one cost. @@ -12,12 +12,12 @@ tiles, its task graph, and the boundary between it and the host — which is wha makes the order the work happened in worth recording. The mechanisms live elsewhere: -[Performance Tuning](performance-tuning.md) for how to measure and capture, -[Cube Tile Tuning](cube-tile-tuning.md) for choosing tiles, -[Dependencies and Scheduling](dependency-and-scheduling.md) for the task graph +[Performance Tuning](../../debug-and-tune/performance-tuning.md) for how to measure and capture, +[Cube Tile Tuning](../../debug-and-tune/cube-tile-tuning.md) for choosing tiles, +[Dependencies and Scheduling](../../debug-and-tune/dependency-and-scheduling.md) for the task graph and the scheduler, -[Ring Heap and Scope Stats](ring-heap-and-scope-stats.md) for scope placement, -and [Precision Tuning](precision-tuning.md) for thresholds and rounding. Read +[Ring Heap and Scope Stats](../../debug-and-tune/ring-heap-and-scope-stats.md) for scope placement, +and [Precision Tuning](../../debug-and-tune/precision-tuning.md) for thresholds and rounding. Read those for *how*; read this for *in what order, and what to expect*. Numbers in parentheses are pypto-lib pull requests, kept so a claim can be @@ -42,7 +42,7 @@ move independently, and this page says which one a change moved. None of the three is what a new measurement should use. Today's number is the `mean=` field of `PYPTO_BENCH=1`'s `[RUN] effective_us` line, which is also what -daily CI reports — see [Performance Tuning](performance-tuning.md). Do not +daily CI reports — see [Performance Tuning](../../debug-and-tune/performance-tuning.md). Do not compare a fresh `effective_us` against a figure on this page. ## The shape of the work @@ -91,7 +91,7 @@ The tree spent three changes finding an answer: `UP_DOWN`. That needed the toolchain to accept the `valid_row=0` subview the replay rewrites the trim into — ptoas >= 0.43 and the pto-isa `GetValidRow`/`GetValidCol` relaxation, both named as hard prerequisites in - #420. [constants.py](../../models/qwen3_14b/constants.py) still asserts the + #420. [constants.py](../../../models/qwen3_14b/constants.py) still asserts the geometry it requires: `Q_HEAD_PAD % 4 == 0` and `Q_HEAD_PAD // 2 >= Q_HEAD_BATCH`. @@ -399,7 +399,7 @@ ties the C++ worker guard to the Python `ROPE_CORES` value. > boundary the in-house one could not. The methodology that made #796 possible is written up separately in -[CCE In-Core Profiling](cce-incore-profiling.md). +[CCE In-Core Profiling](../../debug-and-tune/cce-incore-profiling.md). ### 2.2 Sampling: host → device, then approximate → exact @@ -468,7 +468,7 @@ both host goldens mirror the pack and unpack rather than reimplementing them. ## 3. Scheduling Once the arithmetic is mined out, dispatch shape becomes first-order. The -mechanisms are in [Dependencies and Scheduling](dependency-and-scheduling.md); +mechanisms are in [Dependencies and Scheduling](../../debug-and-tune/dependency-and-scheduling.md); what follows is what they bought here. ### 3.1 A disjoint-slice write is not a parallel write @@ -514,7 +514,7 @@ go out as `pl.spmd` per K-split gated on the corresponding cast, and tiles The two ranges write disjoint columns and atomic-add over K, so the split is value-equivalent to the fused form — the only thing that changes is issue order. Both idioms are catalogued in -[Deliberately delaying a task](dependency-and-scheduling.md#deliberately-delaying-a-task). +[Deliberately delaying a task](../../debug-and-tune/dependency-and-scheduling.md#deliberately-delaying-a-task). ### 3.3 Early dispatch, and the one place it hangs @@ -561,20 +561,20 @@ This is not a micro-optimization. Under auto-scope the 40-layer prefill no working value: 131072 deadlocks, 524288 clears it but OOMs the static arena at 6.25 GB. With manual rings the same kernel passes at 262144 / 4 GiB. See -[Ring Heap and Scope Stats](ring-heap-and-scope-stats.md). +[Ring Heap and Scope Stats](../../debug-and-tune/ring-heap-and-scope-stats.md). ## See also -- [Performance Tuning](performance-tuning.md) — measurement, capture, and the +- [Performance Tuning](../../debug-and-tune/performance-tuning.md) — measurement, capture, and the L2 / L1 / L0 tuning rules -- [Cube Tile Tuning](cube-tile-tuning.md) — choosing row, N and K tiles against +- [Cube Tile Tuning](../../debug-and-tune/cube-tile-tuning.md) — choosing row, N and K tiles against the compiler's memory report -- [Dependencies and Scheduling](dependency-and-scheduling.md) — how edges form, +- [Dependencies and Scheduling](../../debug-and-tune/dependency-and-scheduling.md) — how edges form, when the scheduler issues, early dispatch, and dummy-task idioms -- [Ring Heap and Scope Stats](ring-heap-and-scope-stats.md) — manual scope +- [Ring Heap and Scope Stats](../../debug-and-tune/ring-heap-and-scope-stats.md) — manual scope placement and per-ring heap and task-window pressure -- [CCE Extern Kernel](../pypto-coding/cce-extern-kernel.md) — how a hand-written +- [CCE Extern Kernel](../../pypto-coding/cce-extern-kernel.md) — how a hand-written mixed CCE kernel is authored and bound behind `pl.jit.extern` -- [CCE In-Core Profiling](cce-incore-profiling.md) — phase partitioning inside +- [CCE In-Core Profiling](../../debug-and-tune/cce-incore-profiling.md) — phase partitioning inside the external attention kernel -- [Qwen3-14B](../models/qwen3_14b/index.md) — the model this page follows, top down +- [Qwen3-14B](index.md) — the model this page follows, top down diff --git a/docs/pypto-coding/distributed-programming.md b/docs/pypto-coding/distributed-programming.md index 11ce8c267..5d229c126 100644 --- a/docs/pypto-coding/distributed-programming.md +++ b/docs/pypto-coding/distributed-programming.md @@ -380,5 +380,5 @@ Per-rank timing, start skew, and the fastest-rank convention are in — the ring resources an L3 program's intermediates land in. - [`examples/advanced/allreduce.py`](../../examples/advanced/allreduce.py) — the smallest complete L3 program. -- [DeepSeek V4-Flash MTP](../models/deepseek_v4_flash_mtp.md) — the MoE +- [DeepSeek V4-Flash MTP](../models/deepseek_v4_flash_mtp/index.md) — the MoE dispatch / combine and LM-head exchanges this page is drawn from. diff --git a/mkdocs.yml b/mkdocs.yml index 2301134bb..e81891a01 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -92,15 +92,17 @@ nav: - Qwen3-14B: - Model overview: models/qwen3_14b/index.md - PyPTO Page Attention: models/qwen3_14b/paged_attention_pypto.md - - DeepSeek V4-Flash (MTP): models/deepseek_v4_flash_mtp.md - - DeepSeek V4-Pro: models/deepseek_v4_pro.md + - Optimization: models/qwen3_14b/optimization.md + - DeepSeek V4-Flash (MTP): + - Model overview: models/deepseek_v4_flash_mtp/index.md + - Decode optimization: models/deepseek_v4_flash_mtp/decode_optimization.md + - DeepSeek V4-Flash (DSpark): models/deepseek_v4_flash_dspark/index.md + - DeepSeek V4-Pro: models/deepseek_v4_pro/index.md - Debug and Tune: - Overview: debug-and-tune/index.md - Debugging: debug-and-tune/debugging.md - Precision tuning: debug-and-tune/precision-tuning.md - Performance tuning: debug-and-tune/performance-tuning.md - - DeepSeek V4 decode optimization: debug-and-tune/deepseek-v4-decode-optimization.md - - Qwen3-14B optimization: debug-and-tune/qwen3-14b-optimization.md - Dependencies and scheduling: debug-and-tune/dependency-and-scheduling.md - Ring heap and scope stats: debug-and-tune/ring-heap-and-scope-stats.md - Cube tile tuning: debug-and-tune/cube-tile-tuning.md