[Bug] DeepSeek V4 Flash EP8 GBS32 first fused MTP decode fails with TENSOR_WAIT_TIMEOUT

[vegetabledoww]

Diagnosis

The failure is a PTO2 tensor wait inside the DeepSeek V4 Flash fused main/MTP decode. The exact producer/owner is not identified by the retained logs. The MoE dispatch window publication/reuse protocol in pypto-lib is one candidate investigation direction, possibly through an interaction with the PTO2 tensor lifecycle in simpler.

Both reproductions complete compilation, weight loading, all eight chip-process initializations, host-worker/KV initialization, and prefill. The first host-side fused main/MTP decode (actual_batch=32, trace step_id=2) then fails on all 8 ranks with orch_error_code=8 TENSOR_WAIT_TIMEOUT. The device-side detector messages are:

Run 1: FATAL(code=8): Timeout (750000000 cycles): producer (ring=1, local=841) not completed
Run 2: FATAL(code=8): Timeout (750000000 cycles): producer (ring=1, local=596) not completed

Static inspection of the generated fused-decode graph shows that AIV function ID 186 is dispatch_meta_1; this kernel publishes/modifies recv_count_out. This overlaps the MoE window-publication and reuse area rewritten by draft PR #948 (related to #929). The reproducing checkout is based on 77c78c9, which is the parent of that candidate fix, with the local patch disclosed below applied.

This is a candidate path, not a proven owner mapping: the retained timeout logs report the producer ring/local ID but do not include the tensor address or a precise producer-to-generated-function mapping. The different local IDs in the two runs further prevent identifying function ID 186 as the timed-out producer. PR #948 has also been described/validated for persistent-worker EP2 A/B/A numerical contamination, while this report is an EP8/GBS32 timeout during the first host decode. No EP8 A/B run with #948 has been performed yet, so this report should not be treated as a duplicate or as confirmation that #948 fixes it.

The first host decode still contains multiple MoE dispatch windows (dispatch_meta_0 through at least dispatch_meta_4) inside the fused graph, so window reuse within that invocation remains possible.

Description

Configuration

Model: DeepSeek V4 Flash W8A8
Parallelism: overlapped attention DP8 / MoE EP8 / TP1 on 8 NPUs
Global batch size: 32
Prompt length: 64 tokens
Requests: 32
Maximum output: 256 tokens/request
MTP: enabled (MTP1)
Chunked-prefill mode: enabled (Seq64 is below the 128-token threshold)
Sampling: temperature=0, top_p=1, ignore_eos=true

Reproduction

Using this fixed prompt (shell $'...' syntax), which tokenizes to exactly 64 tokens:

export DSV4_HC_PRE_IMPL=separate
export PTO2_RING_DEP_POOL=131072
export PTO2_RING_TASK_WINDOW=131072
export PTO2_RING_HEAP=2147483648
export SIMPLER_SCHEDULER_TIMEOUT_MS=60000
export SIMPLER_OP_EXECUTE_TIMEOUT_US=120000000
export SIMPLER_STREAM_SYNC_TIMEOUT_MS=150000
export ALIGN_PROMPT=$'<\uff5cbegin\u2581of\u2581sentence\uff5c><\uff5cUser\uff5c>Please read a part of the book below, and then give me the summary.\n[start of the book]\n\n\n\n[end of the book]\n\nNow you have read it. Please summarize it for me. First, tell me the title and the author, and then tell the story in 256 words.\n\n <\uff5cAssistant\uff5c></think>'

python examples/model/deepseek_v4/npu_generate.py \
  --model-dir <deepseek-v4-flash-w8a8-model> \
  --prompt "$ALIGN_PROMPT" \
  --platform a2a3 \
  --devices <eight-devices> \
  --max-seq-len 512 \
  --max-new-tokens 256 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 2048 \
  --long-prefill-token-threshold 128 \
  --num-prompts 32 \
  --temperature 0 \
  --top-p 1 \
  --ignore-eos \
  --enable-mtp \
  --enable-chunked-prefill

Expected tokenizer IDs (also checked before each run):

[0, 128803, 12473, 1733, 260, 892, 294, 270, 2339, 4167, 14, 305,
 1539, 3475, 678, 270, 13672, 603, 68909, 294, 270, 2339, 1839, 271,
 112672, 294, 270, 2339, 1839, 8197, 440, 611, 1733, 436, 16, 11992,
 45706, 436, 362, 678, 16, 5978, 14, 4575, 678, 270, 6407, 305, 270,
 3725, 14, 305, 1539, 4575, 270, 4093, 295, 223, 9636, 3345, 339, 223,
 128804, 128822]

Expected result

All 32 requests continue decoding after prefill and produce 256 output tokens, allowing TPOT to be measured under the aligned GBS32/Seq64/MTP1 condition.

Actual result

The problem reproduced twice with identical pinned commits, source-diff hashes, prompt hash, runtime artifact paths/metadata, and timeout configuration:

Run	Task	Local log
1	task_20260812_203822_78514330338	/tmp/pypto-dsv4-cann-aligned-jkfo3m/perf.log
2	task_20260812_203939_87371118636	/tmp/pypto-dsv4-cann-aligned-Hdc6Xb/perf.log

For each run:

1. All eight chip processes initialize, followed by the host worker; KV initialization succeeds.
2. Prefill succeeds. The subsequent error summaries record out=1 for each request.
3. The trace records the first fused main/MTP decode at step_id=2, actual_batch=32, with ranks (0,1,2,3,4,5,6,7) repeated four times. The asynchronously dispatched step_id=3 is later drained as stale.
4. All eight device processes surface ACL 507018; the runtime then emits eight identical classifications:

PTO2 runtime failed: orch_error_code=8 sched_error_code=0 runtime_status=-8
error detail: orch_error_code=8 TENSOR_WAIT_TIMEOUT - waiting for tensor data timed out:
the producing task never completed, or a consumer never released its fanout reference

5. All requests finish as prompt=64 out=1 reason=error.

ACL 507018 is the first host-visible symptom; the runtime subsequently classifies the failure as TENSOR_WAIT_TIMEOUT. Later bounded-drain, stream teardown, domain release, and cleanup errors appear to be recovery fallout. No allocation-failure or other concrete OOM signature was observed; the word OOM only occurs in generic diagnostic guidance.

Environment

Component	Version
pypto-lib	based on 77c78c99648bfcab6ed041c1f6a5a8f366fad205, with the local patch below
pypto-serving	based on ff1fce1c9c291df79715576255c1e5f4e28b3d8f, with the local patch below
pypto	71020585278b68f56c72c40d5570f07dbb20bc8b
simpler / PTO2 runtime	3165cc89b6ea6b58a0bc01cbec2d5f72f2029c35
PTO-ISA	83d01313d9bfc247c4b7c8bcf969d1019f0d106f
PTOAS	0.57
CANN	9.0.0

Both checkouts were intentionally dirty. The following values are SHA-256 hashes of the respective git diff --binary --full-index output and were identical between the two runs:

pypto-lib:     13bab524b53d4102e5f0f28bda594e087ad9480c7ccd60c977b4e77de1ca99f8
pypto-serving: 679c828866e4b1fb607abf409c06ec0d85aae2e1557b1f84eef598211f78e906

The only modified lib files (45 insertions, 19 deletions) were:

• models/deepseek_v4_flash_mtp/hc_pre.py: uses a deterministic single-writer HC projection and explicit dependencies for the separate implementation.
• models/deepseek_v4_flash_mtp/moe.py: removes allow_early_resolve=True from dispatch_gather while retaining its dependencies on _wait_tid and _meta_tid (the local mitigation for the EP8 producer-starvation path tracked separately as Add: DeepSeek V4 Flash CP prefill layer and multi-layer forward #899).

Thus, the reported timeout occurs even with the dispatch_gather early-resolution mitigation present. The hc_pre.py patch changes task topology/dependencies and is a possible confounder; no clean-base A/B has been run.

The serving patch adds profiling/observability fields, committed/MTP token counters, and associated async step-protocol tests. Apart from the separately disclosed pypto-lib patch, it does not change generated model kernels or scheduling decisions, but it is disclosed because it changes host-side result accounting.

Host Platform

Linux (aarch64)

Additional Context

• Draft PR Fix: fence DeepSeek V4 MoE window reuse #948 is a relevant candidate fix because it changes MoE dispatch-meta publication, epochs, consumer acknowledgement, and window reuse fencing on top of the checkout's base commit. It has not yet been validated against this EP8 GBS32 timeout.
• Issue [Bug] DeepSeek V4 MoE corrupts changing inputs after persistent worker reuse #929 / PR Fix: fence DeepSeek V4 MoE window reuse #948 concern persistent-worker multi-invocation numerical contamination. This report differs in symptom and topology: the first host fused decode reproducibly fails with a tensor wait timeout across all EP8 ranks.
• The failure is reproducible at the stage and error-class level, but the producer local ID, completed-task count, and first device to surface the error differ between runs.
• Full host/device logs, traces, and generated deepseek_v4_decode_mtp_fused artifacts are retained for both runs and can be uploaded on request.

[MaxwellF1]

#893 adds the missing producer-to-waiter dependencies in moe.py, where dispatch_meta and recv_count_out live. It turns the EP8 stall with the same TENSOR_WAIT_TIMEOUT into a pass, with the same ring settings you used. Worth re-running your repro on that branch to see if it is the same root cause.