[Bug] Daily full-scene run #31722222789 exposes manual PA regressions and sim lane poisoning

[doraemonmj]

Summary

Daily Full Scene Tests run #31722222789 at 9a6a785934c3747d29547c67ee8c5849cd5b2c72 failed in 5 of 7 jobs.

The run contains 26 pytest failure records, but only 20 are direct failures. The remaining 6 are cascades after an a2a3sim run lane was poisoned by a scheduler timeout.

This is not caused by Daily using a different execution environment from Per-PR. Daily and Per-PR reuse the same _st-* workflows and platforms. The important difference is selection:

• Per-PR runs with manual_mode=exclude.
• Daily runs with manual_mode=include, i.e. the Per-PR scene set plus manual cases.
• The pytest node is usually reported as Class::test_run, while that method loops through multiple CASES. Therefore, the same class passing in Per-PR only proves that its non-manual cases passed.

For comparison, the corresponding scene jobs passed in PR run #31697418830. The Daily failures are all reached through manual cases; they do not contradict the Per-PR result.

Failure classification

Category	Direct failure records	Affected coverage	Observed cause
Host Build Graph task-ring exhaustion	2	a2a3 and a5 onboard	Task Allocator Deadlock - Task Ring Full, active=16383/16384; these HBG cases build roughly 64K tasks before scheduling
Paged Attention golden mismatch	12	a2a3 and a5 onboard	11 large BF16 Case3 workloads fail with max_diff around 1.05–1.08; new A5 SPMD zero-query case fails with max_diff=inf
SPMD PA compile failures	4	a2a3/a5 sim, macOS and Ubuntu	Clang rejects unsigned long to uint32_t narrowing; Linux also calls nonexistent setEntryOffset instead of setentryOffset
HighPerf PA scheduler stall	2	a2a3sim on macOS and Ubuntu	Manual b4_h32_kv8_s512_bs128_fp16 stalls with SCHEDULER_TIMEOUT, completed=0/1, stuck_task_id=0
Poisoned-lane cascades	6	tests following the HighPerf timeout	chip run lane is poisoned: finalize_native_run failed with code -100; these are not independent defects

Precision failures

The previously green Per-PR runs did not exercise the same shapes:

• 11 failures are the manual Case3 shape: batch=64, num_heads=64, kv_head_num=1, head_dim=256, block_size=64, context_len=8192, max_model_len=32768, BF16. The smaller non-manual cases in the same classes passed in Per-PR. This points to a scale/shape-dependent correctness problem, rather than an environment difference.
• TestSpmdPagedAttentionA5::TwoWayOneBlockZeroQuery is a new manual-only case from test: add manual cross-runtime SPMD paged attention coverage #1817. It is small (batch=2, context_len=128) and fails with max_diff=inf, so this one is not explained by workload scale and should be treated as a new SPMD PA correctness regression. It overlaps with, but is not fully covered by, SPMD paged attention fails with multi-block and multi-core A5 workloads #1816.

The large Case3 failures affect:

• a2a3: TestPagedAttention, TestPagedAttentionManualScope, TestBatchPagedAttention, TestPagedAttentionUnrollManualScope, TestPagedAttentionUnroll, and TestPagedAttentionUnroll4dims
• a5: TestPagedAttention, TestBatchPagedAttention, TestPagedAttentionUnrollManualScope, TestPagedAttentionUnroll, and TestPagedAttentionUnroll4dims

Watch history / CI optimization impact

Of the 20 direct failure records:

• 15 are historical manual cases that have been outside Per-PR coverage since their introduction or port.
• 5 are new manual-only SPMD PA coverage added by test: add manual cross-runtime SPMD paged attention coverage #1817 (9ec90ecc5979327182e8765399ff4520e37d30f3).
• 0 were recently removed from Per-PR by the CI-time optimization in CI: move redundant scene tests to daily sweep #1790 / d31395bf.

In other words, #1790 exposed the existing manual backlog through the new Daily sweep, but none of this run's failed cases were newly down-selected by that change.

Detailed direct failures

1. Onboard Host Build Graph cases

• TestPagedAttentionHostBuildGraph::Case1 on a2a3
• TestPagedAttentionHostBuildGraphA5::Case1 on a5

Both fail during prepare_native_run with:

FATAL: Task Allocator Deadlock - Task Ring Full!
active=16383/16384 (100.0%)

The heap still has capacity, so the immediate resource is the task ring/window rather than host memory. The cases document roughly 64K tasks, while the current ring window is 16K.

2. SPMD PA compile portability

• a2a3 HBG on macOS: narrowing conversions in spmd_paged_attention_orch.cpp:118-120
• a2a3 HBG on Ubuntu: setEntryOffset at paged_attention_parallel.cpp:695,701; available API is setentryOffset
• a5 HBG and TMR on macOS: narrowing conversions in spmd_paged_attention_orch.cpp:136-138

3. a2a3sim HighPerf stall and cascades

The non-manual b1_h32_kv8_s128_bs128_fp16 case passes. The next, manual b4_h32_kv8_s512_bs128_fp16 case times out on both runner OSes:

SCHEDULER_TIMEOUT
S1:running-stalled
completed=0/1, running=1, stuck_task_id=0, stuck_core=0

After this, six later tests fail only because the shared lane remains poisoned:

• macOS: TestSpmdSyncStart, TestSpmdSyncStartEarlyDispatch, TestSpmdSyncStartStress, TestSyncStartEarlyLocalOwner
• Ubuntu: TestSpmdSyncStartEdge, TestChipSwimlaneMixed

All six passed in the referenced Per-PR run. The first investigation step should be to raise PTO2_SCHEDULER_TIMEOUT_MS to distinguish a genuinely stuck kernel from a workload that simply exceeds the current simulator budget.

Expected behavior

• Daily remains a superset of Per-PR and all selected manual cases pass, or unsupported/resource-bound cases are skipped with an explicit reason.
• One failed simulator case does not turn subsequent independent tests into false failures.
• CI output identifies the failing entry inside CASES, rather than only reporting the enclosing Class::test_run node.

Suggested acceptance criteria

• Fix or explicitly constrain the large BF16 Case3 workloads on a2a3 and a5.
• Resolve the A5 SPMD zero-query inf mismatch together with, or as a scoped follow-up to, SPMD paged attention fails with multi-block and multi-core A5 workloads #1816.
• Fix the SPMD orchestration narrowing conversions and the setEntryOffset spelling/API mismatch on both compiler paths.
• Make HBG ring sizing support the documented task count, or reject/reshape these cases before allocator deadlock.
• Determine whether the HighPerf b4/s512 failure is a real deadlock or a timeout-budget issue, then prevent lane poisoning from being counted as six additional root failures.
• Improve parameterized-case reporting so the failing case name and parameters are present in the pytest failure node or assertion context.

Related: #1816, #1772, #1790, #1817.

[doraemonmj]

HBG Paged Attention Case1 follow-up: defer until dynamic ready queue support

The two HBG Case1 failures have a second capacity constraint behind the task-ring failure reported in the issue body. This comment records the complete chain and why we are not applying the static queue-size workaround.

Background and coverage history

The large Case1 parameters have existed since the PA scene was migrated into tests/st in April 2026. They were marked manual when the host-build-graph runtime was introduced, so Per-PR excludes them while Daily includes them. That is why this appeared in the Daily sweep rather than the corresponding Per-PR run.

Affected cases:

• tests/st/a2a3/host_build_graph/paged_attention::Case1: batch=256, context_len=8100, block_size=128, num_heads=16
• tests/st/a5/host_build_graph/paged_attention::Case1: batch=256, context_len=8192, block_size=128, num_heads=16

Task-count analysis

Each sequence needs 64 logical KV blocks:

• a2a3: ceil(8100 / 128) = 64; 63 full blocks cover 8064 tokens and the final block contains 36 valid tokens.
• a5: 8192 / 128 = 64 exactly.

For this shape, q_tile=min(num_heads, 128)=16, so q_loop=1; the 16 heads are processed in one tile and do not add another task-count multiplier.

Each KV block submits four stages:

1. AIC QK_MATMUL
2. AIV SOFTMAX_PREPARE
3. AIC PV_MATMUL
4. AIV ONLINE_UPDATE

There is also one alloc_tensors node per batch for the running online-softmax state. Therefore:

per batch = 1 alloc + 64 blocks * 4 tasks = 257 tasks
total     = 256 * 257 = 65,792 tasks

This explains the first observed failure:

Task Allocator Deadlock - Task Ring Full
active=16383/16384

The task ring/window must represent the complete graph while host orchestration builds it. The existing case-level runtime configuration can isolate this part to these two cases; a trial used ring_task_window=131072 and ring_heap=512 MiB.

Secondary ready-queue constraint

Increasing the case-local ring/window exposes a separate initial-ready-width issue. Every QK task reads only the external Query and its own K-cache block, so QK tasks from different blocks have no producer dependency. Graph activation therefore has:

AIC roots:   256 batches * 64 blocks = 16,384 QK tasks
Dummy roots: 256 alloc nodes, routed through the separate dummy queue
Other tasks: not initially ready because they depend on QK/softmax/PV or online state

The current HBG ready queue is a fixed-capacity, per-resource-shape MPMC ring with capacity 8,192. Root routing is eager and its push path has no backpressure or retry: a full queue latches READY_QUEUE_OVERFLOW. Although scheduler threads consume concurrently, graph activation can publish the 16,384 AIC roots much faster than the limited AIC cores can dispatch them. The AIC queue therefore needs to absorb the boot-time burst, and 8,192 cannot cover a 16,384-wide root set.

This is not a requirement of the PA algorithm itself. It is a consequence of the current eager root-routing design. An incremental/dynamic ready queue can pause root publication, retain the route cursor, and resume after consumers create space instead of requiring static capacity for the maximum graph width.

Options evaluated

1. Case-local task window and heap: supported and scoped appropriately. It solves only the first task-ring/heap constraint.
2. Case-local ready queue: not currently supported because ready queues are created in the shared scheduler arena from a compile-time capacity.
3. Raise the global queue from 8,192 to 32,768: technically simple, but it changes every HBG run on both onboard and sim.

The global-memory impact of option 3 is approximately:

9 queues = 3 regular + 3 sync + dummy + graph-ready + graph-prepare
sizeof(PTO2ReadyQueueSlot) = 24 bytes
current: 9 * 8,192  * 24 = 1,769,472 bytes (~1.69 MiB)
32K:     9 * 32,768 * 24 = 7,077,888 bytes (~6.75 MiB)
delta:                         5,308,416 bytes (~5.06 MiB) per runtime/device

It also increases queue-slot initialization work for every HBG runtime. Push/pop remains O(1), so the expected steady-state performance effect is small, but onboard always pays the additional arena memory and initialization cost; sim also pays the larger allocation/initialization footprint, where initialization overhead can be more visible. These are quick structural estimates rather than precision benchmarks.

Decision / known CI effect

We reverted the trial changes and will not push a global ready-queue increase. This HBG Case1 item is deferred until the planned dynamic ready queue / incremental root-routing support is available.

Until then, both manual Case1 entries remain known Daily failures. With the current defaults Daily normally stops first at the 16K task-ring limit; if only the per-case task window/heap is enlarged, the 8K AIC ready queue becomes the next blocker. Therefore this part of #1832 should remain open or explicitly tracked as deferred rather than considered fixed by the other PRs.