test_qwen3_5_decode_graph_seam SEGFAULTs in build-test-cpu, and build-test-cpu is red on every open PR

[localai-bot]

CI job build-test-cpu fails on every open pull request. Two tests are responsible, and they have different owners.

The following tests FAILED:
	101 - test_qwen3_5_decode_graph_seam (SEGFAULT)
	174 - test_nemotron_h_paged_forward (Failed)
Errors while running CTest
##[error]Process completed with exit code 8.

Observed on run 32289803793 job 96190957731, build-test-cpu, step Test, Total Test time (real) = 526.18 sec, and reproduced on the same job across #1348, #1081 and #1289.

The second one already has an owner

test_nemotron_h_paged_forward is #1371 — CPU has no attention backend once FLASH_ATTN declines a head size. PR #1392 fixes it and is in review. This issue is not about that test.

This issue is the SEGFAULT

test_qwen3_5_decode_graph_seam segfaults. A segfault is not an assertion failure: the process dies, so the harness reports no case counts, no Status: line and no assertion totals. Nothing downstream of the crash ran, and a reader who greps for Status: or assertions: finds neither and can mistake the silence for a skip.

tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp was last touched by:

• 5f68e60df — feat(ENG-CUDAGRAPH-BREAK): widen graph eligibility to the step's actual query length, and key the spec ring so it cannot collide (#1374, #1020)
• 3b76ccbfe — feat(ENG-CUDAGRAPH-BREAK): make the persistent step input a seam capability, and correct the record the token-id path never existed (#1307, #1163, #1319)

Both are on main, so this is very likely a main defect inherited by every branch rather than any PR's own regression. That is a hypothesis, not a finding: I could not confirm it from a main baseline, because every recent main CI run is completed/cancelled, superseded by the next push before it finished. There is currently no green or red verdict for main itself to subtract.

Establishing that baseline is the first task here, and it is the part that must not be skipped. Run the test at origin/main directly and record the result. If main is red, this is a main regression and the owning row is ENG-CUDAGRAPH-BREAK; if main is green, the cause is elsewhere and this issue needs re-scoping rather than closing.

Why it matters beyond itself

build-test-cpu red on every PR means no PR can be landed on a clean gate, and it trains readers to treat a red build-test-cpu as background noise — which is exactly how a real regression gets merged. Two of this repository's standing traps apply: a cancelled run reads as a failure and can leave a branch with no successor at all, and a crashed binary produces no counts, so its silence must never be read as a pass.

Notes for whoever takes this

• Get a stack. A segfault under ctest gives nothing on its own; run the binary directly and under a debugger or ASan.
• The neighbouring seam tests (test_qwen3_moe_decode_graph_seam, test_voxtral_decode_graph_seam, test_deepseek_v2_decode_graph_seam, test_breakable_graph) are the natural controls — do they also crash, or is this one specific?
• Do not "fix" it by deleting or disabling the case. A gate made green by removing the assertion is the failure this repository's policy names explicitly.

[localai-bot]

The SEGFAULT is INTERMITTENT, and that changes what it probably is.

Two build-test-cpu runs, both on the same job name, hours apart:

run	job	failed tests
32289803793	96190957731	101 - test_qwen3_5_decode_graph_seam (SEGFAULT) and 174 - test_nemotron_h_paged_forward (Failed)
#1251's run	96255349510	only 174 - test_nemotron_h_paged_forward (Failed)

Same test binary, same CI job, same main lineage. The segfault fired once and not the other time.

A deterministic null dereference does not do that. Intermittency points at a race, an uninitialized read, a use-after-free, or an ordering dependency — the classes that also produce green runs, which is the dangerous part: a flaky segfault that passes on a rerun looks like it was fixed by the rerun.

This raises the priority of getting a stack rather than a reproduction count. Suggested order:

1. Run the binary directly in a loop and record the failure RATE. A rate is the measurement; "I could not reproduce it" is not evidence of absence.
2. Get the stack under ASan/TSan rather than plain ctest. sanitize-cpu (address,undefined) and sanitize-cpu (thread) exist as jobs, but they are PR-only with no main baseline (#584) and are currently red on every PR, so they are not currently a usable signal for this — they need baseline-subtracting first.
3. tests/CMakeLists.txt runs ctest in parallel. This repository has a recorded history of tests that are flaky specifically under parallel ctest and clean in isolation, so establish which regime this crashes in before concluding anything about the code.

Do not close this on a green rerun. Under ctest a crashed binary produces no case counts, no Status: line and no assertion totals, so its silence is indistinguishable from a skip unless you are looking for it — and a flaky crash gives you a green whenever you most want one.

The main baseline this issue asks for is still owed and still the first task: every recent main CI run is completed/cancelled, superseded before finishing, so there is no verdict for main itself to subtract.

Unrelated to the segfault, for anyone reading this issue as "why is build-test-cpu red": the other failing test, test_nemotron_h_paged_forward, is #1371 and is fixed by PR #1392, which now passes it 12/12 at 3256 assertions and is in fresh re-review.

[localai-bot]

Closing as a duplicate of #1390, which was filed 5 hours earlier and diagnosed this better than I did. My error.

#1390 (2026-08-19T18:16, now CLOSED) names the mechanism precisely: "the W6 spec-shape case crashes in the CPU paged-attention kernel, but only when the earlier cases ran first" — measured at 5f68e60df on origin/main exactly, with the crash localised to W6: two spec shapes of EQUAL S and different q get two graphs, exit 139.

That explains the intermittency I reported above as a new finding. It is order-dependence, not flakiness in the usual sense, and #1390 had already established it. My two disagreeing CI runs are exactly what an order-dependent crash produces.

Current state: build-test-cpu reads COMPLETED/SUCCESS on PR #1392, with no segfault reproduced — consistent with #1390 being resolved.

How I filed a duplicate, since the mechanism is reusable

I did search first. These returned nothing:

gh issue list --state all --search "qwen3_5_decode_graph_seam"
gh issue list --state all --search "decode_graph_seam SEGFAULT"
gh pr list --state open --search "decode_graph_seam"

#1390's title contains the literal string test_qwen3_5_decode_graph_seam. So the search returned a false negative on an exact token that was present, and I read the empty result as absence — the specific error I have a written rule against.

The lesson for anyone else working this repository: gh issue list --search is not reliable for identifier-shaped terms. Underscore-heavy symbols tokenize badly. Corroborate with a listing you filter yourself before concluding nothing exists:

gh issue list --state all --limit 200 --json number,state,title \
  | jq -r ".[] | select(.title | test(\"decode_graph_seam\"))"

A null search result is evidence your query is wrong at least as often as it is evidence the thing is absent.

The only content here worth keeping is the CI-triage summary, which is not specific to this crash and is already recorded where it belongs: windows-msvc-* and sanitize-cpu-* are PR-only jobs with no main baseline (#584) and red on every PR; test_nemotron_h_paged_forward is #1371, fixed by #1392 which now passes it 12/12 at 3256 assertions.