test_engine_core_proc fails intermittently. Measured on 98b7e2a45 merged onto a7583ac75, CPU-only Release build, one device-free box: 29 pass / 1 fail out of 30 consecutive ctest --test-dir build -R '^test_engine_core_proc$' runs, each run 0.03-0.05 s. The failing run reports ***Failed with no doctest output; the same binary invoked directly (./build/tests/test_engine_core_proc) exits 0 with 14 cases / 110 assertions / 0 failed and prints engine-fatal: EngineCore busy loop threw: injected forward failure.
So the flake is in the process's exit status, not in an assertion. The suite deliberately throws from the engine busy loop (tests/vllm/v1/test_engine_core_proc.cpp:178, throw std::runtime_error("injected forward failure")) to exercise the fatal path, and the exit status of that case depends on whether the loop's thread has been joined and its fatal handled before the process leaves main. A race in the test's own teardown is the shape that fits the evidence; that is a hypothesis, not a measured root cause, and it should be confirmed before it is fixed.
Why this matters beyond one red square: the failure carries no diagnostic, so it presents as an unexplained red on any branch, and this repository has recorded the cost of that shape three times (.agents/verification.md, "tests that starve under ctest -j"). At 1 in 30 it will land on roughly one CI run in ten for a suite that runs it once, and it is indistinguishable at a glance from a real regression in the engine-core loop.
NOT the box being loaded. The 30-run measurement was taken with the box at load average 62, and the earlier serial re-runs that passed were taken at both load average 18 and load average 62, so the outcome does not track load. It is also not related to PR #1119, during whose gate it was found: grep -c 'weight_residency\|WeightResidency\|FromModelDir' tests/vllm/v1/test_engine_core_proc.cpp returns 0, and that change touches no file under src/vllm/v1/.
Two candidate directions, neither taken here: join the busy-loop thread and assert the fatal was observed inside the case rather than at process exit, or make the fatal path's exit status deterministic. Either changes what a landed gate asserts, which is the normal row-spec-fresh-review path rather than an in-flow repair, so it is filed rather than fixed in the flow that found it.
Owning row: ENG-CORE-PROC-FLAKE.
test_engine_core_procfails intermittently. Measured on98b7e2a45merged ontoa7583ac75, CPU-only Release build, one device-free box: 29 pass / 1 fail out of 30 consecutivectest --test-dir build -R '^test_engine_core_proc$'runs, each run 0.03-0.05 s. The failing run reports***Failedwith no doctest output; the same binary invoked directly (./build/tests/test_engine_core_proc) exits 0 with14 cases / 110 assertions / 0 failedand printsengine-fatal: EngineCore busy loop threw: injected forward failure.So the flake is in the process's exit status, not in an assertion. The suite deliberately throws from the engine busy loop (
tests/vllm/v1/test_engine_core_proc.cpp:178,throw std::runtime_error("injected forward failure")) to exercise the fatal path, and the exit status of that case depends on whether the loop's thread has been joined and its fatal handled before the process leavesmain. A race in the test's own teardown is the shape that fits the evidence; that is a hypothesis, not a measured root cause, and it should be confirmed before it is fixed.Why this matters beyond one red square: the failure carries no diagnostic, so it presents as an unexplained red on any branch, and this repository has recorded the cost of that shape three times (
.agents/verification.md, "tests that starve underctest -j"). At 1 in 30 it will land on roughly one CI run in ten for a suite that runs it once, and it is indistinguishable at a glance from a real regression in the engine-core loop.NOT the box being loaded. The 30-run measurement was taken with the box at load average 62, and the earlier serial re-runs that passed were taken at both load average 18 and load average 62, so the outcome does not track load. It is also not related to PR #1119, during whose gate it was found:
grep -c 'weight_residency\|WeightResidency\|FromModelDir' tests/vllm/v1/test_engine_core_proc.cppreturns 0, and that change touches no file undersrc/vllm/v1/.Two candidate directions, neither taken here: join the busy-loop thread and assert the fatal was observed inside the case rather than at process exit, or make the fatal path's exit status deterministic. Either changes what a landed gate asserts, which is the normal row-spec-fresh-review path rather than an in-flow repair, so it is filed rather than fixed in the flow that found it.
Owning row:
ENG-CORE-PROC-FLAKE.