test_engine_core_proc's immediate-shutdown case is load-dependent: a FIXED 1000-frame budget racing an unbounded producer, and no issue names it

[localai-bot]

tests/vllm/v1/test_engine_core_proc.cpp:481, in "EngineCoreProc: immediate
shutdown aborts in-flight requests":

bool abort_seen = false;
for (int i = 0; i < 1000 && !abort_seen; ++i) {
  EngineCoreOutputItem item;
  REQUIRE(client.proc().output_queue.try_get(item));
  ...
}
CHECK(abort_seen);

The request is created with max_tokens=100000 and the busy loop keeps producing
token-delta frames while client.shutdown() propagates. The budget for finding
the abort frame is a FIXED 1000 dequeues. Nothing bounds how many deltas the loop
emits before the shutdown lands, so the budget is a bet on scheduling, and the
bet loses under contention.

Measured on row/LTX25-T2A-ONE-STAGE at 37e680cab, same binary throughout,
CPU-only Release, 20 cores:

Regime	Runs	Failures
ctest -j4, full 492-test suite	3	2 (CHECK( abort_seen ) is NOT correct!)
the binary alone, idle box (load 3.34)	25	0
the binary alone, 20 spinning processes	25	0
ctest -R '^test_engine_core_proc$'	2	0, Passed 0.03 sec

So it is not reproducible by CPU pressure alone; it needs the -j4 harness, where
other suites contend for the same cores in a different pattern. The third -j4
run failed test_cpu_threadpool INSTEAD, which is on the same load-dependent
list, so the identity of the failing test rotates between runs of an unchanged
binary. Both pass alone with exit 0.

No issue names this test. The PR body of
#1032 attributes its flake to
#294, and #294 is a different
defect in a different test — "test_async_llm: reusing an aborted request id races
the core abort". A misattributed flake is worse than an untracked one: the next
reader who checks the citation finds an open issue about something else and stops
looking.

What would close it. Either drain the queue until it is empty rather than for
1000 frames, or bound the request so the producer cannot outrun the shutdown, or
make the abort frame discoverable regardless of how many deltas precede it. The
test asserts a real guarantee (an in-flight request gets a kAbort finish on
immediate shutdown); the budget is the part that is a guess.

Found while repairing the fresh review of
#1039 on that PR, whose branch
touches no file under tests/vllm/v1/ or src/vllm/v1/. Not fixed there because
it is unrelated engine code. Listed under ## Owed in
.agents/specs/ltx25-t2a-one-stage.md.

[localai-bot]

A fresh measurement of the same flake, taken 2026-08-17 during PR #1119's gate, on a CPU-only Release build at 98b7e2a45 merged onto a7583ac75. It adds two things to the record in .agents/environment.md:540-547.

A rate for the whole binary through ctest, not just the case in isolation: 29 pass / 1 fail out of 30 consecutive ctest --test-dir build -R '^test_engine_core_proc$' runs, each 0.03-0.05 s.

And an asymmetry worth knowing before anyone debugs it: the failure does not show up in doctest's own output. The failing ctest run reports ***Failed with no assertion detail, while ./build/tests/test_engine_core_proc invoked directly exits 0 and prints 14 cases / 110 assertions / 0 failed | Status: SUCCESS!, with engine-fatal: EngineCore busy loop threw: injected forward failure on stderr from the deliberate fatal case. So a debugging session that reaches for the binary rather than for ctest will see green and conclude the flake is gone. That is the "0 assertions is a skip wearing a pass" shape one step over: here it is a non-zero process exit with a clean doctest summary.

Not load. The 30-run sweep ran at load average 62; earlier serial re-runs of the same test passed at load average 18 and at 62. The outcome does not track load, so "under heavy parallel ctest" understates it — it also fails when it is the only test running.

Not #1119. grep -c 'weight_residency\|WeightResidency\|FromModelDir' tests/vllm/v1/test_engine_core_proc.cpp returns 0 and that change touches no file under src/vllm/v1/.

One thing this does not settle: whether the non-zero exit comes from the same 1000-frame budget this issue names or from the busy-loop thread's teardown around the injected fatal. The clean doctest summary on a direct run is consistent with the second, but a case that misses its budget would also pass on the next run, so the two are not separated by this data.