Repository navigation
Coverage job: fuzz-worker fork dies after unit-core completes (2 occurrences on PR #2035) #2053
Description
The Coverage job's second Vitest invocation — pnpm test:fuzz-worker, which runs the single-file fuzz-worker project — loses a fork with no test reporting a failure. Twice on PR #2035, on commits that cannot reach it.
Signature
Error: [vitest-pool]: Worker forks emitted error.
Caused by: Error: Worker exited unexpectedly
It lands in the second invocation. pnpm test:coverage:ci is vitest run --coverage --project=!fuzz-worker && pnpm test:fuzz-worker; the first leg reports every file green immediately before the death.
| run | job | leg result before the error |
|---|---|---|
| 32868994199 | Coverage (1) | fuzz-worker: Tests 8 passed (11), Errors 1 error |
| 32962380010 | Coverage (2) | unit-core: Test Files 568 passed (568), then fuzz-worker Test Files (1) |
Both re-ran green on the same commit, so it is not deterministic.
Why this is probably not #1824
#1824 was closed as fixed by #1854: Apple runner tests fabricated child pids (4141/4242/4343/4444) and delivered real signals to sibling vitest forks, and src/__tests__/hermetic-signal-setup.ts now refuses that. That mechanism was in unit-core. These deaths are in the fuzz-worker leg, which runs one file (scripts/fuzz/corpus-replay.test.ts) with fileParallelism: false, maxWorkers: 1 — a single worker, after the unit-core run has already finished. #1824's own thread called this shape "(B) tail deaths" and explicitly did not count it against that fix.
Why it matters
The second occurrence is the load-bearing one: the commit under test (a1cd050d52) differed from a fully green head (adcc35b8db) only in commit messages — git diff between them is empty and the trees are byte-identical. So the failure is independent of the change, and with the contention retry removed (#1781 A4) there is no rerun layer in front of it. It reads as a red required check on an unrelated PR, which is the same class of noise #1874 was filed for.
Suggested first step
corpus-replay.test.ts replays the fuzz corpus through the worker watchdog, waiting a real per-case budget. Worth reading the fork's exit code/signal — the log records neither, and that is the fact that would separate a watchdog-killed child from an OOM or an inherited signal. poolOptions.forks.execArgv or NODE_OPTIONS=--trace-gc on one Coverage run would settle it, the same cheap-confirmation shape #1824 used.
Generated by Claude Code
Activity
thymikee commentedon Aug 26, 2026
Triage: confirmed as a valid CI-infrastructure bug, distinct from #1824's fixed fabricated-signal mechanism. Two byte-identical-tree reruns show the failure is independent of PR code, and the next investigation step is concrete: capture the fuzz-worker fork's exit code/signal before changing retry policy or timeouts. This is sufficiently specified for an agent; moving it to ready-for-agent.
chore(ci): run the probe on pull_request (#2053)
chore(ci): retrigger probe (#2053)
fix(fuzz): run parser cases in a worker process, not the runner's thr…
thymikee commentedon Aug 26, 2026
Investigated. Six occurrences in the leg, not two: runs 32773842455 (08-24 20:28), 32780182194 (21:38), 32780811227 (21:45) and 32854636963 (08-25 13:45), plus the two above — found by scanning the last 60 failed CI runs. Every one is the
--project fuzz-workerleg, 4–8 s in,Tests 3 passed (11)in four of them and8 passedin two.On the suggested first step. The log records no exit code or signal because Vitest throws both away:
emitUnexpectedExit = () => {…}in the pool takes no arguments. Rather than instrument a run and wait for a recurrence, I seeded each candidate death into this project and compared what the run prints with what the CI logs contain:fuzz-workerprojectprocess.exit(1)inside a testTest Files 1 failed, pointing at the lineFATAL ERROR: … JavaScript heap out of memory+ native stackprocess.abort()node::Abortnative stackERR_WORKER_OUT_OF_MEMORY, an ordinary test failureTest Files (1),Tests N passed (11),Errors 1 error,tests 0msThe CI failures carry no
FATAL, no native stack, no failing test, so only the last row fits. That separates the shapes this issue asked about: it is not the OOM #1824 left open for shape (B) — a heap-limit abort prints its banner and the log would have it — and not a silentprocess.exit. It is an uncatchable signal, in a leg whose whole process tree is pnpm, Vitest and one worker, with nothing spawned and 16 GB free.Which points at the file, not the environment. Before #1994 split it out, the vanished file was this one six for six out of ~1100. Of those ~1100 it is the only one that runs adversarial parser cases inside the Vitest worker, on worker threads it creates and terminates eight times per run — and a worker thread is not a fault boundary: a fault in one ends the whole process. The same machinery in a plain process (the nightly, ~266 000 cases a night) has been green throughout.
What I could not get is the signal itself: it needs Linux, and it did not reproduce on macOS in 40 solo runs, 25 under 10-way CPU contention, 2 100 pure-Node create/terminate cycles or 1 400 in-worker cycles (Node 24.13 and 26.1). So this names the class, not the instruction pointer.
#2055 moves case execution into a worker process, where such a fault becomes a
crashfailure naming the input and quoting the exit code/signal plus the worker's stderr tail, instead of a dead runner. Thefuzz-workerproject and the second leg stay as the mitigation, with the criterion for removing both recorded invitest.config.ts: 30 consecutive Coverage runs with noWorker exited unexpectedly.Generated by Claude Code
fix(fuzz): run parser cases in a worker process, not the runner's thr…
fix(fuzz): run parser cases in a worker process, not the runner's thr…