Repository navigation
The reaper shuts down a device the warm pool booted for a queued request, and the pool boots it again #802
Description
What happens
A stopped device the warm pool boots again for a waiting request that is not first in line can be shut down by the cleanup reaper while it waits, then booted again by the pool on its next pass. This repeats until the request reaches the head of the queue.
Only virtual waiters at the head plan a grant (src/leasing/lease-acquisition-coordinator.ts:931-934), so a device booted for the 3rd waiter sits ready and unleased until the 1st and 2nd are served.
Where
src/core/cleanup/idle-shutdown.ts:5-23shuts down a ready device whose last lease ended more thanidle.shutdownAfterMs(10 min) ago, unless it is inview.targeted.view.targetedcomes fromtargetedDevices(src/core/warm-pool/policy.ts:526-539): devices kept by configured targets and learned kinds, up to each kind's count. A device a waiting request would be granted is not in it.- The pool's own rules already spare such devices:
neverLeasedIdle(policy.ts:546,:566) and the budget's choice (policy.ts:199,:281). The pool boots one device per waiter in queue order (policy.ts:333-346, from The warm pool boots or creates one device per queued request, and the boot cap defaults from the CPU #774).
A newly created device is safe (no lastLeaseEndedAt, so the reaper skips it). Since #776, the first peak devices of a learned kind are safe too; a device beyond that count is not.
What a user sees
With several requests queued, a simulator or emulator is booted, shut down a few seconds later with an idle … reason, and booted again, over and over, while CPU and boot slots go to devices nobody can use yet. The waiting requester may be quoted a boot that keeps restarting.
Found
Reading the code for a maintainer question during delivery of #776 on 2026-10-21, at commit 8b3b067 (branch task/776, stack #772 → #788 → #792). Not reproduced; no test proves the loop yet.
Written by an agent.
Activity
V3RON commentedon Oct 21, 2026
Reproduction
Confirmed: two reaper passes shut down the same device twice while its requester remains behind a blocked queue head. The pool boots it back between passes.
Test-only branch:
bug/802-repro, commitba79c0b5.src/leasing/create-leasing.test.ts, “keeps an old released device ready across reaper passes while its requester waits behind the queue head”. Uses real core/leasing/reaper surfaces, fake drivers and clock; the old lease ends before pool startup, leaving no learned protection.pnpm exec vitest run --project unit src/leasing/create-leasing.test.tsRoot cause
Defect.
converger.ts:298–304supplies only configured/spared targets and learned kinds to the reaper's protected set;policy.ts:526–539never considers queue demand. Consequentlyidle-shutdown.ts:9–16sees the newly booted device as idle using its unchanged old lease-end time. Queue FIFO prevents its requester taking it yet, and each shutdown triggers another pool boot.Simplest fix
Include current waiting demand in the pool's cleanup-protection calculation. Union the existing protected set with idle ready devices serving a non-in-flight waiter, reusing the policy's existing grantability, requirement and mode matching. Read demand afresh each time so cancellation removes protection. Update the affected manuals/comments. No fix is on this branch.
Alternatives rejected
Risk
Written by an agent.
test: cancellation restores idle cleanup for queued devices (#802)
test: queue cleanup protection preserves grantability checks (#802)
fix(warm-pool): consolidate audited queue protection (#802)
V3RON commentedon Oct 21, 2026
Delivery run
Merged: #804 (#802)
Parked: #790 — gate refused twice, red CI only from flaky-test #807 (also fails on main); PR #805 is ready and reviewed
Waiting: #790 — on #807
Written by an agent.