The warm pool learns the kinds it leases and keeps them ready, across restarts #776

Part of #771.

Scope

A worker with no warmPool.targets keeps what it has leased: for every kind it granted a lease of, as many devices as were held at once in the last 24 hours, at least one, counting the ones leased right now, newest kinds first within today's budget, and booted again after a daemon restart. A newer learned kind may take an older learned kind's slot. A kind is forgotten when its model or runtime is gone, after three failed boots in a row, when its platform holds 32 kinds and it is the oldest, and on simlock nuke. The "released under idle.shutdownAfterMs" reboot rule goes: a learned kind covers it.

Kept whole at about 25 files: the record, the policy's use of it and the converger's learning are one behaviour a test can only prove together (a lease ends, a device comes back), and the reboot rule it replaces must go in the same change or the two rules would fight over the same device.

In short

Today a released device is booted back only if its lease ended under 10 minutes ago and a slot is free; nothing is booted ahead of the first request of a kind, and a restart starts cold. After this PR the pool learns a kind (platform, model, OS version, mode, image tag, what sameSpec compares) from every lease it grants and keeps peak devices of it, where peak is the most leases of that kind held at the same moment in the last 24 hours (counted from now, a lease held across the window's start counts, never below 1). A kind has its idle ready devices plus its leased and reclaiming ones, so while a burst is held nothing more is made, and the released devices come back ready. The record lives in the pool's own state.db tables, so the first pass after a restart boots them again.

flowchart LR
  G[lease.granted] --> L["record the kind and the lease (new) *"]
  E[lease ends] --> L
  L --> P["pass: configured targets, then learned kinds newest first (changed) *"]
  P --> F{fits the budget?}
  F -- yes --> B[boot a shut-down device of the kind, else create one]
  F -- no, slot held by an older learned kind's idle device --> D["shut it down: displaced (new) *"]
  D --> B
  R[daemon start] --> P
  N[simlock nuke] --> X["record cleared (new) *"]

• learned kind: a kind a lease was granted of on this worker, kept ready like a target, origin learned.
• displaced: an idle ready device of an older learned kind shut down (initiator warm-pool; the reason displaced is in the daemon log and the catalog's prose, since device.shutdown carries no reason field) so a newer learned kind short for a running slot can boot or create into it. Never a configured target's device, never one that serves a waiting request.
• forgotten: the kind's rows are deleted; its next lease learns it again.

User's session, a worker with no config:

$ simlock lease --platform ios --device "iPhone 16" --os 18.4 --agent-id a1
# ... source: provisioned (60 s); work; release
# today: erased and shut down; booted back only if a slot is free within 10 min; after 10 min stays down
# after: erased, shut down, booted back at once and kept ready (not shut down idle, whatever the time);
#        while a1 held it, nothing more was made: the kind had one, its peak
$ simlock config set warmPool.enabled false && simlock daemon stop && simlock status   # the pool off shuts it down
$ simlock config set warmPool.enabled true && simlock daemon stop && simlock status    # on again: a restart
# today: nothing ready
# after: the iPhone 16 / 18.4 is booting, then ready, before any lease: the record kept the kind while the pool was off
$ simlock lease --platform ios --device "iPhone 16" --os 18.4 --agent-id a2
# after: source: warm
$ simlock nuke --yes
# after: nothing is booted back; the record is empty until the next lease

What stays ready, budget of two running iOS slots, no configured targets:

If this goes wrong, a user would notice devices booting that nobody asked for, a device shut down and booted again in a loop between two kinds, or a worker that holds more RAM than its running limit allows.

Technical spec

Facts

Commit 97bc8eaf.

Modules touched

Searches: recently-released, lastLeaseEndedAt, targetedDevices, ResolvedTarget, resolveTargets, NukeServiceOptions, WarmPoolOptions, createCore(, warmPoolReader over src/, e2e/, docs/.

• src/core/warm-pool/learned.ts (new): the owner warm-pool (its SqliteSchemaOwner), LearnedKinds over a StateDatabase: learn(spec, leaseId, at), ended(leaseId, at), closeOpen(heldLeaseIds, at), forget(spec), clear(), kinds(now) → { spec, count, lastLeasedAt }[], newest first; pruning and the 32-per-platform cap inside learn. Every method catches nothing: the converger does.
• src/core/warm-pool/policy.ts: view.learned (kinds with count and last lease); TargetReport.origin (configured or learned) and configuredCount (the configured count of a merged entry); planTargets takes configured then learned; a kind in both is one entry, origin configured, count the larger, in the configured position, needing the larger of (configured count − ready) and (learned count − has), where has = ready idle + leased + reclaiming devices of the kind (F8); the displacement rule; boots() loses the F6 rule; targetedDevices and neverLeasedIdle's kept set count learned kinds (F8b).
• src/core/warm-pool/retry.ts: failures(spec) answers the count in a row (F11d).
• src/core/warm-pool/converger.ts: subscribes to lease.granted, lease.released, lease.expired for learning (F1, F2); closes open rows at start() against registry.snapshot.leases (F3); a catalog check that runs beside the pass, one listCatalog(platform) per platform with a learned kind, bounded to 30 s by the clock (F11a), whose result the next pass applies: a kind whose model is not in models or whose OS version is not in modelRuntimes[model], or a tagged kind with no images entry of that runtime and tag, or whose platform has no driver, is forgotten; a read that throws, times out or has not answered yet keeps every kind of that platform and plans nothing for them that pass (ADR 0038 §4); forgets a kind on its third failure in a row (F9, F11d) with one warn; forget() for nuke; figures() and #note read origin: configured reports only in this task (F8a), so status is unchanged; #announceMissed judges and reports a merged entry by configuredCount (F8c); every LearnedKinds call is wrapped: a throw is logged once per pass or event and the pool goes on without learned kinds until a read succeeds.
• src/core/warm-pool/index.ts and src/core/index.ts: export warmPoolOwner.
• src/core/create-core.ts: options state: StateDatabase and catalog: Pick<DriverCatalog, "listCatalog"> for the pool (F16); NukeService gets warmPool: { forget }. Every createCore( caller of F11f passes them.
• src/core/nuke-service.ts: calls warmPool.forget() after beginMaintenance and before resetting devices (F12).
• src/daemon/main.ts: warmPoolOwner in the worker owner list (F15); state into createCore.
• src/leasing/testing.ts and the test files of F11f: build the core over openMemoryStateDatabase([... , warmPoolOwner]).
• Tests: src/core/warm-pool/learned.test.ts (new), policy.test.ts, converger.test.ts, retry.test.ts, src/core/reaper.test.ts, src/core/nuke-service.test.ts (F11f), e2e/warm-pool-learned.test.ts (new), e2e/warm-pool-reserve.test.ts (its expected makeReady counts re-derived with learning on; its end states hold, since the reserve is unchanged and held devices count), e2e/slow-warm-pool.test.ts.
• Docs: F18, F19, F20; docs/EVENTS.md:49 and docs/internal/EVENTS.md:87 (prose: the pool also shuts a device down to displace it); docs/CLI.md simlock nuke (clears the record); docs/internal/ARCHITECTURE.md if it names the pool's state; src/core/warm-pool/AGENTS.md (files, steps, state table, known gaps); the ADR README is the ADR PR's.

flowchart LR
  M["daemon/main.ts *"] --> ST[state openStateDatabase]
  M --> CC["createCore *"]
  CC --> WP["warm-pool converger *"]
  WP --> LK["warm-pool learned (new) *"]
  LK --> ST
  WP --> PO["warm-pool policy *"]
  WP --> CAT[core DriverCatalog]
  WP --> BUS[bus]
  NK["core NukeService *"] --> WP
  RP[core reaper] --> WP

Surface and extension points

src/core/warm-pool/index.ts grows by warmPoolOwner (the SqliteSchemaOwner), for the composition root's owner list. No new module directory.

The tables

Owner warm-pool, version 1, both STRICT, every value a column:

Rules: learn inserts the kind or moves its last_leased_at, inserts the lease row, deletes lease rows with ended_at < now - 24 h, then, if the platform holds more than 32 kinds, deletes the kinds with the oldest last_leased_at and their lease rows, all in one transaction. kinds(now) computes each kind's count as the peak of overlapping [granted_at, ended_at ?? now] intervals clipped to [now - 24 h, now], at least 1. forget and clear delete rows. The registry never reads these tables, and they never name a device id.

Contract and event changes

None on the wire: status.get is unchanged in this task (figures() returns origin: configured reports only, F8a, F17; the last task adds learned kinds). Events: no payload change; device.shutdown's catalog rows add the displacement to the initiator's prose (F11c); no new event. warm-pool.target-missed keeps firing for configured targets only (F24) and judges a merged entry by its configured count (F8c), so a configured target met at its own count emits nothing however short the learned count is. Searched src/contract for targets, warmPool: no change.

Other code on the same state

Searched lastLeaseEndedAt, targeted, state === "shutdown", beginMaintenance, listCatalog( over src/: 9 files.

• The reaper's idle-shutdown (F7): spares learned kinds' devices through targetedDevices, before the first pass too (targeted() resolves first, as for configured targets); a device beyond a kind's count is shut down after idle.shutdownAfterMs as today. Both act on ready devices under their own claims; the pool's cleanup claim and the reaper's are the same kind, the second finds the first.
• neverLeasedIdle (F8b): a device the pool created for a learned kind has no lastLeaseEndedAt; it is in the kept set, so it is not shut down as never leased and booted again in a loop. One beyond the kind's count is shut down after idle.shutdownAfterMs, as a configured target's surplus is today.
• Over-budget and displacement, both in evaluate: over-budget runs first and releases slots; displacement only when a learned kind is still short for a running slot after that; a device proposed by one is taken for the other.
• Leasing's eviction for a request: a request evicts an LRU idle device regardless of kind (ADR 0017 §3); the next pass sees the slot taken and reports the kind short. The pool never shuts a device down that serves a waiting request (F25, kept).
• Nuke (F12, F13): forget() runs under maintenance, before devices are reset, so the pass after endMaintenance finds no kind; devices shut down under nuke stay down (F13).
• The registry's own state.db transactions: separate owner, separate transactions; no row is shared.
• A second daemon on the same state.db: refused at open by state.db opens and checks at start, and the worker's registry lives in it #761, not here.

Failure modes

State written: the two tables. A restart between learn's steps leaves nothing half written (one transaction). A restart with a lease row open whose lease ended while no daemon ran: closed at start() with the start time. Two writers at once: the bus calls subscribers synchronously, one at a time.

Rules in play

• ADR 0038 §1–§5: learned kinds, the 24-hour peak, "newest kinds fill first ... may displace an older one", forgetting, "the record lives in state.db, owned by the pool".
• ADR 0036 §2, "each owner creates its own tables and keeps its own schema version".
• ADR 0017 §3, "the pool reacts to committed facts on the bus": learning subscribes to the bus, never to the lease path. §5 is narrowed by 0038 as its status line says.
• Architecture rule 16 (pure policy): the peak is computed in learned.ts and passed in the view; evaluate stays pure.
• Safety rule 2 (revalidate before acting): a displacement shutdown goes through #shutdown's revalidation like any other.
• Testing rule 10: every "newest first", "never a configured target's device" and "forgotten when" claim lists its cases below.

Tests

Seams: LearnedKinds over openMemoryStateDatabase([warmPoolOwner]) in src/core/warm-pool/learned.test.ts (model: src/core/registry.test.ts after #761); evaluate in policy.test.ts; WarmPool in converger.test.ts; CleanupReaper in src/core/reaper.test.ts; NukeService in src/core/nuke.test.ts; the CLI over the fake driver in e2e/warm-pool-learned.test.ts (model: e2e/warm-pool-reserve.test.ts); real devices in e2e/slow-warm-pool.test.ts.

• LearnedKinds.kinds counts a kind's peak over the last 24 hours, at least one (cases: three overlapping leases → 3; two in sequence → 1; one granted 25 h ago and still held → 1; one ended 24 h and 1 ms ago → pruned, 1; no lease rows → 1), newest last_leased_at first, and a full untagged kind leased twice is one row (F11e).
• RetrySchedule.failures(spec) answers the failures in a row and 0 after a success (retry.test.ts).
• LearnedKinds.learn keeps at most 32 kinds per platform, dropping the oldest by last lease with its lease rows, and a kind of the other platform is not counted (cases: 33rd iOS kind; 32 iOS plus 1 Android).
• LearnedKinds survives a reopen of the same file, and forget and clear remove the rows (cases: forget one of two; clear).
• warmPoolOwner version 1 creates both STRICT tables with the columns above (extends the owner conformance the state module gives, if any; else a direct sqlite_master read).
• evaluate fills configured targets in config order, then learned kinds newest first, booting a shut-down device of the kind before creating, and counts a learned kind's leased and reclaiming devices as had (cases: two learned kinds and room for one → the newer; peak 3 with three leased → no proposal; peak 3 with one leased and one ready → one proposal; a kind both configured (1) and learned (2) with one leased and nothing ready → one entry, origin configured, count 2, configuredCount 1, one proposal, in the configured position).
• evaluate keeps a learned kind's never-leased device out of never-leased-idle while the kind wants it, and proposes one beyond its count after idle.shutdownAfterMs (extends "shuts a never-leased device no target counts down ...").
• evaluate displaces: a learned kind short for a running slot gets the LRU idle ready device of an older learned kind shut down with reason displaced and its own boot or creation proposed, and never displaces a configured target's device, a device that serves a waiting request, for a kind short for ram-budget, device-limit or reserve, or for a configured target (one case each).
• evaluate proposes no recently-released boot: a shut-down device released a moment ago whose kind is not learned is left down (replaces "with room for one boot and two recently released shut-down devices ..." and "does not boot a shut-down device released longer ago ...").
• targetedDevices lists a learned kind's ready devices up to its count, and WarmPool.targeted() includes them before the first pass and after a restart (converger.test.ts; reaper.test.ts already proves the reaper spares whatever the set names).
• WarmPool learns on lease.granted the leased device's kind, closes it on lease.released and lease.expired, skips a physical device and learns nothing with warmPool.enabled: false (cases: virtual grant; physical grant; pool off).
• WarmPool.start() closes open lease rows whose lease the registry no longer holds, and the first pass boots the recorded kinds (restart case over one memory database shared by two WarmPools).
• WarmPool starts one catalog read per platform with a learned kind beside a pass, never two at once, and on the next pass forgets a learned kind whose model is not in models, whose OS is not in modelRuntimes[model], whose tag has no image, or whose platform has no driver, and keeps every kind of a platform whose read throws or passes 30 s on the fake clock (one case each; the listCatalog call count asserted; a pass is not delayed by a pending read).
• WarmPool.figures() and the short log list configured targets only, with a learned kind short in the same pass (status unchanged in this task).
• WarmPool emits warm-pool.target-missed for a merged entry by its configured count: a configured count 1 met with a learned count 2 short emits nothing, and ready 0 emits with count: 1 (extends the target-missed tests).
• WarmPool logs a LearnedKinds throw once and runs the pass with no learned kinds; start() survives a closeOpen throw, and the next pass calls closeOpen again and closes the rows once it succeeds.
• WarmPool forgets a learned kind on its third failed boot or creation in a row, logging one warning, and a success in between resets the count (cases: 3 in a row; 2, success, 2).
• NukeService clears the record before it resets devices, and a nuke whose forget throws still resets them (nuke-service.test.ts, F11f).
• e2e (fake driver, knownModels and availableOsVersions scripted as F11b, reclaimResult: "shutdown"): with no targets and two iOS running slots, nothing is created while the first lease is held, the released device is booted back, and the next lease of its kind is granted source: warm; then withConfig({ warmPool: { enabled: false } }) restarts and the pool shuts the device down (F11i), and the restore's restart boots it again from the record before any lease (F11g: a graceful stop would have let a boot finish, so the pool is turned off to leave the device down); after simlock nuke --yes and a restart nothing boots (three cases in one file).
• Slow lane, e2e/slow-warm-pool.test.ts: with no target, a released simulator and a released emulator are ready again before the next lease; with warmPool.enabled set false and the daemon restarted both are shut down, and with it set true and the daemon restarted both are booted again from the record (real simctl and emulator, F11g, F11i).

Done when

• pnpm test:changed passes.
• pnpm test:e2e e2e/warm-pool-learned.test.ts e2e/warm-pool-reserve.test.ts passes.
• rg -n "recently-released" src docs --glob '!docs/internal/adr/**' finds nothing.
• Needs real devices: scripts/slow-e2e.sh e2e/slow-warm-pool.test.ts passes its new "with no target" case.

Out of scope

• Learned kinds in status, the console and doctor, and the origin field (the last task).
• Fleet-level learning on the gateway.
• Any change to lease.identity: a learned kind under fresh pre-creates as a configured target does.

Depends on

• state.db opens and checks at start, and the worker's registry lives in it #761
• Tokens live in state.db on both daemon kinds #758 (both add an owner to the worker's owner list in src/daemon/main.ts)
• The warm pool boots or creates one device per queued request, and the boot cap defaults from the CPU #774

Written by an agent.

Assumptions

• Assumption: a learned kind whose platform's catalog read failed, timed out or has not answered yet is kept and planned for nothing that pass, as ADR 0038 §4 says; Tests: a converger test where a pending or failed read plans no boot for the kind and a later successful read boots it (decided in delivery, pending the maintainer)