state.db opens and checks at start, and the worker's registry lives in it #761

Part of #754.

Scope

A worker keeps its devices, leases, lease requests and components in state.db, one row per record in real columns, and writes only the rows a change touches. Each daemon opens and checks its own state.db at start. It refuses to start with state.json in its folder, with a damaged file, or with a schema version newer than its own. This task also builds the state module the later tasks add their owners to, the DeviceDataTable seam, and the fake driver's table. The iOS and Android tables are task 2. The stack merges together (delivery rule 15), so a real driver's devices are not storable until task 2 lands on top.

Kept whole, at about 40 test files: the registry cannot store a device without the state module, the seam and a device-data table, and every test that builds a registry switches to the new helper in the same change. Those tests also change their driverData to the fake's one shape, { fakeDeviceId }. While doing so, drop a test that only repeats what another proves, or that guards a race the synchronous commit removes. The maintainer asked for that pruning (spec session, 2026-10-20).

In short

The worker's registry moves from state.json to state.db. A renew, a grant or any other change commits as one SQLite transaction that writes only the rows it changed, so a renew writes one lease row however many lease requests are stored. Every refusal is the daemon's start error, logged as Daemon failed to start, exit 1. The exact messages are in the feature's Shared values.

flowchart LR
  A[startDaemon] --> B["openStateDatabase (new)"]
  B -->|old file, damage, lock, newer version| X[throw: Daemon failed to start]
  B --> C["Registry.load from rows (changed)"]
  C --> D[createCore, leasing, server as today]
  D --> E["each change: one transaction, changed rows only (changed)"]

• device-data table: a driver's own table, one row per device, e.g. fake_device_data(device_id, fake_device_id) for driverData: { fakeDeviceId: "d1" }.
• replay: a repeat of a lease.request with the same Idempotency-Key. After this task, its answer's device is the device as it is at replay time. Its lease, environment and timing are as first answered.

If this goes wrong, a user would notice a worker that forgets or changes a device, lease or request after a restart, or one that will not start.

Technical spec

Facts

Commit 8db8112b.

Modules touched

Searches: Registry.load, statePath, state.json, RegistryOptions, defaultTtlMs, driverData, #commit, createDrivers over src/, e2e/ and docs/.

• New src/state/ (index.ts, testing.ts, AGENTS.md): openStateDatabase, the old-file check, the checks and migrations.
• src/ports/sqlite.ts: SqliteSchemaOwner type; :memory: is not pre-created; passesQuickCheck and isLockError move here from src/history/database.ts, which then imports them. configureDatabase stays history's: the state module sets its own pragmas (busy_timeout 5000, WAL, synchronous=NORMAL) and creates only its own tables.
• src/core/registry.ts, src/core/driver.ts (DeviceDataTable), src/core/fake-driver.ts (fakeDeviceDataTable), src/core/testing.ts (the registry test helper, describeDeviceDataTable), src/core/index.ts.
• src/daemon/main.ts, src/daemon/test-relay-support.ts; e2e/fake-driver/index.ts (exports deviceDataTables, with its own table defined in e2e/fake-driver/, F29); e2e/helpers/physical-state.ts, e2e/helpers/physical-presence.ts, e2e/physical-enrollment.test.ts:669, e2e/daemon-folders.test.ts:111 (asserts the gateway's state.db has no devices table).
• src/core/testing.ts: HeldWritesFilesystem and the three test groups of F28 go. A synchronous commit has no write to hold, and the commit-interleaving test below replaces them.
• Comments naming state.json that the Done when rg finds: src/core/lease-request-store.ts:33, src/core/domain.ts:26, :238, src/core/instance-identity.ts:6, :34, src/gateway/drain-store.ts:17, src/ports/paths.ts:7, src/ports/logger.ts:121, src/contract/schemas.ts:268 (a comment only; the console lane then runs), src/drivers/ios/index.ts:2137, src/daemon/error-code.ts:92, docs/internal/flows/startup.md:127, src/leasing/AGENTS.md:106.
• The 28 to 36 test files of F22.
• Docs: docs/CLI.md (new section), docs/internal/ARCHITECTURE.md:29 and the module table, docs/internal/flows/startup.md:17, :46, src/core/AGENTS.md, src/daemon/AGENTS.md, src/leasing/AGENTS.md:92, src/ports/AGENTS.md, src/history/AGENTS.md if a helper moves.

flowchart LR
  daemon["daemon/main.ts *"] --> state["state (new) *"]
  daemon --> core["core Registry *"]
  daemon --> e2efake["e2e fake drivers module *"]
  state --> ports["ports/sqlite.ts *"]
  core --> ports
  core --> dd["DeviceDataTable in core/driver.ts *"]
  history[history] --> ports

Surface and extension points

• src/state/index.ts exports openStateDatabase(options): Promise<StateDatabase> and its types. Options: dataDirectory, sqlite: SqliteOpener, filesystem (old-file check only), owners: readonly SqliteSchemaOwner[], oldFiles: readonly string[], upgradeStep: string (the role's <step> of the old-file message in the feature's Shared values), logger. StateDatabase: database: SqliteDatabase, transaction<T>(work: () => T): T, close(). testing.ts exports openMemoryStateDatabase(owners) (:memory: through the real adapter).
• Extension point (architecture rules 19, 20): DeviceDataTable has four implementations named (fake now, iOS and Android in task 2, Vega later). It differs from rule 20's shape in two ways, both because a table belongs to a driver (rule 2): the registry is keyed by platform, not by name, and each driver's own test runs the shared suite with its own examples. Definition: DeviceDataTable extends SqliteSchemaOwner with write(database, deviceId, driverData): void (insert or replace) and read(database, deviceId): unknown (throws when the row is missing). Its registry is the ReadonlyMap<Platform, DeviceDataTable> the composition root passes to Registry.load. The conformance suite is describeDeviceDataTable(name, table, examples) in src/core/testing.ts: each implementation's own test file runs it with that driver's example values, since only the driver knows its shapes. This task runs it for fakeDeviceDataTable. The registry never names a platform (architecture rule 21).
• Registry.load(options) takes state: StateDatabase, deviceData: ReadonlyMap<Platform, DeviceDataTable>, clock, idGenerator, eventBus, leaseIdentity?, leaseRequestLimits?. filesystem, statePath and defaultTtlMs go away. registryOwner (the registry SqliteSchemaOwner) is exported from core for the composition root.

The tables

Owner registry, version 1. Every table is STRICT, and a required field is NOT NULL. Every value is a column. Optional fields are NULL when absent. Booleans are 0/1.

Owner fake-device-data, version 1: fake_device_data(device_id key, fake_device_id TEXT NOT NULL), STRICT, in src/core/fake-driver.ts; write refuses anything but { fakeDeviceId: string } with Invalid fake driver data for device <id>. Every unit test's driverData becomes { fakeDeviceId: <its device's id> } (F27). The e2e fake module defines its own e2e-fake-device-data table with the same columns and rule, since e2e cannot import src/core/fake-driver.ts (F29). One fake table serves both platforms in a map.

Rules:

• A virtual device has exactly one row in its platform's device-data table. A physical device has none and loads with driverData: {} (F10).
• A commit writes the rows whose record object differs from the committed one (records are immutable), deletes the rows of records no longer present (pruned requests), and writes the device-data row of every device row it writes, all in one StateDatabase.transaction. Memory and listeners update after it returns (F2). A throw rolls back and leaves memory as it was.
• Load reads every table. A row with a value outside its type, a virtual device with no device-data row, a device whose platform has no table in the map, or a grant naming a missing device throws RegistryLoadError naming the table and key.
• In memory a granted request keeps its lease, environment and timing. Its grant.device is the current device record, read when the grant is read (F13).
• The load-time fixes of F4 and the unknown-field passthrough are removed.

Start (src/daemon/main.ts)

Before createAuth (F15), both roles: openStateDatabase on the role's data directory. Worker owners: registryOwner plus each table of the device-data map. Gateway owners: none in this task. Worker old files: state.json. Gateway: none in this task. Its steps, in this order, each refusing with the feature's Shared values message:

1. Any old file present → refuse.
2. node:sqlite unavailable → refuse.
3. Open, then set the state module's own pragmas: busy_timeout=5000, journal_mode=WAL, synchronous=NORMAL. A lock error past 5 s → lock refusal. Any other open error → damage refusal (open: <message>).
4. PRAGMA quick_check not exactly one ok row → damage refusal (quick_check: <first row>).
5. In one BEGIN IMMEDIATE transaction: create schema_versions if absent, then read each owner's version inside it. A stored version above owner.version → newer-version refusal. Below it → run its steps and set its version. A throw rolls back the whole transaction, every owner's steps included → migration refusal. Reading the versions inside the write transaction is what makes two daemons starting at once migrate once.

Before step 1, startDaemon probes the daemon socket (refuseWhenDaemonAnswers in src/daemon/connection-host.ts): a socket that answers throws DaemonAlreadyRunningError and nothing touches state.db, so a second daemon never migrates a live daemon's file. A lock error at steps 3, 4 or 5 past 5 s gives the lock message, never the damage one. A start that fails after the open (e.g. the socket claim is lost) closes state.db before it throws.

The device-data map comes from one function, resolveDeviceDataTables(options) in src/daemon/main.ts, which startDaemon calls once and passes to both openStateDatabase (each distinct table once, as an owner) and Registry.load. Its order: StartDaemonOptions.deviceDataTables (new, a ReadonlyMap<Platform, DeviceDataTable>; the 37 main.test.ts starts that inject drivers pass the fake table for both platforms, and so do the drivers-injecting starts in src/agent-device/edge.test.ts, src/agent-device/gateway-front.test.ts, src/daemon/gateway-fleet.e2e.test.ts, gateway-modules.test.ts, lease-requires.e2e.test.ts, optional-modules.test.ts, server.test.ts, stream-gateway.e2e.test.ts and worker-capabilities.e2e.test.ts); else, with SIMLOCK_DRIVERS_MODULE, the module's deviceDataTables export (the same map type), read before the state opens, and a module without it fails start, as a missing createDrivers does (F17); else productionDeviceDataTables(), which is empty in this task (task 2 fills it). statePath leaves StartDaemonOptions. The leftover sweep keeps dataDirectory only. state.db is never deleted or renamed by Simlock. state.close() runs last in each role's dispose, after eventHistory.stop() (src/daemon/main.ts:553-569, :904-909); dispose runs after stopAuxiliary has stopped HTTP and the uplinks.

Contract and event changes

None: no operation, schema, error code or event changes (searched src/contract for statePath, state.json: none). driverData stays in list.get and doctor answers, from the device-data row (F14).

Other code on the same state

Searched state.json, statePath, Registry over src/ and e2e/ (F22, F23, F24). Every registry write stays inside SerializedDecision.run as today (src/core/AGENTS.md rule). The commit is now synchronous, so no other commit can run between its read and its write. The CLI reads no registry state (src/cli imports no Registry). History writes its own file (ADR 0032). Tasks 3 and 4 add owners to the same state.db, through Shared values.

Failure modes

Restart in the middle of a commit: the transaction either committed or did not (WAL). Two daemons on one folder: the second loses the socket claim as today, and its open waits on the lock at most 5 s.

Rules in play

• ADR 0036 §1–§5: the whole task.
• ADR 0019 §4, "the same … write that commits the lease marks that request granted": createLease stays one transaction.
• ADR 0018 §2, "a grant and the device's move to leased are one … write": one transaction.
• ADR 0022 §1, physical records "in their own top-level list": now devices.physical = 1, with enrolled apps in a child table.
• ADR 0027, every component guarded: src/state is found by its index.ts.
• Architecture rules 3 and 21, "Adding a driver must require zero core changes": a driver adds a table to the map.
• Architecture rule 9, ports: SQLite only through SqliteOpener; src/ports/AGENTS.md "sqlite.ts is the only file that names node:sqlite".
• Architecture rule 18: src/state exports one factory and its types.
• Safety rule 1, registry-only destruction: nothing destroys a device that state.db does not hold; the upgrade step orphans nothing.
• src/core/AGENTS.md rule "Only Registry writes state.json" becomes "Only Registry writes the registry tables".

Tests

Seams: Registry.load over openMemoryStateDatabase (model: src/core/registry.test.ts), openStateDatabase (new src/state/state.test.ts, model: src/history/history.test.ts, temp dir), and startDaemon (model: src/daemon/main.test.ts:2564).

• A registry restarted on the same state.db returns every field of a virtual device (driverData: { fakeDeviceId }), a physical device (enrolled apps in order), a lease, a request in each of open, granted, failed, cancelled, and a component (receipt pairs) exactly as committed, with optional fields both set and absent (temp file; extends the restart test at src/core/registry.test.ts:195).
• A renew with 1,000 settled requests stored changes exactly one row (SELECT total_changes() before and after).
• A grant writes the lease, the device's move to leased and the request's granted result in one transaction: a DeviceDataTable.write that throws during the grant leaves no lease, the device ready and the request open, both in memory and after a restart.
• A commit that throws in SQLite leaves snapshot and the commit listeners as before the call, and the caller's promise rejects.
• Pruning deletes the rows of dropped requests and their grant and environment rows (retention and maxRecords cases; extends the existing pruning test).
• A replayed answer's device is the current device record, before and after a restart, while its lease, environment and timing are as first granted.
• Load refuses with RegistryLoadError naming the table and key for: a virtual device with no device-data row, a device whose platform has no table, a grant naming a missing device, a state value not in the type.
• A physical device loads with driverData: {} and has no device-data row.
• A commit's write runs with no await between reading the registry's current records and swapping them in memory: two transitions of two devices started in one tick both persist, in memory and after a restart. This replaces the F28 groups.
• describeDeviceDataTable asserts, for each example: write then read returns it deep-equal with no undefined keys, a second write of another example replaces it, and read of a missing id throws. Run for fakeDeviceDataTable.
• resolveDeviceDataTables answers the option's map when given, else the drivers module's export, else productionDeviceDataTables(). startDaemon passes the same map to the open and to Registry.load: a fake-table device committed before a restart loads after it.
• openStateDatabase refuses with each Shared values message, word for word, for: an old file present (and creates no state.db), a file failing quick_check (bytes overwritten in the page area), a file that is not SQLite, a lock held past 5 s by another connection, a stored version one above the owner's, a migration step that throws (its version bump is rolled back).
• openStateDatabase on an empty folder creates state.db mode 0600 and records each owner's version. A second open runs no migration.
• The versions are read inside the write transaction: with an opener whose connection, on its BEGIN IMMEDIATE, first migrates the same file synchronously on a second raw connection (the owners' steps run there, versions set), the first open runs no migration step (steps counted).
• A lock held past 5 s by another connection at the quick_check and at BEGIN IMMEDIATE gives the lock message, not the damage one.
• A second startDaemon on a folder whose daemon runs rejects with DaemonAlreadyRunningError and leaves state.db's schema_versions rows and bytes unchanged, even with an owner at a newer version (a new case beside src/daemon/main.test.ts:2564).
• startDaemon as a worker with state.json in its folder throws the old-file message with the worker's step, and creates no state.db.
• The built daemon entry (node dist/daemon/main.js, SIMLOCK_HOME a temp dir holding state.json) exits 1 and its daemon.log ends with a Daemon failed to start line whose fields.message is that message (a new case beside e2e/daemon-folders.test.ts:170, which already runs the entry and checks its Daemon failed to start line).
• startDaemon with a SIMLOCK_DRIVERS_MODULE that has no deviceDataTables export fails start naming it (extends src/daemon/main.test.ts:2564).
• A worker started with startDaemon and fake drivers, leased, stopped and started again, lists the same device and lease (extends src/daemon/main.test.ts:897).
• The e2e physical-device flows pass with seedPhysicalState, patchPhysicalRecords and physicalRecords reading and writing state.db (the 6 files of F23, unchanged assertions).

Done when

• pnpm test:changed passes.
• pnpm test:e2e e2e/physical-enrollment.test.ts e2e/daemon-folders.test.ts e2e/physical-devices-core.test.ts e2e/physical-leasing.test.ts e2e/physical-reclaim.test.ts e2e/physical-presence-loss.test.ts e2e/physical-presence-loss-recovery.test.ts e2e/physical-gateway.test.ts passes.
• rg -n "state\.json" src docs e2e --glob '!docs/internal/adr/**' finds only the old-file check, its tests and the docs/CLI.md upgrade section.
• docs/CLI.md has ### Breaking: state moves to state.db under simlock daemon, with the upgrade step (release every lease, simlock nuke --delete-devices with the older Simlock, remove state.json, add physical devices again with simlock device add), how to read the refusal (simlock daemon logs), and the downgrade step (release every lease and simlock nuke --delete-devices before starting an older Simlock).

Out of scope

• The iOS and Android device-data tables, and filling the production map (task 2).
• Tokens and tokens.json (task 3), drained workers and workers.json (task 4).
• The CLI printing the refusal (task 5).
• Any import of state.json.

Depends on

• docs(adr): 0036 daemon state is a SQLite database, and each owner keeps its own tables #755 (ADR 0036)

Written by an agent.