Skip to content

fix(dispatch): scope task claim and reclaim to the owning daemon - #319

Merged
saucam merged 3 commits into
mainfrom
fix/dispatch-daemon-scope
Sep 5, 2026
Merged

fix(dispatch): scope task claim and reclaim to the owning daemon#319
saucam merged 3 commits into
mainfrom
fix/dispatch-daemon-scope

Conversation

@saucam

@saucam saucam commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

The bug

Two daemons sharing one ~/.codeoid/codeoid.db steal each other's dispatch work, and the conductor is told successful work failed.

Found by driving a real conductor dispatch end to end. The worker did its job:

README.md contains one line of text: hello from the probe workdirNothing was modified.

The conductor was told:

task c0234348 (spawn/scout) auto-BLOCKED after 2 failed attempt(s): reclaimed: stale claim

The task was marked blocked at 11:39:21 — four seconds before the worker finished at 11:39:25.

Cause

claim_owner is a daemon boot id, and dispatchReclaimStale matched:

WHERE status IN ('claimed','running')
  AND (claim_owner IS NOT ?bootId OR claimed_at IS NULL OR claimed_at + ? < ?)

claim_owner IS NOT ?bootId cannot distinguish my own crashed run from another daemon's healthy claim. Any second daemon's ordinary 5s tick reclaimed live tasks with ten minutes of lease remaining; two ticks reached failure_limit and auto-blocked work that had succeeded.

dispatchClaimNext was worse — no ownership predicate at all:

WHERE status = 'queued' AND (not_before IS NULL OR not_before <= ?) …

So a daemon could claim and execute a task belonging to another tenant, against the rule that every query is scoped to (account_id, project_id).

This is reachable, not theoretical: local-mode.md port-scopes its token file precisely so two daemons can run on one machine.

The fix

An additive owner_daemon column (host:port), set at enqueue and required by both predicates.

Scoping reclaim to the daemon's own tasks is what makes the boot-id fast path correct rather than theft — so first-tick restart recovery is preserved rather than traded away for a lease-only rule. Port-scoped for the same reason local mode scopes its token file: the port distinguishes two daemons on one machine and survives restarts.

Pre-upgrade rows have a NULL owner: still claimable by anyone so nothing is stranded, but reclaimable only on lease expiry, so the theft can't persist for exactly the rows that predate the fix.

Verification

End to end, under the real failing condition — two fixed daemons sharing one database, same dispatch that previously auto-blocked:

★ task a4e16b99 → queued
★ task a4e16b99 → running
★ task a4e16b99 → done  digest="task a4e16b99 (spawn/scout) in …"
★ event task_done

Regression tests are real — the 5 new tests fail 3/5 against the old store, pass against the new one (verified by stashing the store change and re-running).

Suite: 2434 pass, 0 fail. Typecheck and lint clean.

⚠️ Known limit — worth knowing before merging

A daemon still running the old code ignores owner_daemon and will keep reclaiming a fixed daemon's tasks. Every daemon sharing a database must be restarted for the fix to take effect.

Confirmed the hard way: my first post-fix verification still showed blocked, because an unupgraded daemon was running alongside. The new row had owner_daemon: "127.0.0.1:7455" recorded correctly — the old daemon simply didn't consult it.

Relevance to federation

This is the same bug class as docs/federation-design.md §8.1 (machine-scoped claim), appearing today with one machine and two daemons — and it settles an open schema question there: owner_daemon (executor identity) and machine_id (routing/display) are not the same column, since two daemons can share a machine. F0 keeps machine_id as a descriptive attribute.

🤖 Generated with Claude Code

saucam and others added 2 commits September 5, 2026 17:26
The bundled gemini-cli smoke test failed roughly 2 runs in 3 on the full
suite, while the CLI itself was fine. Timed directly, the spawn costs
3.3-4.2s: bun has to parse and run a large JS bundle before --version
prints. That fits inside bun's 5s default only on an idle machine, so
under the suite's parallel load it tipped over and reported a timeout
that looked like a broken provider.

Gives the test an explicit 30s ceiling, matching the convention already
used for slow tests in local-mode-server.test.ts and store-lock.test.ts.
It stays a real smoke test — it still spawns the actual binary and
asserts exit 0 plus a semver — it just no longer races the default.

Verified: the CLI returns 0.50.0 in 3.3-4.2s standalone, and the suite
now passes this test across repeated full runs.

Note for anyone chasing suite flakes: a SECOND, unrelated flake remains
and is NOT addressed here. About 1 run in 4, bun drops
src/tests/sanitize-terminal.test.ts entirely with "Cannot call
describe() after the test run has completed" — the pass count falls by
exactly its 11 tests. Confirmed pre-existing by re-running with this
change reverted. It is a runner-level module-load race rather than a
product bug, but it silently skips the terminal-escape sanitisation
tests, so it deserves its own issue.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two daemons sharing one ~/.codeoid/codeoid.db stole each other's dispatch
work. Found by driving a real conductor dispatch end to end: the worker
read the file and reported correctly, and the conductor was still told

  task c0234348 (spawn/scout) auto-BLOCKED after 2 failed attempt(s):
  reclaimed: stale claim

The task was marked blocked four seconds BEFORE the worker finished.

Cause: claim_owner is a daemon BOOT id, and dispatchReclaimStale matched
`claim_owner IS NOT ?bootId` — a predicate that cannot tell "my own
crashed run" from "another daemon's healthy claim". Any second daemon's
ordinary 5s tick therefore reclaimed live tasks with ten minutes of lease
remaining; two ticks reached failure_limit and auto-blocked work that had
in fact succeeded. dispatchClaimNext was worse: it had no ownership
predicate at all, so a daemon could claim and execute a task belonging to
another tenant, against the multi-tenancy rule that every query is scoped.

Running two daemons on one machine is supported — local-mode.md
port-scopes its token file for exactly that — so this was reachable, and
it silently reported successful work as failed.

Adds an additive `owner_daemon` column (host:port), set at enqueue and
required by both predicates. Scoping reclaim to the daemon's own tasks is
what makes the boot-id fast path CORRECT rather than theft, so
restart-recovery on the first tick is preserved rather than traded away
for a lease-only rule. Port-scoped for the same reason local mode scopes
its token file: the port distinguishes two daemons on one machine and
survives restarts. Pre-upgrade rows have a NULL owner — still claimable
by anyone so nothing is stranded, but reclaimable only on lease expiry,
so the theft cannot persist for exactly the rows that predate the fix.

Verified end to end: two fixed daemons sharing one database, same
dispatch that previously auto-blocked now reaches `done` with a result
digest and a task_done event. The five new tests fail 3/5 against the old
store and pass against the new one; full suite 2434 pass, 0 fail.

Known limit: a daemon still running the OLD code ignores owner_daemon and
will keep reclaiming a fixed daemon's tasks. Every daemon sharing a
database must be restarted for the fix to take effect — confirmed while
an unupgraded daemon was running alongside.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@saucam
saucam merged commit 329349a into main Sep 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants