fix(dispatch): scope task claim and reclaim to the owning daemon - #319
Merged
Conversation
The bundled gemini-cli smoke test failed roughly 2 runs in 3 on the full suite, while the CLI itself was fine. Timed directly, the spawn costs 3.3-4.2s: bun has to parse and run a large JS bundle before --version prints. That fits inside bun's 5s default only on an idle machine, so under the suite's parallel load it tipped over and reported a timeout that looked like a broken provider. Gives the test an explicit 30s ceiling, matching the convention already used for slow tests in local-mode-server.test.ts and store-lock.test.ts. It stays a real smoke test — it still spawns the actual binary and asserts exit 0 plus a semver — it just no longer races the default. Verified: the CLI returns 0.50.0 in 3.3-4.2s standalone, and the suite now passes this test across repeated full runs. Note for anyone chasing suite flakes: a SECOND, unrelated flake remains and is NOT addressed here. About 1 run in 4, bun drops src/tests/sanitize-terminal.test.ts entirely with "Cannot call describe() after the test run has completed" — the pass count falls by exactly its 11 tests. Confirmed pre-existing by re-running with this change reverted. It is a runner-level module-load race rather than a product bug, but it silently skips the terminal-escape sanitisation tests, so it deserves its own issue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two daemons sharing one ~/.codeoid/codeoid.db stole each other's dispatch work. Found by driving a real conductor dispatch end to end: the worker read the file and reported correctly, and the conductor was still told task c0234348 (spawn/scout) auto-BLOCKED after 2 failed attempt(s): reclaimed: stale claim The task was marked blocked four seconds BEFORE the worker finished. Cause: claim_owner is a daemon BOOT id, and dispatchReclaimStale matched `claim_owner IS NOT ?bootId` — a predicate that cannot tell "my own crashed run" from "another daemon's healthy claim". Any second daemon's ordinary 5s tick therefore reclaimed live tasks with ten minutes of lease remaining; two ticks reached failure_limit and auto-blocked work that had in fact succeeded. dispatchClaimNext was worse: it had no ownership predicate at all, so a daemon could claim and execute a task belonging to another tenant, against the multi-tenancy rule that every query is scoped. Running two daemons on one machine is supported — local-mode.md port-scopes its token file for exactly that — so this was reachable, and it silently reported successful work as failed. Adds an additive `owner_daemon` column (host:port), set at enqueue and required by both predicates. Scoping reclaim to the daemon's own tasks is what makes the boot-id fast path CORRECT rather than theft, so restart-recovery on the first tick is preserved rather than traded away for a lease-only rule. Port-scoped for the same reason local mode scopes its token file: the port distinguishes two daemons on one machine and survives restarts. Pre-upgrade rows have a NULL owner — still claimable by anyone so nothing is stranded, but reclaimable only on lease expiry, so the theft cannot persist for exactly the rows that predate the fix. Verified end to end: two fixed daemons sharing one database, same dispatch that previously auto-blocked now reaches `done` with a result digest and a task_done event. The five new tests fail 3/5 against the old store and pass against the new one; full suite 2434 pass, 0 fail. Known limit: a daemon still running the OLD code ignores owner_daemon and will keep reclaiming a fixed daemon's tasks. Every daemon sharing a database must be restarted for the fix to take effect — confirmed while an unupgraded daemon was running alongside. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
akhiljavelin
approved these changes
Sep 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Two daemons sharing one
~/.codeoid/codeoid.dbsteal each other's dispatch work, and the conductor is told successful work failed.Found by driving a real conductor dispatch end to end. The worker did its job:
The conductor was told:
The task was marked
blockedat 11:39:21 — four seconds before the worker finished at 11:39:25.Cause
claim_owneris a daemon boot id, anddispatchReclaimStalematched:claim_owner IS NOT ?bootIdcannot distinguish my own crashed run from another daemon's healthy claim. Any second daemon's ordinary 5s tick reclaimed live tasks with ten minutes of lease remaining; two ticks reachedfailure_limitand auto-blocked work that had succeeded.dispatchClaimNextwas worse — no ownership predicate at all:So a daemon could claim and execute a task belonging to another tenant, against the rule that every query is scoped to
(account_id, project_id).This is reachable, not theoretical:
local-mode.mdport-scopes its token file precisely so two daemons can run on one machine.The fix
An additive
owner_daemoncolumn (host:port), set at enqueue and required by both predicates.Scoping reclaim to the daemon's own tasks is what makes the boot-id fast path correct rather than theft — so first-tick restart recovery is preserved rather than traded away for a lease-only rule. Port-scoped for the same reason local mode scopes its token file: the port distinguishes two daemons on one machine and survives restarts.
Pre-upgrade rows have a
NULLowner: still claimable by anyone so nothing is stranded, but reclaimable only on lease expiry, so the theft can't persist for exactly the rows that predate the fix.Verification
End to end, under the real failing condition — two fixed daemons sharing one database, same dispatch that previously auto-blocked:
Regression tests are real — the 5 new tests fail 3/5 against the old store, pass against the new one (verified by stashing the store change and re-running).
Suite: 2434 pass, 0 fail. Typecheck and lint clean.
A daemon still running the old code ignores
owner_daemonand will keep reclaiming a fixed daemon's tasks. Every daemon sharing a database must be restarted for the fix to take effect.Confirmed the hard way: my first post-fix verification still showed
blocked, because an unupgraded daemon was running alongside. The new row hadowner_daemon: "127.0.0.1:7455"recorded correctly — the old daemon simply didn't consult it.Relevance to federation
This is the same bug class as
docs/federation-design.md§8.1 (machine-scoped claim), appearing today with one machine and two daemons — and it settles an open schema question there:owner_daemon(executor identity) andmachine_id(routing/display) are not the same column, since two daemons can share a machine. F0 keepsmachine_idas a descriptive attribute.🤖 Generated with Claude Code