🐛 bug: a hub restart makes every in-flight contributor task unresumable — the agent is interrupted, then handed the same issue back seconds later

[Danathar]

Summary

When the hub restarts to take an upgrade, every contributor that is mid-task has its
work thrown away — and then is handed the identical issue back seconds later.

A contributor relay holds one task at a time and keeps working through a brief
disconnect, re-asserting the task when it reconnects. The hub only honors that
re-assertion if it can match a server-issued lease (lookupLease), which is
deliberate: #C4 established that a client must never be able to assert ownership of
work the server did not assign. But leases live only in the hub's memory. A restart
empties that store, so after an upgrade no in-flight resume can ever match, the
relay is told no active lease for this task, and the revoke path interrupts the
agent mid-turn.

The waste is visible in the ordering: the hub revoked the task at 14:24:40 and
re-assigned the same issue to the same relay at 14:24:44. Ownership was never in
question — only the record of it. The agent restarted the identical work from zero,
discarding about two and a half minutes of a turn that was progressing normally.

Blast radius: every contributor holding a task at the moment of any hub restart.
Self-upgrade rolls make this routine rather than rare.

Observed, 2026-09-02

Contributor Danathar, claude backend, relay protocol 1.2, working issue #5617.
Relay console (timestamps local, UTC-4):

[10:22:03] Task assigned: issue kubestellar/hive#5617 — [v5] reviewer lane follow-ups
[10:22:06] Task prompt sent to CLI
[10:24:09] Connection closed (code=1012 service restart: hub restarting for upgrade). Reconnecting in 1000ms...
[10:24:39] Connection closed (code=1006 abnormal closure — no close frame). Reconnecting in 1000ms...
[10:24:40] Connected / Authenticated as c-ca8187f7480c (tier: contributor)
[10:24:40] Reconnected while working on kubestellar/hive#5617 — resuming
[10:24:40] Task revoked: ct-kubestellar/hive-5617-1788358923 — no active lease for this task
[10:24:43] Relaunching claude after task revoke: …
[10:24:44] Task assigned: issue kubestellar/hive#5617 — [v5] reviewer lane follow-ups
[10:24:45] Task prompt sent to CLI

Hub activity feed, same window:

2026-09-02T14:24:40Z | joined
2026-09-02T14:24:44Z | picked up | issue kubestellar/hive#5617

The agent's pane at the moment of the revoke:

● I'll start by getting the checkout and reading the issue.
  Searched for 9 patterns, read 3 files, listed 3 directories, ran 13 shell commands
  ⎿  Interrupted · What should Claude do instead?

Thirteen shell commands and a fork/clone in, then killed. The task file went from
gen=75 to gen=2 — a fresh task record for the same issue.

Note the 1012 is #5390's fix working correctly: the hub now closes cleanly and says
why, instead of dropping a silent 1006. This issue is about what happens after
that clean close.

Why the resume cannot succeed

src/pkg/dashboard/contribute_ws.go — the resume branch of task_progress:

lease := h.lookupLease(identity, msg.TaskID, canonRepo, msg.Number, msg.TaskGen, time.Now())
if lease == nil {
    …
    _ = sendJSON(conn, WSMessage{Type: "task_revoke", …, Reason: "no active lease for this task"})
    continue
}

The rejection is correct given what it knows. The gap is upstream: leases are held in
process memory (lastLeaseRenew on the in-memory contributor, expiresAt: now.Add(leaseTTL)), and nothing reconstructs them across a restart. leaseTTL == wsTaskTimeout, so a lease that was minutes from expiry is indistinguishable after a
restart from one that never existed — both look like nil.

#4260 established the contract that a relay reconnecting inside the backoff window
resumes rather than losing its task, and
contribute_reconnect_resume_test.go pins it: "It must be resumed, not told 'no
active lease for this task'." That test exercises a reconnect to a live hub, so
it passes while this case fails — the contract simply has no coverage across a
process boundary.

Fix shape

Make the lease survive a restart, without weakening #C4. The relay's claim must still
be checked against a server-issued record; the record just needs to outlive the
process. Options, roughly in order of intrusiveness:

1. Persist the lease next to the contributor profile, which is already written to
   disk on every task transition (saveContributorProfile). On boot, load unexpired
   leases so lookupLease can match a legitimate resume. The lease is small and
   already has an absolute expiresAt, so a stale one expires on its own.
2. Reconstruct from the durable run record. appendTaskRun/the task-run log
   already persists assignment facts; an assignment with no terminal record and a
   timestamp inside leaseTTL is exactly a live lease.
3. Drain before exiting. 🐛 hub does not drain contributor WebSockets on SIGTERM — every upgrade roll is a silent 1006 #5390 made the close clean; going further and letting
   in-flight tasks reach a terminal report before the process exits would shrink the
   window, though it cannot close it — an upgrade will not wait out a 20-minute turn.

(1) is the smallest change that actually fixes it. (3) is complementary, not a
substitute.

Whatever the mechanism, the observable contract to pin in a test is the one this
incident violated: a relay that reconnects after a hub restart, inside leaseTTL,
with a matching {identity, task_id, repo, number, generation}, resumes its task —
and the hub does not re-offer that issue to anyone, least of all to the relay it just
revoked it from.

Related: #4260 (the resume contract), #5390 (clean close on upgrade), #5391
(self-upgrade roll frequency, which sets how often this fires), #5090.

[clubanderson]

Closed as already fixed on target branch v5 at 67430b2 by merged PR #5688. Evidence: src/pkg/dashboard/contribute_ws.go now persists task leases via taskLeasesFile (lines 507-517), restores them at startup in loadLeases (lines 843-930), calls loadLeases before relays can reconnect (lines 1150-1156), and resumes only through lookupLease with the restored server-issued tuple (lines 720-759 and 3718-3790). That directly addresses the restart-empty-lease-store failure described here.