🐛 fix(contribute): make the task lease outlive the hub process (#5681)

CHANGELOG.md

@@ -22,6 +22,7 @@ Hive did not historically maintain a complete changelog. This file starts a prag
 ### Fixed
 
 - The reviewer lane's escalated-PR work list is now ordered by each PR's real creation time instead of a PR-number proxy ([#5617](https://github.com/kubestellar/hive/issues/5617)). The list is capped at three PRs per kick and documented "oldest first", but its rows carried no age signal at all — `ci-failing.json` recorded no creation time, so ordering fell back to (repo name, PR number). PR numbers are monotonic only *within* a repo, so that proxy sorted by repo **name** first, and against the per-kick cap the result was starvation rather than mere cosmetic disorder: on a multi-repo hive, a month-old escalated PR in a late-alphabet repo sat behind newer ones from an early-alphabet repo on every kick, indefinitely. The creation time was already fetched during PR enumeration and simply never threaded into the work-list artifact; it now is, and each row renders an `opened:` line so the reviewer can confirm the ordering without running `gh pr list` (which the kick's invariants forbid). Rows from an older hub that recorded no creation time keep the previous (repo, number) proxy among themselves and sort after every row whose age is known, so a stale artifact degrades to the old behavior rather than being reshuffled.
+- A hub restart no longer throws away every in-flight contributor task ([#5681](https://github.com/kubestellar/hive/issues/5681)). A contributor relay holds one task at a time and keeps working through a brief disconnect, re-asserting the task when it reconnects; the hub honours that re-assertion only against a **server-issued lease**, which is deliberate — a client must never be able to assert ownership of work the server did not assign. But leases lived only in the hub's memory, so a restart emptied the registry and *no* in-flight resume could match: the relay was told `no active lease for this task`, its agent was interrupted mid-turn, and the identical issue was handed straight back as a fresh assignment seconds later. Observed 2026-09-02: revoked at 14:24:40, reassigned to the same relay at 14:24:44, discarding two and a half minutes of a turn that was progressing normally. Self-upgrade rolls made this routine rather than rare, and it hit every contributor holding a task at the moment of any restart. The lease registry is now persisted to `/data/contributors/task-leases.json` (owner-only, beside the existing contributor ledgers) on every assignment, renewal and release, and restored at startup. Nothing about who may claim what is loosened: the restored record is one the hub itself wrote, a resume is still matched exactly on identity, task id, repo, number and generation, and a lease already past its window is dropped at load rather than restored. Two couplings move with it — the renewed window is what gets persisted (so a task that has been progressing for longer than the 30-minute lease TTL does not come back already expired), and the in-memory assignment-generation counter is advanced past every restored lease at boot, without which a post-restart assignment could mint a fencing token that aliased a pre-restart one. For the first two minutes after a restart a restored lease also holds its work item, so an issue whose relay is still reconnecting is not offered to a second contributor in the meantime.
 
 - Hold-gated PRs can no longer silently carry unreviewed commits into a merge ([#5589](https://github.com/kubestellar/hive/issues/5589)). A PR sitting under a hold label could accumulate commits from other authors or agents (typically a worktree cut from a contaminated base), and once the hold lifted the merge lanes treated it like any other approved PR — the diff a human saw under the hold was not the diff that merged. The governor now snapshots each hold-gated PR's head SHA plus commit/author sets into a PVC ledger (`/data/metrics/hold-guard.json`) and compares at lift time: an unchanged head clears silently, while any drift keeps the PR out of `merge-eligible.json` and `ci-failing.json`, posts a one-time evidence comment naming the unreviewed commits and authors (plain text, never @-mentions), re-applies the `hold` label, and re-pins the snapshot to the drifted head so removing the re-applied hold after reading the evidence is the fresh approval. Both auto-merge sweeps additionally now respect hold and `do-not-merge` labels directly — the self-authored sweep previously listed PRs independently of the enumeration hold gate and would squash a hold-labelled App PR on green. from [#5480](https://github.com/kubestellar/hive/issues/5480): an App-bot comment review submitted through `hive-review` and attributed as `agent_pr_reviewed`, plus an advisory bead included in the advisory digest. `hive-review` now prints its asynchronous result path, and the kick requires a confirmed `ok` result plus the bead before removing `needs-human`, applying a terminal reviewer label, or closing; if either write fails, the PR remains in the human queue for a later retry instead of becoming silently adjudicated.
 - Version flips no longer trap hosted spokes with large, long-lived `/data` PVCs in a startup-probe death loop ([#5525](https://github.com/kubestellar/hive/issues/5525)). The v5 entrypoint synchronously ran recursive per-agent `chown` and shared-home `chmod` passes before the Go server could bind `:3002`; on an NFS/RWX volume with months of worktrees the walk exceeded the probe budget, kubelet killed the container with exit 137, and the next boot restarted the same walk from the beginning. Size-dependent permission repair now runs in a root background worker, so the dashboard and health endpoint start independently of PVC size. Protected completion markers bind each finished pass to the ownership-schema revision and target UID, and the agent manager waits for both the shared-home and per-agent markers before touching that agent's tree, preserving UID isolation without putting the server behind the migration. Completed steady-state boots skip the recursive pass, while a changed UID invalidates its marker and repairs only the affected agent before launch.

src/docs/contributor-relay.md

@@ -408,6 +408,10 @@ Task prompt sent to CLI
 
 That last line types a fresh prompt into a pane whose CLI is still mid-turn, interrupting it. Renewing the lease on every progress report keeps the two clocks together: a task the hub still considers alive is a task the relay can still resume.
 
+**The lease has to outlive the hub process.** Leases used to live only in the hub's memory, so a restart — which self-upgrade rolls ([#5391](https://github.com/kubestellar/hive/issues/5391)) make routine rather than rare — emptied the registry while every relay carried on working. After the roll *no* in-flight resume could match, and each one produced the same four lines above, this time for a reason the relay could do nothing about ([#5681](https://github.com/kubestellar/hive/issues/5681)). Observed 2026-09-02: revoked at 14:24:40, the same issue reassigned to the same relay at 14:24:44, discarding two and a half minutes of a turn that was progressing normally. The registry is now written to `/data/contributors/task-leases.json` (owner-only) on every assignment, renewal and release, and read back at startup, so a restart is just a longer-than-usual disconnect.
+
+This does not loosen who may claim what. The restored record is one the *hub itself* wrote, and a resume is matched against it exactly as before; a lease whose window has already passed is dropped at startup rather than restored. For the first two minutes after a restart the hub also treats a restored lease as a hold on its work item, so an issue whose relay is still reconnecting is not handed to somebody else in the meantime.
+
 A resume that is genuinely refused — an operator yanked the task, or the relay stopped reporting for longer than the lease window — still ends in `task_revoke`, and that is correct. The relay clears its task and asks for new work.
 
 **A dropped socket is not a failed issue.** The disconnect books a short cooldown on the issue so a second session cannot pick it up during the reconnect window and file a duplicate PR ([#2356](https://github.com/kubestellar/hive/issues/2356)). That cooldown no longer counts toward the consecutive-failure quarantine: three drops on a flaky connection used to park a perfectly workable issue for six hours with nothing having actually failed. Real failures — `task_failed`, the relay's own progress watchdog giving up, the wedged-task backstop — still count, and still quarantine.

src/docs/design/agent-state-inventory.md

@@ -184,8 +184,8 @@ key generations (`/data/saas/hub-generations.json`).
 | 28 | Failure cooldowns + quarantine counters | `failedTasks` / `consecutiveFailures` (`:373`, `:381`), persisted to `/data/contributors/failed-tasks.json` | Survives (#2435 livelock fix, made durable with the ledger) | **fixed** |
 | 29 | No-PR completion streaks | `noPRStreaks` (`:394`), `/data/contributors/no-pr-streaks.json` | Survives (#3980; geometric backoff must outlive the completion entry it escalates) | **fixed** |
 | 30 | `no_work_needed` verdicts | `noWorkVerdicts` (`:416`), `/data/contributors/no-work-verdicts.json` | Survives (#3987/#3997) — "persisted in the same PVC-backed ledger dir as the cooldowns so a pod restart does not forget the verdict" | **fixed** |
-| 31 | Task leases — the server-authoritative record of what was issued to whom | `leases` (`:467`), in-memory only | **All active leases void.** A reconnecting relay's `task_progress` can only re-adopt against an exact unexpired lease (the C4 fix), so after a roll no in-flight contributor task can resume — the relay is asked to re-`ready` and the task is re-dispatched from scratch. Deliberate security posture (never rebuild ownership from client-supplied fields); the durability cost is accepted, not accidental | **volatile** (by security choice) |
-| 32 | Assignment generation counter (`taskGen`) — fencing tokens | `:341`, `atomic.Uint64`, in-memory | Restarts at zero. Safe *only because* row 31 also dies: re-adoption requires a matching lease, so a stale pre-roll generation can never coincide with a live post-roll lease. The fencing guarantee is load-bearing across two volatile structures | **volatile** (correct today, fragile coupling) |
+| 31 | Task leases — the server-authoritative record of what was issued to whom | `leases`, persisted to `/data/contributors/task-leases.json` (0600) | Survives ([#5681](https://github.com/kubestellar/hive/issues/5681)). Previously **all active leases went void**: a reconnecting relay could only re-adopt against an exact unexpired lease (the C4 fix), so after a roll no in-flight contributor task could resume — the agent was interrupted mid-turn and handed the identical issue back seconds later. The durability cost was accepted as a security posture, but the posture never required volatility: the restored record is one the *server* wrote, matched exactly as before, so nothing is rebuilt from client-supplied fields. Expired records are dropped at load rather than restored | **fixed** |
+| 32 | Assignment generation counter (`taskGen`) — fencing tokens | `:341`, `atomic.Uint64`, in-memory; high-water mark re-derived from row 31 at boot | Restarts at zero, then `loadLeases` advances it past every restored lease's generation ([#5681](https://github.com/kubestellar/hive/issues/5681)). This is what residual 2 below warned about: making row 31 durable without this would let a post-roll assignment mint a generation that ALIASES a restored one, and the #2568 Gate would accept a pre-roll straggler against a brand-new task. The fence no longer depends on two structures dying together | **fixed** (re-derived, not persisted) |
 | 33 | Per-tier rate-limit ledger (`assignmentTimes`, #2436/#2566) | `:435`, in-memory only | **Every contributor's rolling per-hour/per-day assignment count resets to zero.** The tier limits an operator set are silently un-enforced for up to a day's window after each roll — same "admin-visible number is inert" class that #2566 fixed, reintroduced at roll frequency | **volatile** |
 | 34 | Live connections + `currentTask` + pending-auth counter | `connections` / `ContributorConnection.currentTask` (`:73`), `pendingConns` | Connection-scoped by nature; relays reconnect with backoff | **benign** (given row 31) |
 | 35 | Activity feed (50 entries) + SSE fan-out registry | `activity` (`:351`), `contribute_sse.go` | Operations view starts empty | **benign** |
@@ -301,11 +301,14 @@ Honest gaps in the current state of things, and in this inventory:
    cycle's worth of pause/override/LastKick mutations. Nobody has been burned
    hard enough to make it write-through; the RFC's journal, if it lands,
    should not inherit this cadence.
-2. **Row 32's coupling is undocumented elsewhere.** The generation-fencing
-   guarantee holds because `taskGen` and `leases` share a process lifetime.
-   Any change that persists one without the other (e.g. "let's make leases
-   survive restarts") silently breaks the fence. This document is currently
-   the only place that states it.
+2. ~~**Row 32's coupling is undocumented elsewhere.**~~ **Resolved by
+   [#5681](https://github.com/kubestellar/hive/issues/5681).** The warning was
+   accurate and it was load-bearing: leases were made to survive restarts, and
+   the fence would have broken silently had `taskGen` not been re-derived from
+   the restored leases in the same change. The coupling is now stated in
+   `loadLeases` itself and pinned by a test
+   (`TestLeaseRestart_GenerationAdvancesPastRestoredLeases`), so it is no longer
+   carried only by this document.
 3. **Rate limits reset on every roll** (row 33). Known now; not yet an
    incident; cheap to fix inside the existing ledger dir.
 4. **Webhook loss** (row 3) has no re-derivation path. It predates this

src/pkg/dashboard/api_contribute_test.go

@@ -31,16 +31,19 @@ func setupContributeEnv(t *testing.T) {
 func redirectContributeWSDisk(t *testing.T, dir string) {
 	t.Helper()
 	oldActivity, oldCompleted, oldFailed, oldNoPR := activityFilePath, completedTasksFile, failedTasksFile, noPRStreaksFile
+	oldLeases := taskLeasesFile
 	oldAsyncActivitySave := asyncActivitySave
 	oldActivityPersistenceEnabled := activityPersistenceEnabled
 	activityFilePath = filepath.Join(dir, "activity.json")
 	completedTasksFile = filepath.Join(dir, "completed-tasks.json")
 	failedTasksFile = filepath.Join(dir, "failed-tasks.json")
 	noPRStreaksFile = filepath.Join(dir, "no-pr-streaks.json")
+	taskLeasesFile = filepath.Join(dir, "task-leases.json")
 	asyncActivitySave = false
 	activityPersistenceEnabled = false
 	t.Cleanup(func() {
 		activityFilePath, completedTasksFile, failedTasksFile, noPRStreaksFile = oldActivity, oldCompleted, oldFailed, oldNoPR
+		taskLeasesFile = oldLeases
 		asyncActivitySave = oldAsyncActivitySave
 		activityPersistenceEnabled = oldActivityPersistenceEnabled
 	})