🐛 fix(contribute): persist task leases across hub restarts (v4 port of #5688)

changelog.d/fixed-5681-task-lease-restart.md

@@ -0,0 +1 @@
+- A hub restart no longer throws away every in-flight contributor task ([#5681](https://github.com/kubestellar/hive/issues/5681), ported from v5 [#5688](https://github.com/kubestellar/hive/pull/5688)). A contributor relay holds one task at a time and keeps working through a brief disconnect, re-asserting the task when it reconnects; the hub honours that re-assertion only against a **server-issued lease**, which is deliberate — a client must never be able to assert ownership of work the server did not assign. But leases lived only in the hub's memory, so a restart emptied the registry and *no* in-flight resume could match: the relay was told `no active lease for this task`, its agent was interrupted mid-turn, and the identical issue was handed straight back as a fresh assignment seconds later. Self-upgrade rolls made this routine rather than rare, and it hit every contributor holding a task at the moment of any restart. The lease registry is now persisted to `/data/contributors/task-leases.json` (owner-only, beside the existing contributor ledgers, written via the crash-safe unique-temp + fsync + rename idiom) on every assignment, renewal and release, and restored at startup. Nothing about who may claim what is loosened: the restored record is one the hub itself wrote, a resume is still matched exactly on identity, task id, repo, number and generation, and a lease already past its window is dropped at load rather than restored. Two couplings move with it — the renewed window is what gets persisted (so a task that has been progressing for longer than the 30-minute lease TTL does not come back already expired), and the in-memory assignment-generation counter is advanced past every restored lease at boot, without which a post-restart assignment could mint a fencing token that aliased a pre-restart one. For the first two minutes after a restart a restored lease also holds its work item, so an issue whose relay is still reconnecting is not offered to a second contributor in the meantime.

src/docs/contributor-relay.md

@@ -453,6 +453,10 @@ Task prompt sent to CLI
 
 That last line types a fresh prompt into a pane whose CLI is still mid-turn, interrupting it. Renewing the lease on every progress report keeps the two clocks together: a task the hub still considers alive is a task the relay can still resume.
 
+**The lease has to outlive the hub process.** Leases used to live only in the hub's memory, so a restart — which self-upgrade rolls ([#5391](https://github.com/kubestellar/hive/issues/5391)) make routine rather than rare — emptied the registry while every relay carried on working. After the roll *no* in-flight resume could match, and each one produced the same four lines above, this time for a reason the relay could do nothing about ([#5681](https://github.com/kubestellar/hive/issues/5681)). Observed 2026-09-02: revoked at 14:24:40, the same issue reassigned to the same relay at 14:24:44, discarding two and a half minutes of a turn that was progressing normally. The registry is now written to `/data/contributors/task-leases.json` (owner-only) on every assignment, renewal and release, and read back at startup, so a restart is just a longer-than-usual disconnect.
+
+This does not loosen who may claim what. The restored record is one the *hub itself* wrote, and a resume is matched against it exactly as before; a lease whose window has already passed is dropped at startup rather than restored. For the first two minutes after a restart the hub also treats a restored lease as a hold on its work item, so an issue whose relay is still reconnecting is not handed to somebody else in the meantime.
+
 A resume that is genuinely refused — an operator yanked the task, or the relay stopped reporting for longer than the lease window — still ends in `task_revoke`, and that is correct. The relay clears its task and asks for new work.
 
 **A dropped socket is not a failed issue.** The disconnect books a short cooldown on the issue so a second session cannot pick it up during the reconnect window and file a duplicate PR ([#2356](https://github.com/kubestellar/hive/issues/2356)). That cooldown no longer counts toward the consecutive-failure quarantine: three drops on a flaky connection used to park a perfectly workable issue for six hours with nothing having actually failed. Real failures — `task_failed`, the relay's own progress watchdog giving up, the wedged-task backstop — still count, and still quarantine.

src/pkg/dashboard/api_contribute_test.go

@@ -31,16 +31,19 @@ func setupContributeEnv(t *testing.T) {
 func redirectContributeWSDisk(t *testing.T, dir string) {
 	t.Helper()
 	oldActivity, oldCompleted, oldFailed, oldNoPR := activityFilePath, completedTasksFile, failedTasksFile, noPRStreaksFile
+	oldLeases := taskLeasesFile
 	oldAsyncActivitySave := asyncActivitySave
 	oldActivityPersistenceEnabled := activityPersistenceEnabled
 	activityFilePath = filepath.Join(dir, "activity.json")
 	completedTasksFile = filepath.Join(dir, "completed-tasks.json")
 	failedTasksFile = filepath.Join(dir, "failed-tasks.json")
 	noPRStreaksFile = filepath.Join(dir, "no-pr-streaks.json")
+	taskLeasesFile = filepath.Join(dir, "task-leases.json")
 	asyncActivitySave = false
 	activityPersistenceEnabled = false
 	t.Cleanup(func() {
 		activityFilePath, completedTasksFile, failedTasksFile, noPRStreaksFile = oldActivity, oldCompleted, oldFailed, oldNoPR
+		taskLeasesFile = oldLeases
 		asyncActivitySave = oldAsyncActivitySave
 		activityPersistenceEnabled = oldActivityPersistenceEnabled
 	})