[quality] flaky: TestLeaseRestart_ResumeSurvivesHubRestart TempDir cleanup races contribute-ws writes (ws-state: directory not empty)

[clubanderson]

What

test (hub N/3) shards on v5 intermittently fail with:

TempDir RemoveAll cleanup: .../ws-state: directory not empty

in TestLeaseRestart_ResumeSurvivesHubRestart: the contribute-ws hub is still writing lease/state files when t.TempDir() tears the directory down. Seen on runs 33945861132 (v5 branch) and 34037083450 before today's PRs existed, and again on #6214 on 2026-09-08, so it is independent of the change under test.

Ask

Make the test stop the hub (and wait for its writer goroutines to exit) before returning, e.g. a t.Cleanup that calls the hub's shutdown and blocks on its done channel, registered after the TempDir so it runs first. No time.Sleep (TestSleepRatchet is at its baseline). Add a -race -count=3 run of this test to the PR body as evidence.

Related: #6282 (same class of teardown race in pkg/agent).

🤖 Generated with Claude Code

https://claude.ai/code/session_01TmdVsn5zULh5KVpFkYrX59

[clubanderson]

This is the same failure as #6165, with the same mechanism: TestLeaseRestart_ResumeSurvivesHubRestart is the one pkg/dashboard test that keeps task-ledger persistence on while real websocket handlers run against httptest.Server. httptest.Server.Close does not wait for hijacked connections, so HandleWS's disconnect defer runs on the read goroutine after the test body has returned; that defer calls bookReleaseCooldown, which writes failed-tasks.json under ws-state/, and t.TempDir's RemoveAll races that write and reports ws-state: directory not empty. It is a test-side ordering problem, not a hub bug, and it is not the cleanupLoop goroutine leak (that is #6272 and cannot fire inside this test's 1.7s window).

The fix already landed on v4: the second commit of #6158 (e1e5125, "wait out websocket disconnect defers in TestLeaseRestart_ResumeSurvivesHubRestart") added waitContribDisconnect, which blocks on the disconnect defer's last observable act both before the restarted hub boots and, via t.Cleanup, before TempDir removal. #6158 merged to v4 as 3fa6693 on 2026-09-08 and #6165 was closed against it; the full trace of why the wait is sufficient is in the closing comment on #6165.

Why this ticket still saw the flake: all three cited observations (33945861132, 34037083450, and #6214) are v5 runs, and v5 does not have that commit yet. git grep waitContribDisconnect origin/v5 -- src/pkg/dashboard is empty, while origin/v4 has the four call sites. The last v4-into-v5 sync (#6053 / #6060) predates #6158, so v5 picks this up with the next routine sync rather than a separate fix. No new residual beyond #6165 and #6272 is described here, so re-fixing it on v4 would only produce a conflicting duplicate of e1e5125.

Closing as a duplicate of #6165. If it reappears on a v5 run whose base already contains e1e5125, reopen with the run id and the mechanism above needs another writer.

[clubanderson]

Duplicate of #6165 (fixed on v4 by 3fa6693; v5 needs the next v4-into-v5 sync).