Skip to content

fix(load): wait out a busy instance before cleanup gives up - #5115

Merged
shepilov merged 1 commit into
masterfrom
fix/load-retry-instance-busy
Oct 27, 2026
Merged

shepilov merged 1 commit into
masterfrom
fix/load-retry-instance-busy

Conversation

@shepilov

Copy link
Copy Markdown
Member

Part of #5107. On the VM capacity run, the ramp's cleanup and the next step's setup failed with 503 "the instance is busy, please retry later": after the ramp trashes its ~1,600 uploads, the Stack keeps the instance's lock for over a minute (abort after too many failures without getting the lock), and answers 503 with Retry-After: 10.

  • Node runners (fixture-http.ts): GET, HEAD and DELETE wait that out within the request's own timeout; a create is never retried, as it may have committed.
  • Runner cleanup deletes get the cleanup budget instead of 60 s.
  • k6 teardown deletes wait it out for up to 4 minutes, with a 5-minute teardown timeout.

Checked with make node-checks (651/651) and k6 against a fake busy server (503, 503, then 204).

After the ramp trashes a large directory, the Stack keeps the instance's lock
for over a minute and answers 503 with Retry-After: every cleanup delete then
failed at once, and the next step's setup with it. Reads and deletes now wait
that out within their timeout (creates are never retried), runner cleanup
deletes get the cleanup budget, and k6 teardown deletes up to four minutes.
@shepilov
shepilov requested a review from taratatach as a code owner October 27, 2026 17:03
@shepilov
shepilov removed the request for review from taratatach October 27, 2026 17:03
@shepilov
shepilov merged commit 82d8b3d into master Oct 27, 2026
4 checks passed
@shepilov
shepilov deleted the fix/load-retry-instance-busy branch October 27, 2026 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant