Use a soft file lock for the cross-process creation limiter

[samsja]

Motivation

CreationLimiter (verifiers/v1/runtimes/limiters.py) hand-rolls cross-process mutual exclusion with fcntl.flock on the shared bucket file. flock may not provide reliable cross-host exclusion on shared/NFS filesystems. If two writers enter together, the cursor can become corrupt and every later sandbox creation fails.

Changes

• Replace hand-written fcntl.flock calls with SoftFileLock unconditionally. There is no mode switch or environment variable: every process coordinates through the same NFS-safe atomic marker.

• Keep bucket state and mutual exclusion in separate paths:

  • {name}.bucket
  • {name}.bucket.lock

  SoftFileLock removes its marker on release, so it cannot share the cursor file.

• Require filelock>=3.28, the first published release with PID/hostname ownership and same-host stale-marker recovery for SoftFileLock.

• Cap acquisition at 60 seconds rather than hanging indefinitely. A marker left by a failed process on another host cannot be proven stale from the current host, so this fails closed after 60 seconds instead of silently breaking mutual exclusion.

• Recover a malformed bucket cursor by resetting it instead of failing every subsequent creation.

The leaky-bucket behavior is otherwise unchanged: the cursor advances while holding the exclusive lock, waiting happens after releasing it, a backlog above five minutes raises, and cursor timestamps remain comparable wall-clock values.

---

Note

Medium Risk
Changes mutual exclusion for shared rate limiting used by Modal and Prime tunnel creation; incorrect locking could still allow cursor corruption or spurious timeouts under NFS or crashed workers.

Overview
Replaces fcntl.flock on the shared creation-rate limiter bucket with filelock.SoftFileLock so cross-process coordination works on local and shared/NFS filesystems (used by Modal sandbox and Prime tunnel creation).

Bucket cursor state stays in {name}.bucket while the lock marker uses a separate {name}.bucket.lock path, because SoftFileLock deletes its marker on release. Adds filelock>=3.28 for stale-marker recovery, a 60s lock acquisition timeout instead of indefinite blocking, and resets corrupt bucket cursor values instead of failing permanently.

Leaky-bucket semantics are unchanged: reserve under lock, sleep after release, and still fail when backlog exceeds five minutes.

^{Reviewed by Cursor Bugbot for commit fe6843a. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Replace fcntl.flock with SoftFileLock in CreationLimiter._reserve

• Switches cross-process locking in limiters.py from fcntl.flock on the bucket file to a sidecar SoftFileLock with a .lock suffix, so releasing the lock marker does not delete the cursor state file.
• Lock acquisition now waits up to 60 seconds instead of blocking indefinitely; failure raises the lock acquisition error.
• A corrupt or non-numeric bucket cursor is reset to zero instead of propagating ValueError. The five-minute backlog check is unchanged.
• Adds filelock>=3.28 as a project dependency to support stale SoftFileLock marker recovery.
• Risk: any environment with an older filelock or a shared filesystem lacking comparable wall clocks may see lock acquisition failures or stale marker recovery issues.

^{Macroscope summarized fe6843a.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 1d345da. Configure here.}

[samsja]

Addressed in e0d3f3b: native and soft locks now use distinct paths, filelock is pinned to the first published version with PID/hostname stale-marker recovery, and regression tests cover mode switching plus a process exiting while holding the soft lock.