fix(tmux): stabilize server cwd and tolerate async cd on pane cwd verify

[harshitsinghbhandari]

Summary

Consolidated fix for the tmux poisoned-cwd spawn failure (issue #2775 family): a running tmux server can have its cwd pinned inside a Squirrel/ShipIt auto-update staging directory that the update then deletes, and pane cwd verification could fail a spawn that was actually recovering fine.

Evidence, measured live on a real poisoned server (2026-07-25):

• tmux new-session -c <a directory that definitely exists> was silently ignored; the pane landed in the dead ShipIt directory.
• A fresh tmux server on a separate socket honored -c correctly, confirming the poison lives in the server's own cwd, not tmux's -c handling in general.
• buildLaunchCommand's cd '<workspace>' || exit; guard DOES correct the pane, but only once the shell actually runs it: #{pane_current_path} sampled at t+0s (immediately after new-session) was stale, and the same probe sampled at t+0.05s was already correct.
• verifyPaneWorkingDirectory (added by fix: stabilize daemon spawn working directories #2871) sampled #{pane_current_path} exactly once, immediately after new-session, so it lost that race 100% of the time on a poisoned server and every spawn failed with an opaque HTTP 500 with no message.

The poisoned server alone is harmless: the cd guard recovers the pane every time. The single-shot verification is what converted a working, self-correcting spawn into a hard failure.

Changes

1. Pin the tmux CLI cwd (backend/internal/adapters/runtime/tmux/tmux.go, execRunner.Run): sets cmd.Dir = os.TempDir() before running. The first tmux CLI call auto-starts the persistent tmux server, which inherits this process's cwd for its entire lifetime; os.TempDir() outlives app bundle swaps and ShipIt staging dirs, so the server can never be pinned to a path an update will delete.

2. Retry pane-cwd verification (same file, verifyPaneWorkingDirectory): retries up to 5 times, 50ms apart, with a select on ctx.Done() so cancellation is honored, returning the last error if every attempt fails. Extends the existing function and reuses the existing sameDirectory helper (already resolves symlinks and handles empty strings) rather than introducing a parallel implementation. The final error names both the observed pane path and the wanted workspace, and says the worktree may be missing or the tmux server may be pinned to a stale directory. The cwd-mismatch error is sticky across retries: a later transient probe failure (a one-off tmux CLI error, not a mismatch) never overwrites an already-observed mismatch, so the typed sentinel from Fix 4 always survives to the caller.

3. Recreate missing registered worktrees (backend/internal/adapters/workspace/gitworktree/workspace.go): registeredWorktreeDirMissing detects a git worktree registration whose directory no longer exists on disk and lets the caller materialize a fresh worktree at the same path. Applied to both the Create-reuse path (existingWorktree, matching fix(workspace): recreate missing registered worktrees #2776's approach) and the Restore path, since both had the identical bug shape: if rec, ok := findWorktree(records, path); ok { return ... } without ever checking the directory actually exists. The real observed case (session agent-orchestrator-78) still had its branches and worktree registration plus its DB row, but no worktree directory, so cd '<workspace>' || exit exited instantly and restore failed silently, which is exactly the Restore code path. On the recreate path, Restore checks out the registration's own branch (not cfg.Branch): AO workers routinely sit on a child branch (ao/<id>/<feature>), not the root branch AO would otherwise pass through cfg.Branch, and recreating on the wrong branch would silently strand the session's work.

   • Nothing is removed or pruned to recover. Recovery is a single git worktree add --force <path> <branch>, git's own documented override for "<path> is a missing but already registered worktree" (that failure's own hint is "use 'add -f' to override"), so registeredWorktreeDirMissing is a pure observation (stat plus the Locked check) with no mutation. Measured against real git 2.54: --force re-registers a path whose registration outlived its directory and leaves every sibling registration untouched; on a path that exists and is non-empty it refuses outright (fatal: '<path>' already exists) even with -f -f, leaving the existing files intact; and a single --force still refuses a missing-but-locked registration. It is never passed twice.
   • This is what makes the recovery non-destructive under concurrency. The two alternatives both have a failure mode: the repo-wide git worktree prune removes sibling registrations (verified against real git: with two worktrees on distinct branches both missing their directories, one prune removed BOTH, so restoring one session could silently wipe a sibling's registration and reset it to the wrong branch on its own next restore), and the target-specific git worktree remove --force is check-then-delete against the earlier os.Stat, so with two concurrent restores of the same session (RestoreWithMode has no per-session guard) the second one's stale decision deletes the live worktree the first just recreated, uncommitted agent work included. add --force has neither problem: it touches only this path's registration, and a restore that loses the race fails loudly instead of destroying work.
   • The same swap replaces the two pre-existing worktree add stale-registration fallbacks (in addWorktree and createWorkspaceProjectRepo) that recovered via the repo-wide git worktree prune: they now retry with --force, so the sibling-registration problem cannot recur through those paths either.
   • A registration whose directory is missing AND is locked (git worktree lock) is not treated as recoverable. Verified against real git: git worktree prune leaves a locked-but-missing registration in place, and git worktree add (with or without a single --force) and git worktree remove --force at that path all fail with an opaque "missing but locked worktree" git error. registeredWorktreeDirMissing checks the registration's already-parsed Locked attribute first and returns the new typed ports.ErrWorkspaceLocked (adapter-local alias ErrWorktreeLocked, following the existing ports.ErrWorkspace* pattern) instead of attempting recovery. Locking is an explicit operator signal not to touch a worktree, so -f -f is deliberately never used.

4. Stop masking this as a bare 500 (backend/internal/service/session/service.go, toAPIError): adds sentinels ports.ErrRuntimeWorkspaceCwdMismatch and ports.ErrWorkspaceLocked (following the existing ports.ErrWorkspace* pattern) and maps them to typed WORKSPACE_CWD_MISMATCH / WORKSPACE_LOCKED conflict apierrs carrying the real message, instead of falling through to an opaque 500 INTERNAL_ERROR with no message.

Prior art, reuse and credit

• fix(tmux): run tmux CLI from stable cwd and verify pane starts in workspace #2984 (this contributor) contained the cmd.Dir change and useful tests; its verification half is superseded by merged fix: stabilize daemon spawn working directories #2871, so that half was dropped here. Closing in favor of this PR.
• fix(workspace): recreate missing registered worktrees #2776 (this contributor) contained the worktree recreation approach for the Create path, ported here and extended to Restore. Closing in favor of this PR.
• fix(tmux): stable server cwd and pane cwd verify retries #3027 (AllBeingsFuture) independently arrived at the same cmd.Dir pin plus a retry loop. Credit due; this PR supersedes it with the additional worktree-recreation and typed-error fixes.
• Related: fix: stabilize daemon spawn working directories #2871 (merged, added the original single-shot pane cwd verification this PR extends).
• Reviewer illegalcall found three real bugs in Fix 3 across two rounds: repo-wide prune silently wiping sibling registrations, locked-but-missing worktrees being misreported as recovered, and the os.Stat-then-worktree remove --force check-then-delete destroying a worktree recreated by a concurrent restore. All three are fixed above and each has a real-git regression test.

Fixes #2775. Related: #2871, #2984, #2776, #3027.

Tests

New/updated, all passing:

• TestExecRunnerRunsFromStableDir: direct test of execRunner.Run (the real one, not the fakeRunner test seam every other tmux test in the package uses) against a real exec.Cmd, asserting its cwd resolves to os.TempDir(). This is the only test that runs the actual cmd.Dir line and would catch a regression there.
• TestVerifyPaneWorkingDirectoryRetriesUntilMatch: pins the retry behavior, verification succeeds when the first sample is stale and a later sample matches.
• TestVerifyPaneWorkingDirectoryHonorsCancellation: the retry loop's ctx.Done() select actually aborts a pending retry.
• TestVerifyPaneWorkingDirectoryKeepsMismatchErrorAfterLaterProbeFailure: a mismatch on an earlier attempt survives a probe failure on a later attempt, so the caller still gets the typed sentinel.
• TestCreateDestroysAndReturnsErrorWhenPaneCWDDoesNotMatch: updated for the retry sequence (5 stale samples), asserts the wrapped sentinel and exactly paneCwdVerifyAttempts verification calls.
• TestCreateRecreatesMissingRegisteredWorktreeWithForce / TestRestoreRecreatesMissingRegisteredWorktreeWithForce: stale-registration recovery on the Create and Restore paths (fake git), asserting the worktree add --force call and, via a shared assertNoDestructiveRegistrationCleanup helper, that neither worktree prune nor worktree remove is used to recover.
• TestCreateWorkspaceProjectRepoRetriesStaleRegisteredWorktreeWithForce: the workspace-project fallback retries with --force instead of the repo-wide prune it used to run.
• TestCommandArgs: new add existing forced / add new forced cases pin that --force lands in the right position and is passed exactly once.
• TestRestoreRecreatesOnRegisteredBranchNotCfgBranch: regression test for the branch bug in Fix 3, a stale registration recorded on a non-root branch is recreated on THAT branch, not on cfg.Branch.
• TestWorkspaceIntegrationRestoreRecreatesSiblingsIndependently (real git): two sessions in the same repo, both worktree directories deleted out of band, restored sequentially; each is recreated on its own recorded branch and the second's registration survives the first's restore. Confirmed to fail against the pre-fix repo-wide-prune code before landing this fix.
• TestWorkspaceIntegrationRestoreLockedMissingWorktreeIsTypedError (real git): a locked worktree whose directory is deleted; asserts ports.ErrWorkspaceLocked and that git worktree add was never attempted. Confirmed to fail against the pre-fix code (surfaces the raw git "missing but locked worktree" error instead) before landing this fix.
• TestWorkspaceIntegrationRestoreDoesNotDestroyWorktreeRecreatedMidRecovery (real git): the concurrent-restore finding, forced deterministically instead of raced. The hooked runner recreates the worktree at the same path and writes an uncommitted sentinel file immediately before the first registration-mutating git command Restore issues, which is exactly the window between the stat and the cleanup; the test asserts the sentinel survives and that a restore that lost the race never reports success for a worktree it did not materialize. Confirmed to fail against the previous commit (uncommitted work in the concurrently recreated worktree did not survive stale-registration recovery) before landing this fix.
• TestToAPIErrorMapsWorkspaceBranchSentinels: new cases for WORKSPACE_CWD_MISMATCH and WORKSPACE_LOCKED.

Verification run from backend/:

• go build ./...: clean

• go vet ./...: clean

• gofmt -l: clean on all touched files

• go test -race ./internal/adapters/runtime/tmux/... ./internal/adapters/workspace/gitworktree/... ./internal/service/session/... ./internal/session_manager/... ./internal/ports/...: all packages ok. Note: running these five packages together under -race with default package parallelism can intermittently flake one pre-existing, untouched real-tmux integration test (TestRuntimeIntegrationSupervisedExitKeepsInteractiveShell, fixed 5s wall-clock deadlines under race-instrumented CPU contention from the other packages running concurrently); it passes reliably alone, in the full non-race suite, and with -race -p 1 (sequential package execution) across all five packages.

• go test -count=1 ./...: all 99 packages ok.

• golangci-lint run ./internal/adapters/workspace/gitworktree/...: 0 issues.