fix(workers): isolate each worker in its own git worktree#25
fix(workers): isolate each worker in its own git worktree#25shipyard-ci[bot] wants to merge 5 commits into
Conversation
…app#224) Concurrent builtin workers shared the agent's single checkout (runtime_config.workspace_dir), so one worker's git operations could see and clobber a sibling's in-progress edits — the spacedriveapp#211 stale-file sweep, the spacedriveapp#218/spacedriveapp#219 byte-identical blob, and the 2026-07-15 feature/1 stranding. Give every worker its own isolated workspace: - New src/agent/worker_workspace.rs: WorkerWorkspace::provision adds a cheap git worktree (shared object store, private working tree + index) per repo in the shared workspace, on a fresh per-worker branch off HEAD. release() removes them on success; reap_orphaned() bounds retained ones on startup (kept on failure for forensics). - Worker gains an optional isolated_workspace; its shell/file tools operate in that directory instead of the shared workspace when present, and it is released on successful completion. - channel_dispatch provisions and attaches an isolated workspace to every builtin worker (best-effort: falls back to shared workspace on failure, so a single worker sees no behavior change). - cortex startup reaps orphaned per-worker workspaces. - pr-gates skill: explicit-path git add + clean-status-before-commit as a second-layer mitigation. Tests (worker_workspace): a concurrent-dispatch repro proves two workers' git add -A commits cannot contain each other's edits and the shared branch is never mutated.
PR Review: fix(workers): isolate each worker in its own git worktreeSummaryThis PR isolates each agent worker into its own git worktree, preventing cross-worker interference in the shared workspace. It adds a new FindingsP1 (Blocking)None. (The "file truncation" noted in Part 2 is a review-artifact limitation, not a defect in the diff. Re-review the complete P2 (Should Fix)
P3 (Nit)
SecurityNo concerns identified in the reviewed diff. One item to confirm during full-file review: VerdictCOMMENT — No blocking defects in the diff. The P2 items (silent fallback, reaper bounding, provisioning completeness, path validation) should be addressed or explicitly confirmed before merge, but none block on their own. AI Review · Verdict: REQUEST_CHANGES · Diff-Score: 0.88 |
|
🔁 Regelkreis · round 4/2 · error 2 → 2.5 → 0 → 2 · confidence 0.22 → 0.22 → 0.61 → 0.20 |
|
Addressing review feedback. SpaceBot is working on fixes. |
|
🐕 Review-Loop-Watchdog · nudges 1/2 |
P2: - warn on silent fallback to shared workspace when isolated workspace has no worktrees (tool_workspace) [1] - document + harden orphan retention bound: symlink_metadata so planted symlinks aren't followed, clock-skew note; count bound holds regardless [2] - surface isolation-completeness: provision() logs PARTIAL vs complete and a new test asserts the partial-isolation path skips unisolable repos [3] - consistent UTF-8 validation: discover_repo_dirs skips non-UTF-8 repo names loudly instead of silently, so provision can rely on it [4] P3: - document WorkerId->branch git-validity (UUID Display is always a valid ref name) [5] - remove vestigial let _ = &self.shared_root binding (field dropped) [6] - attach_isolated_workspace error log gains Debug + shared_workspace context [7] - pr-gates SKILL.md gains a concrete BAD/GOOD git add anti-pattern example [8]
Review findings addressed (commit
|
Code Review: PR #25 — Worker Git Worktree Isolation1. SummaryThis PR isolates each worker in its own git worktree, provisioning per-worker workspaces with deterministic (UUID-based) branch names and cleaning them up on completion or via a startup reaper. It adds a new 2. FindingsP2 (Should Fix)
P3 (Nit)
3. Security
4. VerdictCOMMENT — No blocking (P1) issues. The isolation design is sound with best-effort degradation, but the error-handling gaps (partial provisioning state, orphan cleanup on abnormal exit, git error context) and the symlink-validation hardening are worth addressing before or shortly after merge. AI Review · Verdict: REQUEST_CHANGES · Diff-Score: 0.88 |
|
🚦 AI fix cycle stopped — confidence plateau. confidence stopped growing (0.22 → 0.22) despite another fix round. Error trajectory: 2 → 2.5 Findings needing a human decision:
|
✅ Test verification (follow-up)
|
… (fp:71617f0ad28f) tool_workspace() no longer silently degrades to the shared checkout when an isolated workspace was provisioned but has no worktrees. It now consults the workspace's Isolation completeness signal: the empty-because-nothing-to- isolate case falls back cleanly, but repos-present-yet-none-isolated trips a debug_assert! in debug builds and logs at error (loud, greppable) in release, documenting why the degradation is non-recoverable (re-opens issue spacedriveapp#224).
…tent (fp:3a69047329f9) Source repo paths and worktree paths were validated at different layers with no stated contract. Document the intentional asymmetry: git consumes the worktree path as a string arg (whole path must be UTF-8, enforced in add_worktree), while the source path is only passed via current_dir (an OsStr, no UTF-8 requirement — only its leaf name is validated, in discover_repo_dirs). Replace the provision-loop expect() on the (guaranteed) UTF-8 name with the same skip-and-log degradation used elsewhere, so a path-encoding regression can never panic a worker.
…al (fp:b51270ffe407) provision() returned a WorkerWorkspace with no way to tell whether isolation actually succeeded for every repo — per-repo failures are non-fatal, so Ok(_) did not imply full isolation. Add an Isolation enum (Fully / Partial / None) computed from isolated-vs-discovered repo counts, store it on the workspace, and expose it via WorkerWorkspace::isolation(). Document the completeness contract on provision() and in channel_dispatch::attach_isolated_workspace(). tool_workspace() (fp:71617f0ad28f) consumes this signal to refuse silent degradation. Covers the signal in the full/empty/partial provisioning tests.
|
Addressed P2 findings [1], [3], [4]. See commits above.
|
Code Review — spacebot#25SummaryThis PR isolates each agent worker in its own git worktree to prevent sibling workers from clobbering each other's edits in a shared workspace. It adds a FindingsP1 (Blocking)Concurrent branch race in provisioning ( P2 (Should Fix)
P3 (Nit)
SecurityNo injection, auth, or secrets issues introduced. Git worktree isolation is a sound approach and the fallback path is logged. Branch names/paths derived from repo state should be confirmed to be shell-safe, but no direct injection vector is visible in the diff. VerdictREQUEST_CHANGES — The concurrent branch race (P1) undermines the isolation guarantee this PR is meant to provide, and the silent fallback/reaping failures (P2) can mask exactly the corruption being fixed. Please also confirm ✅ Verification gate (closed-loop Regelkreis · spacedriveapp#58 Phase 1)An adversarial audit could not ground any blocking finding in the actual diff, so the blocking verdict was downgraded to APPROVE to avoid a false fix cycle (cf. PR spacedriveapp#57/spacedriveapp#48). AuditI'll audit each P1/P2 finding against the actual diff. P1: Concurrent branch race in provisioningThe review claims two workers provisioning for the same repo/branch can interleave delete→add. Let me check the branch naming: fn worker_branch_name(worker_id: WorkerId) -> String {
format!("spacebot/worker/{worker_id}")
}The branch name is already per-worker (namespaced by let branch = worker_branch_name(worker_id);
...
match add_worktree(&source, &worktree, &branch).await {The P1's premise ("same repo/branch", "deterministic per-worker branch name (e.g. include worker/session id)") is exactly what the code already does. The finding is factually wrong about the diff. PHANTOM — the recommended fix ("per-worker branch name including worker/session id") is already implemented in P2.1: Best-effort failures silently degrade isolation (
|
|
✅ PR approved — waiting for a human merge.
|
PR Review: fix(workers): isolate each worker in its own git worktreeSummaryThis PR isolates each agent worker in its own git worktree to prevent cross-worker contamination of the shared workspace. It introduces a new FindingsP2 (Should Fix)
P3 (Nit)
SecurityNo security concerns introduced. Isolation via VerdictCOMMENT — No blocking issues. The isolation approach is sound and well-documented. Recommend addressing the silent-fallback behavior (P2 #1) and confirming cleanup/release error handling (P2 #2) before or shortly after merge, but neither blocks approval given the startup reaper safety net. AI Review · Verdict: REQUEST_CHANGES · Diff-Score: 0.79 |
|
🚦 AI fix cycle stopped — budget. fix budget of 2 round(s) exhausted (error trajectory 2 → 2.5 → 0 → 2). Error trajectory: 2 → 2.5 → 0 → 2 Findings needing a human decision:
|
|
Addressed all 8 findings (P2: [1]–[4], P3: [5]–[8]). See commits above. P2 (blockers):
P3 (refinements):
|
Closes marcmantei/shipyard#224
Problem
Every builtin worker ran its shell and file tools in the
mainagent's singleshared checkout (
runtime_config.workspace_dir=/data/agents/main/workspace).Concurrent workers therefore shared one working tree and index, so one worker's
git add/commitcould see and clobber a sibling's in-progress edits. Confirmedincidents (shipyard#224):
git add -Ain the shared tree).developer_followup_flow.pyblob —one worker's commit swept up the other's edits.
feature/1in the shared checkout instead of thePR branch, then stranded.
worker_runs.worktree_idwas NULL for all 200+ rows —the isolation path the issue asked for was entirely unused.
Fix — one isolated
git worktreeper workerNew
src/agent/worker_workspace.rs:WorkerWorkspace::provision(shared_root, worker_id)adds a cheapgit worktree(shared object store, private working tree + index) for each repo in the
shared workspace, on a fresh per-worker branch off the repo's current
HEAD.release()removes the worktrees on success; on failure they're retained forforensics and
reap_orphaned()bounds retention on startup.workspace, so behaviour is unchanged when only one worker runs.
Wiring:
Workergains an optionalisolated_workspace; its shell/file tools operate inthat directory instead of the shared
workspace_dirwhen present, and it isreleased on successful completion.
channel_dispatchprovisions + attaches an isolated workspace to every builtinworker (best-effort: provisioning failure falls back to the shared workspace and
logs, never blocks dispatch).
pr-gatesskill gains explicit-pathgit add+ clean-status-before-commit rulesas a second-layer mitigation.
Verification
cargo test --lib worker_workspace(4 tests, all pass). The key acceptance testconcurrent_workers_cannot_contaminate_each_otherprovisions two workers on thesame repo, has worker B run the exact
git add -Asweep that caused spacedriveapp#218/spacedriveapp#219, andasserts:
main, neither file present).cargo check,cargo clippy --lib(no new warnings), andcargo fmt --checkare clean.Notes
marcmantei/shipyard; the durable fix is spacebot-side (per theroot-cause trace in that thread), so this PR lands here and closes the cross-repo issue.