Windows: zombie broker + codex app-server trees accumulate — broker/shutdown acks before unbounded cleanup, and taskkill tree-kill breaks under Git Bash SHELL

[JethroTseng]

Summary

On Windows (plugin v1.0.5), zombie broker + codex app-server process trees accumulate at scale — we found 16 brokers, 16 codex app-server children, and 16 shell-wrapper processes (48 processes total) that had survived for 2 days. Every broker's named pipe was still alive and accepting connections. The visible symptom is the recurring startup/exit message:

SessionEnd hook [node "${CLAUDE_PLUGIN_ROOT}/scripts/session-lifecycle-hook.mjs" SessionEnd] failed: Hook cancelled

This is a compound failure chain. Individual links are partially covered by #288, #331, #403, #380, #410, but the ack-before-cleanup shutdown handler and the resulting child-tree leak are not reported anywhere, and it's the combination that makes brokers effectively immortal.

Environment

• Windows 11 Pro (10.0.26200), plugin v1.0.5, Node from C:\Program Files\nodejs
• Brokers spawned from Claude Code Bash-tool background jobs, so their environment has SHELL = Git Bash (sh.exe)

Observed evidence (verified)

1. 16 broker processes (app-server-broker.mjs serve) created 2026-06-30 → 07-01 were still running on 07-02. Each had a matching codex app-server child — wrapped in powershell.exe -c "codex app-server" or sh.exe .../npm/codex app-server (a consequence of shell: process.env.SHELL || true in app-server.mjs:190-194).
2. All 16 named pipes still connected successfully (tested via NamedPipeClientStream).
3. Sending broker/shutdown manually to one zombie broker returned {"id":1,"result":{}} in ~250 ms, but the process was still alive 2+ seconds later (it exited some minutes afterwards).
4. The SessionEnd hook, run manually against this accumulated state, took long enough to exceed the hook's own timeout: 5 (hooks.json) — Node cold start alone costs ~1–2 s on Windows, leaving almost no budget for shutdown + teardown work. Hence the recurring Hook cancelled.
5. Because the hook is cancelled before teardown completes, stale broker.json and cxc-* session dirs persist, and the next SessionEnd repeats the same work — the failure is self-perpetuating.

Root-cause chain (code analysis, v1.0.5)

Broker side — ack-before-cleanup with unbounded await (app-server-broker.mjs:160-163):

if (message.id !== undefined && message.method === "broker/shutdown") {
  send(socket, { id: message.id, result: {} });   // 1. ack FIRST
  await shutdown(server);                          // 2. then cleanup — unbounded
  process.exit(0);
}

shutdown() awaits appClient.close(), which (app-server.mjs:232-265):

• ends the child's stdin,
• after 50 ms calls terminateProcessTree(this.proc.pid) and swallows all errors,
• then does await this.exitPromise with no timeout.

The tree-kill silently fails under Git Bash SHELL (lib/process.mjs):

shell: process.platform === "win32" ? (process.env.SHELL || true) : false,

When SHELL points at Git Bash, taskkill /PID <pid> /T /F runs as bash -c "taskkill /PID ..." and MSYS converts /PID, /T, /F into paths (C:/Program Files/Git/PID, …) — taskkill errors out (same mechanism as #331, which reports it for cancel). The error is swallowed, the shell-wrapped codex app-server child never dies, exitPromise never resolves, and the broker sits forever after having acked the shutdown.

Hook side (session-lifecycle-hook.mjs → broker-lifecycle.mjs):

• sendBrokerShutdown resolves on the ack (first data event), so the hook believes shutdown succeeded.
• Its backstop teardownBrokerSession → terminateProcessTree(broker.pid) is the same broken taskkill, also swallowed.
• Net result: broker + wrapper shell + codex app-server all survive every session end. Multiply by every background review/task and you get the 48-process pile above.

Suggested fixes

1. process.mjs / app-server.mjs: never use process.env.SHELL as the spawn shell on win32 — use shell: true (cmd.exe) or, better, spawn taskkill directly with shell: false (it's an absolute-path-resolvable exe; args as array need no shell). This single change fixes both kill paths and codex-companion cancel taskkill fallback breaks under Git Bash — MSYS translates /PID arg to C:/Program Files/Git/PID #331.
2. app-server-broker.mjs: perform cleanup before acking broker/shutdown (or ack, then process.exit(0) after a bounded cleanup — e.g. Promise.race([shutdown(server), sleep(2000)])).
3. app-server.mjs close(): bound await this.exitPromise (e.g. 2 s), then hard-kill.
4. hooks/hooks.json: timeout: 5 for SessionEnd leaves ~3 s of real budget after Node cold start on Windows; consider 15–30 s.

Related

• sendBrokerShutdown has no timeout — SessionEnd hook can hang indefinitely #288 — sendBrokerShutdown has no timeout (hook-side link of the same chain)
• codex-companion cancel taskkill fallback breaks under Git Bash — MSYS translates /PID arg to C:/Program Files/Git/PID #331 — MSYS mangles taskkill /PID under Git Bash (the enabler; reported there for cancel)
• SessionEnd hook can hit 5s timeout on Windows due to shelling out to git #403 — SessionEnd 5 s timeout too tight on Windows
• SessionEnd cleanup fails when review cwd ≠ session cwd — broker.json looked up by cwd-hash, leaving orphan brokers even on graceful /quit #380 — different orphan-broker path (cwd-hash mismatch)
• Background job state machine retains zombie entries; --fresh does not clear them #410 — zombie job state entries

[JethroTseng]

Follow-up: profiled the actual failure mechanism — it's hook slowness vs Claude Code's exit grace window, not (only) a hang

We instrumented session-lifecycle-hook.mjs with per-step timestamps and reproduced with headless claude -p full session lifecycles. Two findings that change the picture:

1. The SessionEnd hook gets killed even when it finishes in ~1.3s

The hook completed all cleanup work at +1262ms (last log line SessionEnd complete written) and Claude Code still reported Hook cancelled. In another run it was killed mid-cleanup at +1528ms (log ends after cleanupSessionJobs, before broker teardown). The "timeout": 5 in hooks/hooks.json is not the effective limit — Claude Code's exit path appears to grant SessionEnd hooks only ~1–1.5s before cancelling/killing them.

This is the self-perpetuation mechanism: the kill frequently lands between sendBrokerShutdown and terminateProcessTree/clearBrokerSession, stranding live broker trees and stale broker.json files, which then make the next SessionEnd slower/hang → cancelled again.

2. Where the ~1.5s goes: repeated git rev-parse spawns through a shell

resolveWorkspaceRoot() → ensureGitRepository() → runCommand("git", ["rev-parse", "--show-toplevel"]) with shell: process.env.SHELL || true — i.e. every call spawns a Git Bash shell + git (~250ms each on Windows). The SessionEnd path resolves the same cwd ~6 times (loadBrokerSession → resolveStateDir, cleanupSessionJobs → resolveWorkspaceRoot + resolveStateFile + loadState + saveState, clearBrokerSession):

	main() work	E2E claude -p runs
stock v1.0.5	1262ms	1/3 Hook cancelled
with fixes below	387ms (1 git spawn)	0/4 Hook cancelled

Local fixes verified (happy to PR)

1. Memoize resolveWorkspaceRoot per process (lib/workspace.mjs): Map cache keyed by cwd, pre-seeding the resolved root → root mapping. Cuts ~6 shell+git spawns to 1.
2. Timeout guard in sendBrokerShutdown (lib/broker-lifecycle.mjs): the promise only resolves on data/error/close; a zombie broker whose event loop is stuck accepts the pipe connection but never replies, so the await blocks forever. A 2s setTimeout + socket.destroy() guard bounds it (verified with a fake pipe server that accepts and never replies: pre-fix = infinite hang, post-fix = resolves at ~2000ms).

Also worth noting: in claude -p mode the SessionEnd hook is sometimes never spawned at all (no log lines, no cancel message) — fire-and-forget race in the host CLI. Harmless when state is clean, but it means a live broker can be orphaned with zero hook-side recourse; a startup-time sweep of stale broker.json entries (pid liveness check) would close that gap.

[axisrow]

This issue conflates two distinct classes: (1) the orphan half — detached broker/app-server trees left by terminated sessions = scenario 1 in the #163 map, covered by PR #490 (idle-timeout); (2) the taskkill/Windows half — taskkill /T /F breaking under Git Bash SHELL = a separate shell-mangling class (same root as #478 part 3). Worth tracking them separately since they have different fixes. See #163.