Skip to content

Commit c1da123

Browse files
authored
fix(lifecycle): recover owned mux sessions
## Summary - reconcile deterministic lifecycle mux sessions with token-backed ownership, adopting only healthy owned sessions and safely replacing stale owned sessions - include a deterministic SHA-256 fingerprint of the effective overlay and sorted lifecycle env in freshness, so changed env replaces stale host helpers without persisting names, keys, or values - preserve running Compose services, lifecycle helpers, and ownership state when a down.before guard aborts hack down or hack restart - target detached Zellij sessions through the documented global --session selector for owner metadata, inspection, pane creation, exec, and input - teach doctor to detect and safely repair established ownership-proven orphan sessions while preserving foreign collisions and recent in-flight startups - make the local tmux recovery scenario capability-aware while requiring tmux in the dedicated Docker/tmux CI job ## Verification - 850 unit tests passed, 5 intentional skips, 0 failed - typecheck, privacy check, canonical Turbo quality checks, changed-file Ultracite, build, generated agent parity, and CLI reference generation passed - full local E2E with tmux mandatory: 9 local scenarios passed; 3 disabled Docker-tier scenarios skipped - lifecycle recovery E2E proves env refresh, failed down/restart guard preservation, ownership safety, signal cleanup, and doctor repair - compiled binary passed the lifecycle session recovery E2E scenario - no-tmux recovery E2E exits 0 with an explicit local skip and exits 1 with zero skips when HACK_E2E_REQUIRE_TMUX=1 - live Zellij 0.43.1 detached-session probe wrote and read owner metadata through --session and cleaned up successfully - release tarball extracted and its binary passed version smoke - current-head CI passed secret scan, test/release smoke, runtime images, and non-skippable Docker/tmux E2E - local Docker 29.4.0 responds to info but hangs on container creation; Docker scenarios are proven by current-head CI ## Release Release signal: yes. This is a user-visible lifecycle reliability fix and should produce a patch release through semantic-release. No separate changeset is used by this repository release flow.
1 parent f482776 commit c1da123

31 files changed

Lines changed: 3129 additions & 348 deletions

.codex/skills/hack-cli/SKILL.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,8 @@ Use `hack` as the primary interface for local-first development.
113113
- For fixed-port host helpers such as SSM tunnels or local proxies, set `singleton.ports` and usually `onConflict: "adopt"` so Hack reuses a healthy existing listener instead of starting duplicate tunnel stacks.
114114
- `singleton` is a listener guard, not process ownership transfer; adopted external processes are left running on `hack down`.
115115
- Inspect lifecycle status via `hack projects --details` and stream via `hack logs <service-or-process>`.
116+
- Lifecycle session recovery is ownership-proven: Hack adopts healthy token-, definition-, and environment-matched sessions, replaces owned stale sessions, and refuses to kill same-name sessions without deterministic ownership proof.
117+
- `hack doctor --fix` reaps an orphan lifecycle session only when mux ownership is proven and its Compose instance is absent; unverified same-name sessions are never modified.
116118

117119
## Workspaces (mux-managed, tmux-first by default)
118120

.cursor/rules/hack.mdc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,8 @@ Prefer `hack` when shell access is available. Use MCP only when shell access is
6161
- For fixed-port host helpers such as SSM tunnels or local proxies, set `singleton.ports` and usually `onConflict: "adopt"` so Hack reuses a healthy existing listener instead of starting duplicate tunnel stacks.
6262
- `singleton` is a listener guard, not process ownership transfer; adopted external processes are left running on `hack down`.
6363
- Inspect lifecycle status via `hack projects --details` and stream via `hack logs <service-or-process>`.
64+
- Lifecycle session recovery is ownership-proven: Hack adopts healthy token-, definition-, and environment-matched sessions, replaces owned stale sessions, and refuses to kill same-name sessions without deterministic ownership proof.
65+
- `hack doctor --fix` reaps an orphan lifecycle session only when mux ownership is proven and its Compose instance is absent; unverified same-name sessions are never modified.
6466

6567
## Host-side env helpers
6668

.github/workflows/ci.yml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,26 @@ jobs:
5656
- name: Smoke slim runtime mounted-project env flow
5757
run: bash scripts/portable-container-smoke.sh hack-runtime-ci:slim linux/amd64
5858

59+
docker-e2e:
60+
runs-on: blacksmith-4vcpu-ubuntu-2404
61+
permissions:
62+
contents: read
63+
steps:
64+
- name: Checkout
65+
uses: actions/checkout@v4
66+
- name: Setup Bun
67+
uses: oven-sh/setup-bun@v1
68+
with:
69+
bun-version: "1.3.9"
70+
- name: Install tmux
71+
run: sudo apt-get update && sudo apt-get install --yes tmux
72+
- name: Install dependencies
73+
run: bun install
74+
- name: Create isolated Hack network
75+
run: docker network create --subnet 172.30.0.0/16 hack-dev
76+
- name: Run local and Docker E2E
77+
run: HACK_E2E_REQUIRE_DOCKER=1 HACK_E2E_REQUIRE_TMUX=1 bun run test:e2e:local:docker
78+
5979
test:
6080
runs-on: blacksmith-6vcpu-macos-15
6181
steps:

AGENTS.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -299,6 +299,8 @@ Lifecycle + startup:
299299
- For fixed-port host helpers such as SSM tunnels or local proxies, set `singleton.ports` and usually `onConflict: "adopt"` so Hack reuses a healthy existing listener instead of starting duplicate tunnel stacks.
300300
- `singleton` is a listener guard, not process ownership transfer; adopted external processes are left running on `hack down`.
301301
- Inspect lifecycle status via `hack projects --details` and stream via `hack logs <service-or-process>`.
302+
- Lifecycle session recovery is ownership-proven: Hack adopts healthy token-, definition-, and environment-matched sessions, replaces owned stale sessions, and refuses to kill same-name sessions without deterministic ownership proof.
303+
- `hack doctor --fix` reaps an orphan lifecycle session only when mux ownership is proven and its Compose instance is absent; unverified same-name sessions are never modified.
302304

303305
Workspaces (mux-managed, tmux-first by default):
304306
- Picker: `hack session` for persistent project workspaces.

CLAUDE.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,8 @@ Lifecycle + startup:
169169
- For fixed-port host helpers such as SSM tunnels or local proxies, set `singleton.ports` and usually `onConflict: "adopt"` so Hack reuses a healthy existing listener instead of starting duplicate tunnel stacks.
170170
- `singleton` is a listener guard, not process ownership transfer; adopted external processes are left running on `hack down`.
171171
- Inspect lifecycle status via `hack projects --details` and stream via `hack logs <service-or-process>`.
172+
- Lifecycle session recovery is ownership-proven: Hack adopts healthy token-, definition-, and environment-matched sessions, replaces owned stale sessions, and refuses to kill same-name sessions without deterministic ownership proof.
173+
- `hack doctor --fix` reaps an orphan lifecycle session only when mux ownership is proven and its Compose instance is absent; unverified same-name sessions are never modified.
172174

173175
Workspaces (mux-managed, tmux-first by default):
174176
- Picker: `hack session` for persistent project workspaces.

docs/architecture.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,13 +125,20 @@ stable home and can be torn down on `hack down`.
125125
Lifecycle state is checkout-local at `.hack/.internal/lifecycle/state.json`. For tmux processes, the
126126
saved process group belongs to the wrapped user command rather than its mux pane shell. Each compose
127127
instance has its own entry, so starting a branch instance does not replace the base instance's entry
128-
in the same checkout. Cleanup reconciles the saved process group with the live process table; if the group
128+
in the same checkout. A random token mirrored in mux metadata and lifecycle state proves current
129+
session ownership; deterministic session names never authorize teardown. A versioned SHA-256 digest
130+
of the effective overlay and sorted lifecycle environment participates in session freshness without
131+
persisting environment names, keys, or values. Cleanup reconciles the saved process group with the live process table; if the group
129132
leader exited while members remain, `hack doctor` reports the orphan. `hack down` terminates the
130133
persisted leaderless group and descendant process groups only while a matching lifecycle mux session
131134
still proves ownership; without that session, cleanup stays non-destructive because the numeric PGID
132135
may have been reused. A live group leader without its saved pane PID is likewise not trusted as
133136
lifecycle ownership.
134137

138+
Doctor also compares ownership-proven lifecycle sessions with live Compose instances. `doctor --fix`
139+
can remove a proven session only when its corresponding instance is absent and a second liveness and
140+
ownership check still agrees. Foreign or ambiguous same-name sessions remain non-destructive findings.
141+
135142
For fixed-port helpers such as SSM/database/search tunnels, lifecycle config can also declare a
136143
`singleton` listener set. This lets Hack reuse an already-running equivalent helper or fail fast on
137144
partial conflicts instead of launching a competing duplicate supervisor. The intent is to reduce local

docs/lifecycle.md

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -179,16 +179,35 @@ Lifecycle output is now surfaced across CLI/runtime views:
179179
5. Run `docker compose up` (or `up -d` when `--detach`).
180180
6. Run `lifecycle.up.after` hooks.
181181

182+
Before starting persistent work, Hack reconciles the expected lifecycle session with
183+
`.hack/.internal/lifecycle/state.json`:
184+
185+
- A session whose mux ownership token, saved definition hash, and live process windows all match is
186+
adopted without starting duplicate processes.
187+
- A token-owned stale session is replaced. Pre-ownership tmux sessions are replaced only when saved
188+
state, checkout path, creation time, and window count provide deterministic legacy ownership proof.
189+
- A same-name session without matching ownership proof is left untouched and `hack up` fails with an
190+
actionable collision error.
191+
192+
If the operation later fails in Compose or `up.after`, Hack removes only the lifecycle session created
193+
by that operation. The same exact-token cleanup runs for supported `SIGINT` and `SIGTERM` paths, including
194+
signals received while the mux session is still being initialized.
195+
182196
### `hack down`
183197

184198
1. Run `lifecycle.down.before` hooks.
185199
2. Run `docker compose down`.
186200
3. Stop lifecycle processes by killing the lifecycle session.
187201
4. Run `lifecycle.down.after` hooks.
188202

203+
If `lifecycle.down.before` fails, shutdown is aborted before Compose or lifecycle processes are
204+
stopped. `hack restart` preserves the same guard semantics during its down phase.
205+
189206
### `hack restart`
190207

191208
`hack restart` performs the same lifecycle steps as `hack down` followed by `hack up`.
209+
From the primary checkout, it targets only the base Compose/lifecycle instance. A linked worktree uses
210+
its isolated derived branch instance, and `--branch <name>` targets only that explicit branch.
192211

193212
### `--json`
194213

@@ -212,10 +231,19 @@ Lifecycle session name:
212231
Notes:
213232
- If no mux backend is available, lifecycle process startup fails with an actionable error.
214233
- Teardown is implemented by killing the lifecycle session; anything running inside that session will be stopped.
234+
- Current lifecycle sessions carry the same random ownership token in mux metadata and persisted state.
235+
Cleanup requires an exact token match; deterministic names alone never authorize session teardown.
236+
- Healthy-session adoption also matches a stable SHA-256 fingerprint of the effective overlay and
237+
sorted lifecycle environment. Environment names, keys, and values are never persisted; changing an
238+
overlay or value replaces the owned session so host helpers cannot retain stale configuration.
215239
- For tmux-backed lifecycle sessions, Hack also persists the pane PID and the wrapped command's actual
216240
process-group metadata to `.hack/.internal/lifecycle/state.json`, with per-hook/process output logged to
217241
`.hack/.internal/lifecycle/*.log`. If tmux pane state disappears before teardown, `hack down` still uses that persisted metadata to clean up any live lifecycle process groups instead of leaving orphaned host processes behind.
218-
- `hack doctor` reports stale lifecycle state when the persisted lifecycle entry no longer has a live mux session and points operators to `hack down` so cleanup and state removal happen through the supported path. `hack doctor --fix` does not tear down lifecycle sessions itself — use `hack down` for that.
242+
- `hack doctor` reports stale lifecycle state, leaderless process groups, ownership collisions, and
243+
ownership-proven sessions with no running Compose instance. Recently updated sessions are treated
244+
as possible in-flight startups for five minutes and stay untouched. `hack doctor --fix` reaps only
245+
established orphans after rechecking runtime liveness and mux ownership; unverified same-name
246+
sessions stay untouched.
219247

220248
## Tips
221249

src/agents/instruction-source.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,8 @@ export const INSTRUCTION_SECTIONS: readonly InstructionSection[] = [
185185
'For fixed-port host helpers such as SSM tunnels or local proxies, set `singleton.ports` and usually `onConflict: "adopt"` so Hack reuses a healthy existing listener instead of starting duplicate tunnel stacks.',
186186
"`singleton` is a listener guard, not process ownership transfer; adopted external processes are left running on `hack down`.",
187187
"Inspect lifecycle status via `hack projects --details` and stream via `hack logs <service-or-process>`.",
188+
"Lifecycle session recovery is ownership-proven: Hack adopts healthy token-, definition-, and environment-matched sessions, replaces owned stale sessions, and refuses to kill same-name sessions without deterministic ownership proof.",
189+
"`hack doctor --fix` reaps an orphan lifecycle session only when mux ownership is proven and its Compose instance is absent; unverified same-name sessions are never modified.",
188190
],
189191
},
190192
{

src/commands/doctor.ts

Lines changed: 63 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,10 @@ import {
9090
repairLegacyComposeEnvFileReferences,
9191
resolveProjectEnvConfig,
9292
} from "../lib/project-env-config.ts";
93-
import { inspectProjectLifecycleHygiene } from "../lib/project-lifecycle-hygiene.ts";
93+
import {
94+
inspectProjectLifecycleHygiene,
95+
repairProjectLifecycleSessions,
96+
} from "../lib/project-lifecycle-hygiene.ts";
9497
import {
9598
findMissingRegistryEntries,
9699
findOrphanRuntimeProjects,
@@ -1844,10 +1847,13 @@ async function checkProjectLifecycleHygiene({
18441847

18451848
const inspection = await inspectProjectLifecycleHygiene({
18461849
projectDir: ctx.projectDir,
1850+
projectRoot: ctx.projectRoot,
18471851
});
18481852
if (
18491853
inspection.staleEntries.length === 0 &&
1850-
inspection.orphanedProcessGroups.length === 0
1854+
inspection.orphanedProcessGroups.length === 0 &&
1855+
inspection.orphanedSessions.length === 0 &&
1856+
inspection.unverifiedSessions.length === 0
18511857
) {
18521858
return {
18531859
name: "lifecycle hygiene",
@@ -1875,12 +1881,29 @@ async function checkProjectLifecycleHygiene({
18751881
`${orphanedGroups.size} orphaned lifecycle process group${orphanedGroups.size === 1 ? "" : "s"}`,
18761882
]
18771883
: []),
1884+
...(inspection.orphanedSessions.length > 0
1885+
? [
1886+
`${inspection.orphanedSessions.length} owned lifecycle session${inspection.orphanedSessions.length === 1 ? "" : "s"} without a running instance`,
1887+
]
1888+
: []),
1889+
...(inspection.unverifiedSessions.length > 0
1890+
? [
1891+
`${inspection.unverifiedSessions.length} same-name lifecycle session collision${inspection.unverifiedSessions.length === 1 ? "" : "s"} with unverified ownership`,
1892+
]
1893+
: []),
18781894
];
18791895

1896+
let nextStep = "run: hack down";
1897+
if (inspection.orphanedSessions.length > 0) {
1898+
nextStep = "run: hack doctor --fix";
1899+
} else if (inspection.unverifiedSessions.length > 0) {
1900+
nextStep = "manual review required; no session was modified";
1901+
}
1902+
18801903
return {
18811904
name: "lifecycle hygiene",
18821905
status: "warn",
1883-
message: `${details.join("; ")} (run: hack down)`,
1906+
message: `${details.join("; ")} (${nextStep})`,
18841907
};
18851908
}
18861909

@@ -2403,6 +2426,7 @@ async function runDoctorFix(opts: {
24032426
return;
24042427
}
24052428

2429+
await maybeRepairProjectLifecycleSessions({ startDir: opts.startDir });
24062430
await maybeRepairHackd();
24072431

24082432
const paths = getGlobalPaths();
@@ -2680,6 +2704,11 @@ export async function buildDoctorRemediationPlanLines(opts: {
26802704
];
26812705

26822706
const project = await findProjectContext(opts.startDir);
2707+
if (project) {
2708+
steps.push(
2709+
"Reconcile lifecycle sessions and remove only ownership-proven sessions whose Compose instance is absent."
2710+
);
2711+
}
26832712
const trackedGenerated = project
26842713
? await inspectTrackedGeneratedFiles({
26852714
projectRoot: project.projectRoot,
@@ -2897,6 +2926,37 @@ function describeLegacyRepairStatus(input: {
28972926
return input.didPruneLegacy ? "pruned" : "prune failed";
28982927
}
28992928

2929+
async function maybeRepairProjectLifecycleSessions(opts: {
2930+
readonly startDir: string;
2931+
}): Promise<void> {
2932+
const project = await findProjectContext(opts.startDir);
2933+
if (!project) {
2934+
return;
2935+
}
2936+
const repaired = await repairProjectLifecycleSessions({
2937+
projectDir: project.projectDir,
2938+
projectRoot: project.projectRoot,
2939+
});
2940+
if (repaired.repairedSessions.length > 0) {
2941+
note(
2942+
[
2943+
`Removed ${repaired.repairedSessions.length} ownership-proven orphan lifecycle session${repaired.repairedSessions.length === 1 ? "" : "s"}:`,
2944+
...repaired.repairedSessions.map((sessionName) => `- ${sessionName}`),
2945+
].join("\n"),
2946+
"lifecycle repair"
2947+
);
2948+
}
2949+
if (repaired.failures.length > 0) {
2950+
note(
2951+
[
2952+
"Lifecycle sessions left unchanged because safe repair could not be re-proven:",
2953+
...repaired.failures.map((failure) => `- ${failure}`),
2954+
].join("\n"),
2955+
"lifecycle repair"
2956+
);
2957+
}
2958+
}
2959+
29002960
async function maybeInstallMutagenForDoctorFix(): Promise<void> {
29012961
if (getMutagenPath()) {
29022962
return;

0 commit comments

Comments
 (0)