ActiveMemory · CoderMungan · Sep 8, 2026 · Sep 8, 2026 · Sep 8, 2026 · Sep 8, 2026
@@ -28,6 +28,16 @@ DO NOT UPDATE FOR:
 -->
 
 
+## [2026-09-08-153411] Hub cluster mode never started: two defects hid each other
+
+**Context**: Wiring issue #96's leadership fields through the Status RPC looked like plumbing until the built binary was exercised live: ctx hub start --daemon --peers started a standalone hub (RunDaemon built its re-exec argv from --port and --data-dir only, so --peers was parsed and never forwarded), and in the foreground Run advertised fmt.Sprintf(':%d', port+1) to raft.NewTCPTransport, which refuses an unspecified advertise address.
+
+**Lesson**: A daemon re-exec argv is a second, silent flag surface: a flag missing there is a flag the process never sees, while the parent still reports success. The daemon path was also hiding the startup crash the foreground path would have shown, so neither bug was visible from the other side alone. Package tests passed throughout.
+
+**Application**: For any fork/re-exec path, extract the argv into a testable function and pin every flag it must carry (daemonArgs + TestDaemonArgs_ForwardsClusterFlags). Before claiming a CLI feature works, run the built binary through the documented flow, not just the package tests.
+
+---
+
 ## [2026-08-23-170949] Hook commands must survive four shells and hostile cwds; hosts punish pre-ctx aborts
 
 **Context**: Adversarial audit of every ctx hook surface (Claude/Codex/Copilot manifests, 16 Copilot wrapper scripts, OpenCode plugin, trace hook, plugin-reload) after the Codex non-repo-cwd anchor bug: 20 confirmed defects in 7 classes.

@@ -1810,7 +1810,14 @@ links back to the spec for detail.
   `--bootstrap` node calls `BootstrapCluster`, others
   join via `AddVoter`. Persist a `bootstrapped` flag
   in the raft data dir to avoid double-bootstrapping
-  on restart. #priority:medium #added:2026-04-11 #pr:60
+  on restart. Partially done in Phase HL: `--join`
+  starts a node that bootstraps nothing and
+  `ctx hub peer add` calls `AddVoter`, and
+  `ErrCantBootstrap` is tolerated on restart. Still
+  open: making one node the designated bootstrapper
+  instead of every `--peers` node bootstrapping the
+  same list, and persisting the flag.
+  #priority:medium #added:2026-04-11 #pr:60
   #audit:H-12
 - [ ] **H-13** Follower-side replication validation:
   call `validateEntry` on every entry received from
@@ -1844,11 +1851,15 @@ links back to the spec for detail.
   by the sysadmin. Precondition for any non-localhost
   multi-node deployment. #priority:critical
   #added:2026-04-11 #pr:60 #audit:H-10,H-11
-- [ ] **H-28** Decouple Raft bind port from gRPC port.
+- [x] **H-28** Decouple Raft bind port from gRPC port.
   Accept a dedicated `--raft-bind` flag; default to a
   random high port or refuse to start. Makes port
-  scanning less productive. #priority:low
-  #added:2026-04-11 #pr:60 #audit:H-28
+  scanning less productive. Done in Phase HL: the flag
+  is required for cluster mode and rejects wildcard,
+  bare-port and host-less addresses, so a hub refuses
+  to start rather than binding an address no peer can
+  dial. Spec: specs/hub-status-cluster-leadership.md
+  #priority:low #added:2026-04-11 #pr:60 #audit:H-28
 - [ ] Signed-entry mode: publishing clients sign their
   entries with a per-client signing key; followers
   verify on replication. Eliminates the "trust the
@@ -3102,3 +3113,32 @@ work is the delivery layer: plugin root, manifests, deployer, parser, docs.
 - [x] [CX6] Verification gate: make lint, make test, make audit green; live ctx setup codex --write + codex exec hook run (SessionStart context injection, UserPromptSubmit nudges, SessionEnd journal import) recorded in the PR; DECISIONS entries for plugin-root placement, TOML append strategy, skill generation, memories non-goal. Spec: specs/codex-integration.md #priority:medium #session:581183bc #branch:feat/codex-integration #commit:ce5a8328 #added:2026-08-23-120739
 
 - [ ] [CX7] Follow-up: Windows commandWindows overrides for the Codex hooks manifest (hooks currently require a POSIX shell with git on PATH). Spec: specs/codex-integration.md #priority:medium #session:581183bc #branch:feat/codex-integration #commit:ce5a8328 #added:2026-08-23-120739
+
+### Phase HL: Hub Status Cluster Leadership (issue #96)
+
+Spec: `specs/hub-status-cluster-leadership.md`. Read it before starting any
+HL task. The Raft `Cluster` on `Server` is never read by the Status RPC, and
+the three cluster-ish lines `ctx hub status` prints today (role, leader,
+peers) are derived from listener counts, the dialed address and the project
+count. HL wires the real state through response → handler → render.
+Issue: https://github.com/ActiveMemory/ctx/issues/96
+
+- [x] [HL1] Cluster: `LeaderAddr` returns the Raft address it is named for (not the ServerID), `Peers()` reads the committed configuration, and `BootstrapCluster`'s error is checked (tolerating `raft.ErrCantBootstrap` on restart). Spec: specs/hub-status-cluster-leadership.md #priority:medium #branch:fix/hub-status-cluster-leadership #issue:96 #added:2026-09-08-134421
+
+- [x] [HL2] Wire: `StatusResponse` gains `ClusterEnabled`, `IsLeader`, `LeaderAddr`, `ClusterPeers`; `hubStatus` populates them from `s.cluster`; `cfgWarn.HubClusterPeers` for a failed configuration read. Spec: specs/hub-status-cluster-leadership.md #priority:medium #branch:fix/hub-status-cluster-leadership #issue:96 #added:2026-09-08-134421
+
+- [x] [HL3] Render: `ClusterStatus` takes `ClusterStatusInfo`; standalone prints role + entries only, cluster mode prints the real leader and peer count; `RoleLeader`/`RoleStandalone` replace the `RoleActive` listener-count heuristic. Spec: specs/hub-status-cluster-leadership.md #priority:medium #branch:fix/hub-status-cluster-leadership #issue:96 #added:2026-09-08-134421
+
+- [x] [HL4] Docs: hub-cluster recipe expected output, docs/cli/hub.md, commands.yaml description, docs/operations/hub.md monitoring section (drops the nonexistent `--exit-code` flag and the per-peer lag claim), internal/hub/doc.go. Spec: specs/hub-status-cluster-leadership.md #priority:medium #branch:fix/hub-status-cluster-leadership #issue:96 #added:2026-09-08-134421
+
+- [x] [HL5] Tests: standalone vs single-node-Raft Status contract, `Peers()` excludes self, three rendered shapes; `make lint` and `make test` green. Spec: specs/hub-status-cluster-leadership.md #priority:medium #branch:fix/hub-status-cluster-leadership #issue:96 #added:2026-09-08-134421
+
+- [x] [HL6] `ctx hub stepdown` printed "Leadership transferred" and `ctx hub peer add|remove` printed a peer confirmation, none of which reached the Raft node (`Cluster.Stepdown()` had no caller and raft's AddVoter/RemoveServer were never called). Wired in this PR as two admin-token-gated RPCs (Peer, Stepdown) with leader-only preconditions; see HL9. Spec: specs/hub-status-cluster-leadership.md #priority:medium #branch:fix/hub-status-cluster-leadership #issue:96 #added:2026-09-08-134421
+
+- [x] [HL7] Daemon cluster flags: `RunDaemon` built the re-exec argv from --port and --data-dir only, so `ctx hub start --daemon --peers ...` started a standalone hub and reported success. argv extracted to a testable `daemonArgs` that forwards both cluster flags. Spec: specs/hub-status-cluster-leadership.md #priority:medium #branch:fix/hub-status-cluster-leadership #issue:96 #added:2026-09-08-134421
+
+- [x] [HL8] Raft bind address (closes H-28): `--raft-bind` binds and advertises the Raft transport, `--peers` becomes the other nodes' Raft addresses, wildcard/bare-port/host-less values are rejected with a message naming the flag, and --raft-bind alone runs a self-electing single node. Before this, `Run` advertised ":port+1" and every cluster start died on "local bind address is not advertisable". Spec: specs/hub-status-cluster-leadership.md #priority:medium #branch:fix/hub-status-cluster-leadership #issue:96 #added:2026-09-08-134421
+
+- [x] [HL9] Cluster commands wired: admin-gated Peer and Stepdown RPCs (`Cluster.AddPeer`/`RemovePeer` over raft AddVoter/RemoveServer, `Cluster.Stepdown` over LeadershipTransfer), `raft.ErrNotLeader` mapped to FailedPrecondition naming `ctx hub status`, `--token`/`CTX_HUB_ADMIN_TOKEN` resolution shared by revoke/peer/stepdown via core/admin. Spec: specs/hub-status-cluster-leadership.md #priority:medium #branch:fix/hub-status-cluster-leadership #issue:96 #added:2026-09-08-163000
+
+- [x] [HL10] Join mode: `ctx hub start --join` brings up the Raft transport without bootstrapping so a leader can add the node with `ctx hub peer add` (the AddVoter half of H-12; `peer add` is meaningless without it). `--join` with `--peers` is rejected. Verified live: peer add took the leader from Peers: 0 to Peers: 1 and the joiner reported Role: Follower. Spec: specs/hub-status-cluster-leadership.md #priority:medium #branch:fix/hub-status-cluster-leadership #issue:96 #added:2026-09-08-163000
@@ -62,13 +62,34 @@ the daemon with `ctx hub stop` (see below).
 #### Cluster Mode
 
 For high availability, run multiple hubs with Raft-based
-leader election:
+leader election. `--raft-bind` is the address this node binds
+its Raft transport to and advertises to the others, and
+`--peers` lists the `--raft-bind` addresses of the other
+nodes — Raft addresses, not hub ports:
 
 ```bash
 ctx hub start --port 9900 \
+  --raft-bind host1:9901 \
   --peers host2:9901,host3:9901
 ```
 
+`--raft-bind` must name a host a peer can dial. A bare port
+(`:9901`) or a wildcard (`0.0.0.0:9901`) is rejected at
+startup, because Raft refuses to advertise an address that
+does not identify this node to anyone else.
+
+`--raft-bind` on its own — with no `--peers` — runs a
+single-node Raft cluster that elects itself. That is the
+cheapest way to see the leadership fields of
+[`ctx hub status`](#ctx-hub-status) before adding nodes.
+
+To add a node to a cluster that is already running, start it
+with `--join` instead of `--peers`: it brings up its Raft
+transport, bootstraps nothing, and waits for
+[`ctx hub peer add`](#ctx-hub-peer) on the leader to hand it a
+configuration. `--join` and `--peers` together are an error —
+a node either bootstraps a cluster or joins one.
+
 Raft is used **only** for leader election. Data replication
 uses sequence-based gRPC sync on the append-only JSONL log;
 there is no multi-node consensus on writes. See the
@@ -77,12 +98,14 @@ setup and the Raft-lite durability caveat.
 
 #### Flags
 
-| Flag         | Description                                      | Default          |
-|--------------|--------------------------------------------------|------------------|
-| `--port`     | Hub listen port                                  | `9900`           |
-| `--data-dir` | Hub data directory                               | `~/.ctx/hub-data/` |
-| `--daemon`   | Run the hub server in the background             | `false`          |
-| `--peers`    | Comma-separated peer addresses for cluster mode  | *(none)*         |
+| Flag          | Description                                       | Default          |
+|---------------|---------------------------------------------------|------------------|
+| `--port`      | Hub listen port                                   | `9900`           |
+| `--data-dir`  | Hub data directory                                | `~/.ctx/hub-data/` |
+| `--daemon`    | Run the hub server in the background              | `false`          |
+| `--raft-bind` | Raft address this node binds and advertises       | *(none)*         |
+| `--peers`     | Comma-separated peer Raft addresses               | *(none)*         |
+| `--join`      | Wait to be added by a leader (no bootstrap)       | `false`          |
 
 #### Validation
 
@@ -111,8 +134,34 @@ Safe to rerun: if no daemon is running, returns a
 
 ### `ctx hub status`
 
-Show cluster status: role, peers, sync state, entry count,
-and uptime.
+Show what the hub reports about itself: its role, the current
+leader, the entry count and the peer count.
+
+A hub started without `--peers` runs no Raft node, so it has no
+leader and no peers to name:
+
+```
+Role: Standalone
+Entries: 1248
+```
+
+A hub started with peers answers from its Raft node. The role is
+`Leader` or `Follower`, the leader is the address Raft holds for
+the current term, and the peer count is the committed cluster
+configuration minus the node answering:
+
+```
+Role: Leader
+Leader: 10.0.0.5:9901
+Entries: 1248  Peers: 2
+```
+
+While an election is in progress — or after quorum is lost —
+Raft knows no leader, and the line says so:
+
+```
+Leader: unknown (election in progress)
+```
 
 When the hub has disconnected any slow listeners, the output
 gains a `Dropped listeners:` line with the cumulative count.
@@ -128,15 +177,33 @@ ctx hub status
 
 ### `ctx hub peer`
 
-Add or remove peers from the cluster at runtime. Useful for
-scaling up or replacing a decommissioned node without
-restarting the leader.
+Add or remove peers in the cluster's Raft configuration at
+runtime. Useful for scaling up or replacing a decommissioned
+node without restarting the leader.
+
+The address is the peer's **Raft** address (its `--raft-bind`),
+not its hub port. Membership changes are admin-gated, like
+[`ctx hub revoke`](#ctx-hub-revoke): pass `--token` or set
+`CTX_HUB_ADMIN_TOKEN`.
+
+Only the leader can change the configuration. Run the command
+against the leader — [`ctx hub status`](#ctx-hub-status) names
+it — or the hub answers `not the leader`.
+
+A node being added must already be running with
+`--raft-bind <its address> --join`, so that it is waiting for a
+configuration instead of bootstrapping one of its own.
 
 **Examples**:
 
 ```bash
-ctx hub peer add host2:9901
-ctx hub peer remove host2:9901
+# On the new node:
+ctx hub start --daemon --port 9900 \
+  --raft-bind host4:9901 --join
+
+# On the leader:
+ctx hub peer add host4:9901 --token ctx_adm_...
+ctx hub peer remove host3:9901 --token ctx_adm_...
 ```
 
 ### `ctx hub stepdown`
@@ -146,10 +213,17 @@ new election among the remaining followers before the current
 leader steps down. Use before taking the leader offline for
 maintenance.
 
+Admin-gated like [`ctx hub peer`](#ctx-hub-peer), and
+leader-only: a follower answers `not the leader` instead of
+reporting a transfer that did not happen. The confirmation
+prints after the transfer returns;
+[`ctx hub status`](#ctx-hub-status) names the node that won.
+
 **Examples**:
 
 ```bash
-ctx hub stepdown
+ctx hub stepdown --token ctx_adm_...
+CTX_HUB_ADMIN_TOKEN=ctx_adm_... ctx hub stepdown
 ```
 
 ### See Also

@@ -146,9 +146,24 @@ or accept the higher sequence by regenerating `meta.json` from
 
 ### Leader Crash, Clean Shutdown
 
-**What happens:** `ctx hub stop` triggers `stepdown` first, so
-a new leader is elected before the old one exits. In-flight
-writes drain. Clients reconnect to the new leader transparently.
+**What happens:** `ctx hub stop` sends SIGTERM; the hub shuts
+its Raft node down and drains in-flight RPCs. It does **not**
+hand off leadership on its own — the survivors notice the
+missing heartbeat and elect a new leader a couple of seconds
+later, and clients whose streams were on the old leader have
+to be re-run (reconnect is manual, see
+[Client Loses Connection Mid-Stream](#client-loses-connection-mid-stream)).
+
+**What you should do:** for a planned restart, hand off first:
+
+```bash
+ctx hub stepdown --token ctx_adm_...   # on the leader
+ctx hub status                         # confirm the new leader
+ctx hub stop
+```
+
+That way the election happens while the old leader is still
+serving, instead of after it is gone.
 
 ### Leader Crash, Hard Fail (Kill -9, Power Loss)
 

@@ -174,20 +174,31 @@ the sequence counter and loses writes.
 Liveness probe:
 
 ```bash
-ctx hub status --exit-code
+ctx hub status
 ```
 
-Exit code `0` means the node is healthy (leader or in-sync
-follower); non-zero means degraded. Wire this into your monitoring
-of choice.
+The command exits non-zero when the RPC fails — hub unreachable,
+token rejected — so a wrapper can treat that as the liveness
+signal. On a reachable node it prints what the node knows about
+itself, and the grading is yours to do from those lines:
+
+```
+Role: Leader
+Leader: 10.0.0.5:9901
+Entries: 1248  Peers: 2
+```
 
 For cluster deployments, watch for:
 
-- **Role flaps**: the leader changing more than once per hour
-  suggests network instability or disk contention.
-- **Replication lag**: `ctx hub status` shows per-peer sequence
-  offsets. Sustained lag > 100 sequences on a follower is worth
-  investigating.
+- **Role flaps**: the `Role:` line changing more than once per
+  hour suggests network instability or disk contention.
+- **A leader nobody can name**: `Leader: unknown (election in
+  progress)` is normal for a second or two after a node starts
+  or a leader dies. Persisting past that, on a node that is
+  itself reachable, means it cannot see a quorum.
+- **A peer count that disagrees between nodes**: each node
+  reports its own committed Raft configuration, so two nodes
+  disagreeing on `Peers:` means they never agreed on membership.
 - **`entries.jsonl` growth rate**: sudden spikes often indicate a
   misbehaving `ctx connection listen` reconnect loop.