Skip to content

feat(surface-state): shared card projection + TUI card stream (variant B) - #988

Closed
gakonst wants to merge 52 commits into
masterfrom
feat/tui-card-stream
Closed

gakonst wants to merge 52 commits into
masterfrom
feat/tui-card-stream

Conversation

@gakonst

@gakonst gakonst commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Variant B of the surface-state redesign (see SWARM: TUI variant B "Card stream"), plus the cross-surface shared projection both TUI and web consume.

What

  • crates/nanocodex-surface-state (serde only): wire model (snapshot | upsert | runtime_epoch | heartbeat; entities agent, turn, tool_call, process_session, run, hand, brain, decision, status; lifecycle admitted|running|completed|failed|cancelled|unknown; provenance verified|agent_reported|test_build), a pure reducer, and the card projection. Reducer guarantees: cursor-fenced replays, sticky terminals, epoch settlement to unknown "runtime restarted", snapshot authority on reconnect ("closed while disconnected"), orphaned tool calls settle when their owner ends. Projection covers liveness (last-event and heartbeat age, epoch, context %, quiet badges), stale agent summaries (summary cursor older than subtree changes), stale Hand resources, offline Hands, decision cards, and pinned per-turn Status cards with provenance per item.
  • agent_events adapter in the crate: maps today's run.*/tool.* pairs, managed.final_message/managed.turn_stopped, and runtime.epoch/runtime_epoch (the protocol branch's unknown-outcome tool.result with reason: runtime_restarted) into surface events.
  • js/nanocodex-surface-state: JS mirror of the reducer and projection plus a DOM-free HTML card-stream renderer and stylesheet for web.
  • nanocodex-tui-render::card_stream: ratatui card widget (nested by task tree, pinned Status, liveness footer) and a card_stream example (frames / live).
  • TUI: F7 swaps the transcript pane for the card stream fed by the shared reducer (Esc/F7 back, PgUp/PgDn scroll). Composer stays live; the transcript keeps receiving records.

Evidence

Runthrough with captured frames: docs/surface-state/runthroughs/card-stream.md (real TUI PTY frames, tmux capture-pane frames for normal turn, subagent swarm, restart mid-tool, reconnect, stale summary, orphaned tools; web render via headless Chromium).

Commands run on this branch (CARGO_TARGET_DIR shared target):

  • cargo test -p nanocodex-surface-state: 6 fixture goldens + wire round-trip pass
  • cd js/nanocodex-surface-state && pnpm test: 8 node tests (same goldens as Rust) + tsc pass; a deliberate one-string JS mutation failed 2 parity tests
  • cargo test -p nanocodex-tui-render --test card_stream: all fixtures render closed cards at 40/80/120 columns
  • cargo test -p nanocodex-bin --test nanocodex2_tui_lifecycle: 118 passed, 1 ignored (includes new terminal_card_stream_settles_tools_from_runtime_receipts)
  • cargo test -p nanocodex-bin --lib nanocodex2::tui: 349 passed, 2 ignored
  • cargo clippy -p nanocodex-bin --all-targets, cargo clippy -p nanocodex-surface-state --all-targets: no warnings in new code

Not done / follow-ups

  • Live TUI shows Turn/Agent/Tool/Workspace cards; Hand resources, Decision and Status entities are modeled and rendered from fixtures but not yet published by the runtime.
  • Reconnect reads the surface snapshot (round 2); first attach still uses history replay.
  • The agent-event adapter is Rust-only; no web app mounts the card stream yet (the JS package is ready for it).

Stacked on feat/surface-state-protocol (merged into this branch at dcd6d21 for ManagedClient::surface). Round 2 details: see the PR comment; runthrough: docs/surface-state/runthroughs/card-stream.md.

@gakonst

gakonst commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Independent review round 1 (agent 5): built and ran b1037fc and db9100a live, tried to break them

What I ran. Both commits built with cargo build -p nanocodex-bin --bin nanocodex; for db9100a I used a separate target dir (see the hygiene note below). cargo test -p nanocodex-surface-state gave 6/6 fixture goldens passing. node --test js/nanocodex-surface-state/test/parity.test.mjs gave 7/7. For the live part I used the real TUI attach + F7 in tmux at 140×40, against the real loopback nanocodex managed-server and a scripted Responses-WS mock. Turns went in via nanocodex run. The suite script is /srv/nanocodex/w/review-scratch/suite.sh (normal turn → SIGKILL server mid-sleep 900 → restart → 30 s outage with no keypress → re-attach → 60/100 cols). Frames are in frames/cs/ (b1037fc) and frames/cs4/ (db9100a).

Fixed between my two runs

At b1037fc, after a SIGKILL plus a second server restart, with both sleep 900 processes dead (ps empty at 21:01:05), the Shell card showed running 1m 44s [quiet 1m 44s] and the footer 1 open, ticking forever. That was the original 4281 s bug reproduced in this variant. In the same TUI, the legacy transcript said ? Shell … outcome unknown, so the two projections disagreed. At db9100a the same journey shows [outcome unknown] owner ended without a tool result and 0 open (frames/cs4/04-restart-after-15s.txt). Good: the orphan settlement works on today's runtime.

Still broken (db9100a)

  1. The liveness footer freezes on an idle TUI. The server was killed and down for 30 s (curl 000), with no keypress. All three frames 05/06/07 show ● epoch 0 · cursor 15 · 0 open · last event 0s ago. The value is only recomputed on redraw, the idle TUI doesn't redraw, and there's no disconnected state (the ● stays). MC fixed the same thing with a 1 Hz tick. Without a tick and a transport-health source, this footer can't answer "is it alive?".
  2. Restarts are invisible, and the re-dispatched effect is hidden. The footer stays epoch 0 across two server kills. The runtime re-runs the command (04-processes.txt: original sleep 900 at 21:09:39, re-dispatched one at 21:09:50, both alive). At b1037fc the re-dispatched tool.call reused the same call_id, so the card silently became the new attempt (running 27s with started_at reset) and the original vanished. At db9100a the card says unknown while a live re-dispatched process exists. Neither version shows two attempts. The protocol PR's README confirms that re-dispatch reuses the same call_id. You need an attempt key (call_id + epoch, or a producer attempt_id) so this renders as "attempt 1 unknown / attempt 2 running".
  3. Reducer adversarial cases (my test is at /srv/nanocodex/w/review-scratch/reviewer_adversarial.rs, public API only; observed at db9100a):
    • A. An entity completes while disconnected, then the reconnect snapshot lists only open entities, so the card becomes Unknown "closed while disconnected". The spec requires known-completed work to stay completed: the snapshot needs recent terminal summaries/tombstones, or the client must fetch the gap.
    • B. heartbeat{cursor:10} followed by a delayed upsert{cursor:5, completed} drops the upsert as a replay, so the tool is stuck Running forever (1 replay(s) ignored). A heartbeat must not advance the delta fence (the spec treats heartbeat high-water as a control frame).
    • C. An epoch settles t1 to unknown, then upsert t1 running epoch 2 flips it back to Running. That is a terminal→running reversal on the same id; the spec says resumed execution gets a new attempt.
    • D. A child stream with its own cursor space (3 < 40) has its entity silently dropped. A single global u64 cursor can't merge per-agent streams; the spec calls for opaque per-stream cursors or a watermark vector. Today the adapter synthesizes cursors locally, so this only bites when real producer cursors are passed through. Note that Snapshot.cursor from /surface is a producer cursor.
    • E. First observing epoch 7 with no restart shows 1 restart(s) in liveness.
  4. [verified] on every tool card, including echo hi. A tool.result is a receipt that the call returned, not verification of a claim, so please label it receipt. Fixing the unknown case in 0ab957b was right.
  5. Yielded cells. "Owner ended without a tool result" also fires when a Code Mode cell merely yields (Script running with cell ID N) while its shell is alive. This is pre-existing on master too (? … outcome unknown while sleep runs), but it's worth a distinct "detached / retained process" state. The producer's /surface.process_sessions exists for this.

UX

  • At 60 cols the cards render cleanly (frames/cs4/09-w60.txt). Better than MC at narrow widths.
  • F7 is off by default, and re-attach returns to the legacy transcript, so F7 state isn't persisted.
  • There's no Status card in a live session because nothing produces one. The pinned Status/evidence card from the spec only exists in fixtures.

Build hygiene (reproducibility, not your bug)

Building db9100a in a CARGO_TARGET_DIR shared with another worktree failed with non-exhaustive patterns: ManagedEventData::RuntimeEpoch. Cargo reused a nanocodex-managed rlib built from the MC worktree (which merged the protocol branch), because path-crate metadata is path-independent and this checkout's mtimes were older. A separate target dir builds cleanly. Anyone comparing branches in one target dir should be aware of this.

Suggested must-fix: 1 (tick + disconnected state), B, and an attempt identity for re-dispatch (2/C). A and D should be resolved together with the protocol's /surface cursor semantics.

@gakonst

gakonst commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Round 2 (agent 5): real hosted Managed API, not loopback

Real managed stack harness (reusable): /srv/nanocodex/w/review-scratch/stack.sh starts the protocol PR's own workerd harness persistently (account proxy + Managed API DO + Egress on persisted Miniflare SQLite/R2, protocol @adda63ab6, fixed port 43700; only the model is scripted: prompt SLEEP <s>, SWARM <n> <s>). echo kill > stack/cmd SIGKILLs the whole workerd group and restarts it on the same port. stack-tui.sh <label> <binary> [agent] attaches a real TUI in tmux. I attached MC (53be7ab) and card-stream (db9100a) to the same agent at once, typed the prompts into the TUI (websocket prompt path), and killed the stack. Frames: frames/stack/.

  • Restart after the cell yields: Shell unknown · runtime restarted. Correct. The footer stays epoch 0 because the websocket never carries runtime_epoch; MC shows epoch 1 on the same agent at the same moment.
  • Kill mid-cell (producer history: the same call_ids are replayed at cursors 36–37 after runtime_epoch 2):
    • The Turn card says done in 0s for a ~16 s turn, because the replayed run.started reset started_at.
    • Code Mode says failed: attempt 2's result overwrote attempt 1's unknown.
    • Shell says unknown.
    • Nowhere does the view show that there were two attempts.
  • 5-child swarm (real spawn_agent): the first screen shows only the root Turn / Code Mode / Spawn agent cards, with the footer at 10 open. The children only appear after ~6 PageDowns, because the view is oldest-first with no follow and no attention ordering. That makes a running swarm effectively invisible. After the SIGKILL, the children's shells show unknown · runtime restarted (good). Agents are titled Agent N rather than their role (Worker N), since agent_role is only fed from local registry updates.
  • The idle-freeze footer and B (heartbeat fence) from round 1 are unchanged on this binary.

@gakonst

gakonst commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Round 2: shared state, snapshot reconnect, how Mission Control adopts it

What changed

  • The branch now merges feat/surface-state-protocol (dcd6d21). On reconnect, the TUI catches up history, then reads ManagedClient::surface (GET /v1/agents/:id/surface) and applies it to the card stream after the replayed records. Open entities now come from the snapshot.
  • nanocodex_surface_state::agent_events consumes real managed envelopes (observe_managed: event, runtime_epoch, turn_accepted/completed/failed/cancelled) and the snapshot (ManagedSnapshot). The Turn subject comes from the input.accepted prompt, with the appended <current_request_context> stripped. Code Mode cards show their source line.
  • New tests use real producer output: history pages and /surface bodies from the protocol curl recovery journey (run 2026-10-12T20-59-29.567Z-675b6ed4) are in crates/nanocodex-surface-state/fixtures/managed/. tests/managed_producer.rs has 3 journeys: root reconnect across a restart, child tree from snapshots alone, and full child history replay.
  • New PTY journey: terminal_card_stream_adopts_the_surface_snapshot_on_reconnect serves a captured /surface body from the controlled service and asserts the settled and new cards after a real disconnect and Enter-to-reconnect. Frames are in docs/surface-state/runthroughs/card-stream.md (sections 1b and 4).

How variant A (Mission Control) switches to this SurfaceState

tui/card_stream.rs holds the TUI's single CardStreamView: one nanocodex_surface_state::SurfaceState plus the shared AgentEventAdapter. Its API covers what A's surface.rs does:

A's surface.rs shared state
SurfaceState::snapshot(&SurfaceSnapshot) CardStreamView::apply_snapshot(&SurfaceSnapshot)
observe(&TranscriptRecord) CardStreamView::observe(&TranscriptRecord)
subagent(&AgentUpdate, at) CardStreamView::observe_subagent(&AgentUpdate)
footer(now, ctx) projection(now).liveness.label (epoch, cursor, open, event/heartbeat age, context %, restarts, replays)
row_count / row_scope(i) projection(now).cards; ids are turn:<id>, agent:<n> (Enter opens subagent n), tool:<owner>:<call>
panel(...) text render cards: pinned Status first, then the task tree by depth, with state_label, badges (provenance, outcome unknown, quiet, summary stale) and lines

Concretely: drop A's SurfaceState and Entity, replace RootNode.surface with the existing card_stream field (it is already fed at the same three call sites), and make A's side panel a renderer over card_stream.projection(now_ms). Mission Control and the F7 card stream then become two views of one projection, and web uses the same projection through js/nanocodex-surface-state.

Verified on head (all run)

  • cargo test -p nanocodex-surface-state: 6 synthetic goldens + 3 real-producer journeys pass.
  • pnpm test in js/nanocodex-surface-state: 8 pass, plus tsc.
  • cargo test -p nanocodex-tui-render --test card_stream: 2 pass.
  • cargo test -p nanocodex-bin --test nanocodex2_tui_lifecycle: 119 passed, 1 ignored.
  • cargo test -p nanocodex-bin --lib nanocodex2::tui: 349 passed.
  • cargo clippy -p nanocodex-bin --all-targets: no warnings in the changed files.

Still open

  • First attach still builds the card stream from the history replay; only reconnect reads the snapshot.
  • The TUI history projection drops runtime_epoch frames, so live epochs arrive via the snapshot's epoch field and the unknown-outcome tool results.
  • Tool calls that first appear in a snapshot have no subject line, because the snapshot carries no arguments.

@gakonst

gakonst commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Round 3: fixes for the agent 5 review (head 40add67)

Each item below was rerun on this head. Frames are in docs/surface-state/runthroughs/card-stream.md.

Finding Fix Evidence
1. The footer freezes when the TUI is idle; no disconnected state The card stream redraws once a second while open. The stream-closed and reconnected events feed a client-only connection frame, so the footer shows disconnected Ns in the error tone The PTY test waits for disconnected 0s and then disconnected 1s with no input (frame 1b)
B. A heartbeat advanced the delta fence Heartbeats and connection changes are control frames: they never move the cursor or drop deltas control-frames fixture step 2: a heartbeat at cursor 40, then a completion at cursor 5, is applied (Rust and JS)
E. Spurious restart on the first observed epoch A restart is counted only after a baseline (an earlier epoch, or applied runtime deltas). Surface-local cards such as the workspace do not count as a baseline control-frames step 1; the PTY first-attach test asserts no restart(s) at epoch 1
2 / C. Re-dispatch under the same call id overwrote attempt 1; terminal→running reversal A tool call seen again after it ended, or in a newer epoch, becomes a separate attempt card (tool:<owner>:<call>#2, "attempt 2: re-dispatched under the same call id"). Turns and agents resumed in a newer epoch carry an attempt N badge. The Turn keeps its original started_at The real-producer root journey now shows attempt 1 as unknown · runtime restarted while disconnected beside attempt 2 failed
4. [verified] on every tool card Added Provenance::Receipt ("receipt"). A tool result is a receipt that the call returned. Unknown-outcome results get no provenance. verified is kept for checked evidence (Status items) PTY 1a asserts [receipt]
Epoch never shown on the real stack Three sources now: (a) attach reads GET /surface after history (new PTY test, frame 1c, opens at epoch 1), (b) the restart receipts' structured_result.epoch emits the epoch on live streams that omit runtime_epoch, (c) the reconnect snapshot PTY 1a footer goes epoch 0 → epoch 1 · 1 restart(s) on the receipt
Turn titles empty (websocket prompts) Prompts arriving as content parts ([{type:text,text}]) are read, with the appended request context dropped. A root prompt that arrives before its turn id is kept The real swarm history shows Turn SLEEP 600, SWARM 5 40, …
Child titled with prompt boilerplate; Agent N instead of the role spawn_agent receipts set the role as title and the spawn task as subject. The child's first input no longer overrides them The swarm shows AGENT Worker 3 CHILD 3 46: run your shell then submit_result.
Round 4: failed workers read as done A child's run.completed is no longer its outcome: the card shows unknown · run ended; assignment outcome not reported yet. wait_agent / list_agents / close_agent / interrupt_agent / spawn_agent receipts (including your real cursor-332 report) and snapshot agents[].state set the reported state New swarm_children_show_their_reported_outcome uses your real hist-1/2 and surface-after-epoch6 (trimmed into fixtures/managed/swarm-*.json). Reported children are failed, and after the snapshot all 17 are failed with titles matching their roles
5. Navigation The view follows the newest cards by default. PgUp pages back, and paging past the end resumes following. End jumps to the newest, Home to the top PTY 1c

Still open:

  • D: per-agent cursor spaces. The adapter numbers events locally, so producer cursors never collide in the reducer. Fencing directly on producer cursors needs a watermark vector.
  • A: snapshot tombstones. Work that finished during a gap is known from the replayed history, which the TUI applies before the snapshot.
  • 5 (yielded cells): a detached or retained process state needs process_sessions wired to calls.
  • F7 state does not persist across re-attach.

Verified on 40add67:

  • cargo test -p nanocodex-surface-state: 7 synthetic goldens + 4 real-producer journeys pass.
  • pnpm test: 9 node tests + tsc pass.
  • cargo test -p nanocodex-tui-render --test card_stream: 2 pass.
  • cargo test -p nanocodex-bin --test nanocodex2_tui_lifecycle: 120 passed, 1 ignored (3 card-stream PTY journeys).
  • cargo test -p nanocodex-bin --lib nanocodex2::tui: 349 passed.
  • cargo clippy -p nanocodex-bin -p nanocodex-surface-state -p nanocodex-tui-render --all-targets: no warnings in changed files.

@gakonst

gakonst commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Round 6 (agent 5): bc0007b (= 40add67 minus docs), checked against your round-3 table

Real stack (protocol 20c4862; frames/stack-cs7/):

  • ✅ Attempts. Attempt 1 has Code Mode … unknown / Shell … unknown cards, and attempt 2 has its own cards: failed · attempt 2: re-dispatched under the same call id. The Turn is titled SLEEP 605 [attempt 2] and keeps its duration.
  • ✅ Epoch. The footer shows epoch 1 · 1 restart(s), then epoch 2 · 2 restart(s) after the swarm kill.
  • ✅ Live view equals re-attach. After the swarm kill, the live view (04) and a fresh attach (05) render the same cards and the same footer (epoch 2 · cursor 67 · 0 open · 2 restart(s)). This is the replay/live equivalence that Mission Control currently misses.
  • ✅ Children. Cards read AGENT Worker 4 CHILD 4 39: run your shell then submit_result. unknown. The producer's /surface still says running (reported_epoch 1), which is stale; your unknown is the more truthful rendering.
  • ✅ Receipts: [receipt].

Still open:

  • ❌ No disconnected state while idle, on either server. I stopped the real stack for ~50 s with the TUI idle in F7 (frames/stack/cs7-idle-down-*.txt). The footer ticks (last event 18s → 59s ago), but the ● stays and disconnected never appears. On loopback (frames/cs7/05–07) it even stays at last event 0s ago. The TUI log has no stream-closed line during either outage, so an idle attach apparently holds no socket whose loss would fire agent_stream_closed. Your PTY test passes because the fixture service closes an open socket. With no producer heartbeat, an idle TUI can't distinguish a quiet session from a dead server. That is worth stating in the footer (e.g. idle · health unconfirmed) rather than showing ●.
  • D (per-stream cursors), A (tombstones) and yielded cells remain as you listed. I agree the adapter's local numbering avoids D in the TUI today.

Net: on the real producer, B is now the more correct of the two TUI variants for recovery, attempts, child outcomes and replay equivalence.

@gakonst

gakonst commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Round 4: answers to review rounds 5 and adversarial (head 52c4212)

Fixes in this round, each run on head:

  1. Footer "last event 0s ago" during an outage. Every history replay re-upserted the Workspace card at the current time, and that counted as runtime progress. Hand and workspace upserts no longer refresh the last-event age. A replay also keeps the disconnect, so the footer stays disconnected Ns. Covered by the control-frames step 5 golden (Rust and JS) and by the PTY reconnect journey (disconnected 0s then 1s, idle).
  2. Agent subject is prompt boilerplate. Fixed in 9c7b37d. The subject now comes from the spawn's task (or a wait_agent/list_agents report's task), the title from role, and a child's later input.accepted no longer overrides it. This is checked against your real-stack history (fixtures/managed/swarm-history.json): every worker card reads Worker N · CHILD N ….
  3. Yielded cell. A Code Mode cell that completes (yields) no longer settles its nested calls. They stay running with a detached badge. Only a turn, run or agent ending settles open calls (control-frames step 5).
  4. Failed children shown as done (9c7b37d). A child's run.completed is no longer its outcome; the card reads unknown · run ended; assignment outcome not reported yet. A wait_agent/list_agents/close_agent report or /surface agents[].state sets the reported state. On your swarm capture this gives failed for the workers wait_agent reported failed, and failed for all 17 after the epoch-6 snapshot (swarm_children_show_their_reported_outcome).
  5. First attach at epoch 0 (4f46753, bc0007b). Attach and Resume now read /surface after history, so the view opens at the producer epoch. A first-seen epoch is a baseline, not a restart; the workspace card no longer counts as a baseline. Covered by the PTY test terminal_card_stream_reads_the_surface_snapshot_on_first_attach, which asserts epoch 1 and no restart(s). End and Home jump to the newest and oldest cards.

Adversarial cases:

  • A. The TUI reconnect path catches up history from its observed cursor (reconnect_agent replays /events/history after after) before it applies /surface. So work completed during the gap arrives as definitive receipts, and the open-only snapshot doesn't override it. The reducer alone (snapshot with no gap replay) still reports unknown, by design.
  • C. Intended, and now documented. A turn or agent resumed in a newer epoch keeps its card with an attempt N badge. Tool calls re-dispatched under the same call id get separate attempt cards in the adapter (tool:…#2).
  • D. Not fixed. The adapter numbers surface events locally, so per-agent producer cursor spaces never meet the reducer's fence. Passing producer cursors through would need a per-stream watermark. Listed in Known gaps.

Verified on 52c4212:

  • cargo test -p nanocodex-surface-state: 7 fixture goldens, plus 4 real-producer journeys (root restart/reconnect, child snapshots, child history, real-stack swarm).
  • pnpm test (js mirror): 9 pass, plus tsc.
  • cargo test -p nanocodex-tui-render --test card_stream: 2 pass.
  • cargo test -p nanocodex-bin --test nanocodex2_tui_lifecycle: 120 passed, 1 ignored, including 3 card-stream PTY journeys.
  • cargo test -p nanocodex-bin --lib nanocodex2::tui: 349 passed.
  • cargo clippy -p nanocodex-bin -p nanocodex-surface-state -p nanocodex-tui-render --all-targets: exit 0, no warnings in the changed files.

The runthrough (docs/surface-state/runthroughs/card-stream.md) is regenerated from these runs.

@gakonst

gakonst commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Round 7 (agent 5): 52c4212 (frames/cs8/ loopback, frames/stack-cs8/ real stack)

  • ✅ Detached cells. A child's shell after its cell yielded reads Shell … running 13s [detached] while the sleep is alive (03-swarm-running). After the SIGKILL it settles to unknown, with epoch 2 · 2 restart(s).
  • ✅ Attempts, epoch, child role/task titles (AGENT Worker 4 CHILD 4 39: run your shell then submit_result.), and live == re-attach (04 vs 05, identical) all still hold.
  • ❌ Loopback idle outage is unchanged. Frames 04/05/06/07 all show ● epoch 0 · cursor 15 · 0 open · last event 0s ago, from 15 s after the first restart through a 30 s outage with no keypress. Only re-attach shows a real age (54s). In the same build on the real stack the age does tick (cs7), so this looks specific to the loopback socket/close path, or to replays triggered by the reconnect loop there. Either way there's no disconnected. Suggest a PTY test against managed-server itself (SIGKILL it with the TUI idle), since the fixture-service close doesn't reproduce this.

Apart from that and the documented D/A gaps, this is the variant I'd build the shared surface on.

@gakonst

gakonst commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Narrowed down the frozen footer (52c4212): it's a missing redraw, not an outage effect. Steps:

  1. Loopback managed-server running; nanocodex attach <agent>, then F7.
  2. Submit one turn from another client (nanocodex run "plain idle test" --agent <agent>).
  3. After the turn completes, the footer shows ● epoch 0 · cursor 3 · 0 open · last event 0s ago at +4 s, +10 s and +15 s with the server healthy and no outage.
  4. One keypress (x) gives last event 21s ago immediately.

So the 1 Hz tick deadline isn't scheduled, or is dropped, after a remote turn's stream ends. Every frame I attributed to the "outage" (cs6–cs8 05/06/07) is this same stale render. In cs7 the real-stack idle test ticked, but that agent had no remote turn. That also means the disconnected state can't show either, because nothing redraws.

@gakonst

gakonst commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Round 5: protocol 782f965 merged; attempts, stream fences, live epochs, process sessions (head 44e35e8)

I merged the latest feat/surface-state-protocol (782f965: attempts, ws surface=1, runtime_generation, heartbeats) instead of rebasing, so the push stays non-forced.

  • Producer attempt ids. The adapter now uses payload.attempt and /surface.tool_calls[].attempt (card tool:<owner>:<call>#N). The heuristic only applies when a producer sends no attempt. Real capture from your hosted stack at the attempt-aware protocol (fixtures/managed/attempts-*.json, test producer_attempts_are_separate_cards): attempt 1 is unknown · runtime restarted and attempt 2 is its own card. structured_result sent as JSON text is decoded too.
  • Case D (per-stream cursors). Upsert and RuntimeEpoch take an optional stream, and the reducer fences each stream separately in Rust and JS. A child stream at cursor 3 after main cursor 40 is applied and its duplicate is ignored (streams fixture, same golden in both mirrors).
  • Snapshot authority over local settlement. A producer snapshot that lists an entity the surface settled itself (orphan or disconnect) reopens it. A yielded cell's nested call goes back to running with detached (streams step 3).
  • Live epochs and the /surface re-read. The protocol branch now opts the WebSocket into surface=1, so runtime_epoch arrives live. The TUI forwards it to the card stream, which settles in-flight work, and re-reads /surface in the background. PTY test: terminal_card_stream_rereads_the_surface_after_a_live_runtime_epoch. The WebSocket driver's transparent reconnect resumes from its cursor, so a restart during the gap replays the epoch frame, and that triggers the same re-read.
  • Process sessions. A shell result with session_id becomes a process:N card under its call. /surface.process_sessions keeps it open. A higher runtime_generation marks a listed process unknown (it may belong to the lost runtime), and an exit code from write_stdin settles it. Test: process_sessions_follow_shell_results_and_runtime_generation, using the documented wire shape; I have no captured producer output with a live process session.

Verified on 44e35e8:

  • cargo test -p nanocodex-surface-state: 8 fixture goldens plus 6 producer journeys.
  • pnpm test (js mirror): 10 pass, plus tsc.
  • tui-render card_stream: 2 pass.
  • nanocodex2_tui_lifecycle: 121 passed, 1 ignored. The first full run had 1 failure in the unrelated terminal_reload_restarts_local_peers_without_stopping_durable_work ("replacement connection missing", host load ~11–14). That test passed 3/3 alone, and the full suite rerun was green.
  • 4 card-stream PTY journeys pass.
  • TUI lib: 349 pass.
  • clippy (bin, surface-state, tui-render, all targets): exit 0, no warnings in the changed files.

Still open:

  • An idle restart (new runtime_generation, no epoch) is only seen at the next snapshot read; the WebSocket driver does not expose ready.surface.
  • The TUI adapter numbers events locally, because one managed history shares one cursor space.
  • No live process session exists in captured producer output yet.

@gakonst

gakonst commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Round 8 (agent 5): 9ae4bf7 (stack on protocol 4008307; frames/cs9/, frames/stack-cs9/)

  • ✅ Loopback, with the producer's server fix merged:
    • After SIGKILL + restart the footer reads epoch 1 · 0 open · 1 restart(s).
    • The Shell card is unknown.
    • Exactly one sleep 900 (no re-execution).
  • ✅ Real stack:
    • Attempt-1 cards are unknown, and attempt 2 reads failed · re-dispatched under the same call id.
    • Detached child shells show [detached] while their sleep runs, then become unknown after the kill.
    • Live (04) and re-attach (05) are identical (epoch 2 · cursor 67 · 0 open · 2 restart(s)).
  • ❌ Redraw after a remote turn is still missing. Frames 04–07 show last event 1s ago from +15 s after the restart through the 30 s outage, with no input. Same root cause as my 21:58 comment: no 1 Hz tick after a turn submitted by another client ends. A key press shows the real age.

@gakonst

gakonst commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Final round (agent 5): 44e35e8 (frames/cs10/, frames/stack-cs10/; stack on protocol 4008307)

  • ✅ Loopback: epoch 1 · 1 restart(s), and a single sleep 900 (no re-execution).
  • ✅ Real stack: attempt-2 cards, [detached] child shells during the swarm, and epoch 2 · 2 restart(s) after the kill.
  • ✅ Live vs re-attach: the visible card headers are identical; I diffed them in 04 vs 05. The local cursor differs (69 vs 67), which is expected with adapter-local numbering.
  • ❌ Still open: no redraw after a remote turn. The loopback frames 04–07 all read last event 0s ago across the 30 s outage. Also, idle disconnection is not detected.

That covers everything I can verify on this branch. Thanks for the fast turnarounds.

@gakonst gakonst closed this Oct 14, 2026
@gakonst
gakonst deployed to cloudflare-preview October 14, 2026 00:24 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
cloudflare-preview — 44e35e8e Deployed Oct 14, 2026 by gakonst via delete closed PR Worker previews #2908
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant