Tracking: one gateway owns every local session (#106742) — post-merge fixes #134889

Activity

1. unsupportedpastels

   added

   type/featureNew feature or requestNew feature or request

   comp/cliCLI entry point, hermes_cli/, setup wizardCLI entry point, hermes_cli/, setup wizard

   comp/gatewayGateway runner, session dispatch, deliveryGateway runner, session dispatch, delivery

   P1High — major feature broken, no workaroundHigh — major feature broken, no workaround

   sweeper:risk-session-stateSweeper risk: may lose/corrupt/mis-associate session or context stateSweeper risk: may lose/corrupt/mis-associate session or context state

   sweeper:risk-compatibilitySweeper risk: may break existing users, config, migrations, defaults, or upgradesSweeper risk: may break existing users, config, migrations, defaults, or upgrades

   comp/desktopElectron desktop app (apps/desktop/*)Electron desktop app (apps/desktop/*)

   area/sessionsSession lifecycle, resume, persistence, historySession lifecycle, resume, persistence, history

   on Oct 8, 2026
2. alt-glitch

   added

   P2Medium — degraded but workaround existsMedium — degraded but workaround exists

   needs-decisionAwaiting maintainer decision before any implementationAwaiting maintainer decision before any implementation

   comp/agentCore agent runtime: loop, agent_init, prompt builder, context-compression, responses endpointCore agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint

   comp/tuiTerminal UI (ui-tui/ + tui_gateway/)Terminal UI (ui-tui/ + tui_gateway/)

   comp/acpAgent Communication Protocol adapterAgent Communication Protocol adapter

   comp/cronCron scheduler and job managementCron scheduler and job management

   comp/pluginsPlugin system and bundled pluginsPlugin system and bundled plugins

   on Oct 8, 2026
3. 476 remaining items

   Load more

   Load more actions

4. dokterdok commented on Oct 18, 2026

   

   dokterdok

   on Oct 18, 2026

   More actions

   @FengKan880 thanks, I reproduced this on main (9e01b44dd23) with a test, not live.

   The activation event you couldn't trace is the gateway's embedded dashboard.

   • GatewayDashboardMount mounts the dashboard inside the gateway process. The first time it enters another profile's scope, _config_profile_scope calls activate_multi_profile_hosting() (web_server_profiles.py:309).
   • Two routes get there: any dashboard request with ?profile=<other>, and POST /api/profiles writing a new profile's model (_write_profile_model). So Desktop's New Bot or New Profile against that gateway is enough.
   • GatewayRunner.__init__ turns the guard off for a standalone gateway, and nothing arms it at boot. So the gateway works at first and breaks after one of those requests.
   • Arming the guard is right: that process did load another profile's config and secrets. The gap is in owner_scope, as you said.

   The fix, in your direction, for cherry-pick: b3d1a9ba5f on fix/standalone-owner-scope, on main, crediting you. I have three ready PRs open, so it's a branch for now.

   • The standalone branch of owner_scope returns launch_profile_scope_if_multiplexed(). That stays a literal nullcontext until the guard is armed, and becomes the launch profile's scope after.
   • Test: test_standalone_session_create_after_the_dashboard_served_another_profile arms the guard through the real _config_profile_scope("beta"), then sends a real session.create through tui_gateway.ws._dispatch_request.
     • On main it fails with your chain, ending at secret_scope.py:238 UnscopedSecretError.
     • With the fix it passes. Inside the scope, the launch .env key and an env-only injected key resolve; an unscoped read outside it still raises.
     • A guard-off test checks the literal nullcontext.
   • The owner_scope, launch-profile policy, secret scope, bypass policy and multi-profile hosting suites pass (72 tests), and so does scripts/check.

   Two points for the maintainers, not in the commit:

   • Safe mode once the guard is armed. A --safe-mode / ignore_user_config session then reads the launch profile's .env and terminal: settings, so a malformed config.yaml refuses it (TerminalPolicyUnavailable).
     • On main that session fails anyway, and a configured multiplex gateway's bypass already reads its profile through _profile_runtime_scope. So nothing gets worse than either today.
     • Keeping bypass file-free under an armed guard would mean carrying the bypass intent into every owner scope the session touches: the nested session_inventory read, the socket-wide scope at run_api.py:370, and later RPCs. That applies on both branches. A first attempt showed how far that reaches, so I left it out.
   • A request already in flight when the guard arms still fails, as on main. owner_scope decides once at handler entry, and session.create awaits skill preparation before build_policy. This happens once per process, and a retry works. Two ways to close it:
     • arm the guard at boot on a standalone gateway with other profile homes, as hermes serve does with activate_multi_profile_hosting_eagerly;
     • or re-enter the scope after the awaited preparation.

   The same-shaped standalone gates (_routed_scope, _execution_scope, plugin injection, media delivery) showed no failure. Each already runs inside a profile scope, or reads no secret.

   An Opus 5.5 verify round and GPT-6 Astra reviews (Codex CLI, reasoning xhigh) checked the fix; they raised the two points above.
5. dokterdok commented on Oct 18, 2026

   

   dokterdok

   on Oct 18, 2026

   Last edited by dokterdok

   More actions

   @unsupportedpastels @teknium1 thanks for landing package 10. To keep this tracker readable, here's everything I still have open after it, in one place. I'll keep this comment up to date instead of adding new ones. Everything below was found in code review and reproduced with tests; the serial-reader fix also ran live against Signal. Each finding has a probe that fails on main at 54809183676, and still fails at 045b4fd437, after package 12. The probes are in review/pkg10-probes (r2-lane*/).

   1. Fixes ready to cherry-pick

   Thanks @unsupportedpastels for landing the four P1s and #141559 today (#141578–#141582), and @teknium1 for landing CLI-34 (#140348) with its review follow-ups in #141481. The three remaining fixes now have their own issues. Each fix is rebased onto main at 493208879a and checked again there: its tests fail on main and pass with the fix, and scripts/check passes.

   Area	What the user sees on main	Issue
   Docker workspaces	with docker_persist_across_processes: false, idle containers are stopped and never removed	#141607
   Docker workspaces	a preview restart with a cwd leaves the session's workspace container	#141608
   Serial readers (Signal, IRC)	the one reader stalls for another input's whole media capture	#141609

   2. Open findings without a fix from me

   These need a design choice, so I left them to you. The direction for each is in the linked review, already checked once for side effects.

   16 P2 findings, one line each

   • Webhooks (review, direction): a bare webhook queued before a profile_routes change runs on the profile the route no longer names.
   • Gateway client (review): after any RPC lets the reader fill the overflow, the 30 s stall budget covers draining all of it, so a consumer slower than about 546 events/s gets "turn outcome is unknown". This is not the >16,384-frame F5 you listed.
   • RT-37 forward (same review):
     • Stop and queued cancel never reach the owner;
     • clarify, approval, sudo and secret prompts never reach Desktop;
     • two forwarded prompts erase each other's pending card;
     • voice fields are dropped.
   • /model --once (review):
     • /model <current> --once --reasoning high drops the level;
     • a session /reasoning followed by a --once pick is never committed.
   • Docker workspaces (review):
     • the idle reaper stops a container while a foreground command (or a managed worker's) still runs in it;
     • an idle stop still in flight kills a workspace that was just reattached.
   • Resume display (review):
     • a compaction between the two display reads hides the carried turn;
     • after a rotated compression, archived rows sit next to the wrong live row (a full tool result before its call, an old summary inside a turn);
     • a live row rewritten in place pushes the reply past the 8 MiB frame.
   • Desktop Archived (review): pinned archived chats still hide or skip rows, in three places: the merge path, the native per-profile aggregate, and an older backend's prefix.
   • Messaging slash commands (review):
     • a /skill that finishes during /stop or /new's first read still runs;
     • on a FIFO paused on a lost turn, a /skill sent before /new runs in the fresh conversation.

   Smaller, from static reading:

   • DIRECT_ALIASES is one process-wide dict, so two profiles' turns can interleave inside switch_model.
   • _cached_profile_read (hermes_cli/profiles.py:726) caches a failed read of the gateway's own profile.yaml.
   • A preview subfolder outside the home or drive prefixes keeps its host path in the container.
   • An unreadable sibling profile.yaml now logs a warning on every 5 s poll.

   3. Other fixes waiting for a home

   • @FengKan880's standalone owner_scope report (above): fixed on fix/standalone-owner-scope, with a test that reproduces it on main, crediting them.
   • fix(cron): follow a compressed run to its tip in run history and detail #139268 and fix(desktop): gate a cron tab on its run's tip, not an older segment #139269 (CRON-1 follow-up): run history and session detail follow a compressed cron run to its live tip, and Desktop gates a cron tab on that tip. Package 4 closed with CRON-1 fixed by fix(cron): find unified-runtime cron runs by their scheduler admission #139273, which documents this case on list_cron_job_runs as a known limitation, so they ended up in no package. Both apply cleanly on main.
   • fix(gateway): name the other invalid_params refusals, without echoing secrets #139260 names the refusals that still reach clients as a bare invalid_params, and keeps a client's secret out of every refusal: field, detail, log line and CLI. Rebased onto main today and checked again; it merges cleanly.
   • fix(api): make tools and tool_choice part of a keyed retry's identity #140901, fix(gateway): scan retained API images off the owner loop in linear time #140906 and fix(gateway): serve a profile parked by a transient startup failure again #140908 (for API server: a keyed retry with a different tools or tool_choice replays the first answer #140900, Gateway freezes every session at startup while it checks retained API images #140905 and A secondary profile parked by a busy store or a held Bot Chat stays offline after the condition clears #140907, filed this morning) haven't been triaged into a package yet.

   If you'd rather take any of the above another way, or drop some, tell me and I'll close them.
6. teknium1 commented on Oct 18, 2026

   

   teknium1

   on Oct 18, 2026

   Collaborator

   More actions

   Package 10 (#140719) is merged: #140967, #140968, #140969, #140970, #140971, #140972 and #140973, plus the fixes to main #141112, #141168 and #141213. Every row those PRs claimed is now fixed on main, including CLI-36, LC-20, LC-22, LC-23, RT-37, RT-38, RT-39, RT-42, RT-45, MP-16, CMD-2, CFG-11 (--ignore-user-config), KB-6, CLI-35/37, LC-24 and #140805. Still open: KB-7's boot-time delay, F5 (RPCs past the frame cap) and CFG-11 for ordinary managed workers.
7. rickhuizinga commented on Oct 18, 2026

   

   rickhuizinga

   on Oct 18, 2026

   More actions

   [KB candidate] Kanban heartbeat backstop reclaims live workers during long non-streaming LLM calls — contradicts the agent-side local-endpoint trust policy

   Reported by: Rick (5-profile multiplexed gateway on macOS, workers on
   local engines via LiteLLM at an RFC-1918 address)
   Related: #23025 (spawn-then-reclaim loop), #31752 (heartbeat bridge), KB-7 / #106742 tracker

   Summary

   Two layers of the same product make opposite policy decisions about the same call — a
   worker's iteration-budget summary, which is a single non-streaming LLM request:

   1. Agent side (run_agent.py:_compute_non_stream_stale_timeout): with an implicit default
      and a local endpoint (is_local_endpoint → loopback / RFC-1918 / Tailscale CGNAT), the
      stale timeout is float("inf") — trust the endpoint, wait indefinitely, no watchdog.
      Our LiteLLM proxy is at 172.16.x.x, so this path applies.

   2. Dispatcher side (hermes_cli/kanban_db.py:DEFAULT_CLAIM_HEARTBEAT_MAX_STALE_SECONDS,
      hardcoded 3600): during the same call the worker emits no heartbeats — non-streaming
      responses surface no chunks, and the _touch_activity → kanban-heartbeat bridge (Kanban worker runtime activity does not update board heartbeat, causing stale reclaim of active workers #31752)
      only fires on observable progress, which a non-stream call doesn't produce until it
      completes. So last_heartbeat_at goes stale, and release_stale_claims' backstop reclaims
      the claim of a live, healthy worker mid-summary.

   The agent says "wait forever"; the dispatcher kills at 60 minutes. Every worker on a local
   engine that needs >60 min for a budget summary is reclaimed while healthy.

   Observed instances

   • t_79b1cf7e, t_ef8fdb20: live workers reclaimed mid-summary (the reclaim loop the
     release_stale_claims docstring itself describes via recurring stall loop: kanban workers repeatedly reclaimed with stale_lock, zero output across respawns #23025: "no tool calls means no
     kanban_heartbeat, even though the subprocess is healthy").
   • We currently carry a local patch raising the constant 3600 → 5400 (90 min) to fit our
     slow-engine summary times. There is no env/config override for this constant, so a source
     patch is the only mechanism — it must be re-applied after every update.

   Suggested directions (maintainer's choice)

   1. Smarter backstop: when the worker PID is alive and last_activity_desc indicates an
      in-flight API call ("waiting for provider response…"), extend the claim instead of
      reclaiming. That signal already distinguishes "hung LLM call" from the logic-loop case the
      backstop was built to catch (kanban: three gaps in blocked / iteration-exhausted handling that permit infinite "unblock → re-stuck" loops #29747 gap 3).
   2. Make the budget summary streaming: handle_max_iterations already builds kwargs with
      the main loop's builder (_chat_summary_attempt); streaming the summary would feed
      _touch_activity per chunk → heartbeats stay fresh → the race self-resolves on both
      layers, with no new policy needed.

   Happy to provide claim-log evidence or test a candidate fix against our multiplexed setup.
8. RahmanShukri commented on Oct 19, 2026

   

   RahmanShukri

   on Oct 19, 2026

   More actions

   Windows finding: a relaunched client can't start the gateway when its host's job forbids breakaway.

   On Windows, hermes_bootstrap relaunches onto the store interpreter with a bare
   subprocess.call. Every borrowing data root does this, including each Multica per-task
   HERMES_HOME. The child silently breaks away from hermes.exe's
   KILL_ON_JOB_CLOSE | SILENT_BREAKAWAY_OK job into the enclosing one. When that job lacks
   BREAKAWAY_OK, spawn_unmanaged_gateway's CREATE_BREAKAWAY_FROM_JOB Popen fails with WinError 5,
   and ACP session/new returns Gateway inaccessible: windows_breakaway_unavailable on every task.
   That is the Windows face of the #138949 consumer pattern.

   Fix PR: #141458. It keeps the relaunched interpreter in a launcher-equivalent job, and names the OS
   refusal in the verdict.

   Separate note, config only, for anyone running Multica on the same machine as their messaging
   gateway: once the per-task gateway can start, it loads the bot tokens of the profile the overlay was
   seeded from. If the default gateway already serves that bot, the token lock refuses it, the per-task
   gateway exits at startup, and session/new fails with Gateway draining: not_ready. A managed-scope
   config.yaml (via HERMES_MANAGED_DIR in the Multica agent's environment only) with
   platforms.<telegram|slack|discord>.enabled: false avoids it, since an explicit disable beats env
   tokens. With both in place, a Multica task on the Hermes runtime completes end to end on Windows. It
   may be worth having overlay homes never start messaging adapters at all.
9. teknium1 commented on Oct 19, 2026

   

   teknium1

   on Oct 19, 2026

   Collaborator

   More actions

   Fixed on main: package 12 (#141321)

   Rows fixed or closed by package 12, moved here from the tracker. Every lane PR passed two independent review rounds plus Greptile and @unsupportedpastels' reviews, with each finding fixed before merge, and CI was green on the merged head (77b446b6e14). A 🟢 is merge state plus each PR's tests and probes; the live retests still owed are named per row. Findings from review that were out of scope are new rows on the tracker (GW-20 to GW-23, TUI-26, MP-17, KB-8).

   Gateway lifecycle and restart

   ID	Finding	Severity	Status
   GW-1	hermes pause now blocks CLI, TUI and Desktop turns (⏸️ Hermes is paused); main runs them.	behaviour change	🟢 fixed on main: #141357 (eea83bb603c). Kept as decided: every surface shows the same estop.paused_reply() refusal, turns already running finish, hermes pause --help and the CLI docs say interactive turns are held. Live probe on a real gateway passed.
   F1	Restart drain is 0 s by default (agent.restart_drain_timeout: 0, same as main), so restart and update interrupt in-flight turns. With 60 s the turn completed. Keep the default?	decision	🟢 decided and shipped: #141336 (0bd184f3af3). The default stays 0. A turn the restart interrupts carries "Gateway restarting…" taken from its own recorded cause (in-process and managed-worker turns), never from runner flags, so a user Stop or a pending correction is not relabelled. The key and its systemd caveat are documented.
   GW-10	#139532: a gateway restart mid-turn fences the turn unknown, which pauses the session's whole input queue, and no surface explains it or offers a way out. A multiplexed Bot Chat stopped answering for about ten hours behind a cron delivery fenced unknown (paused: unknown_execution), while the Bot panel said the gateway was disconnected. Desktop, TUI and messaging must show why a session is paused and offer resolve or discard in place. Open: should an unknown automation delivery pause the queue at all, or settle as interrupted?	broken	🟢 fixed on main: #141357 (eea83bb603c), salvages #139541 (@KoNit-K). Automation deliveries (cron, automation:*, local_automation_v1, webhook) fenced unknown settle interrupted with a transcript notice and the FIFO moves on, including at a lost managed worker's exit. A person's turn still pauses, with the reason and discard in place on messaging (/discard; /hermes discard on Slack), TUI, ACP and Desktop. Live retest pending: /discard on a real messaging platform; a multiplexed Bot Chat behind a lost cron delivery. Kanban cards: KB-8.

   Classic CLI: flags, commands and input

   ID	Finding	Severity	Status
   CLI-9	/browser connect only changes the CLI process; browser tools run in the gateway.	broken	🟢 fixed on main: #141357 (eea83bb603c). Classic /browser sends browser.manage. Live connect/disconnect now work on every gateway, scoped to the session's profile: per-profile live CDP override, per-profile reap of sessions, supervisors and Browser Use daemons (own BH_RUNTIME_DIR), and the env fallback reads the profile's own .env. Refused only from a classic CLI attached to a remote gateway. Live retest pending: two live Chromes on a multiplexed gateway; macOS. Edges: GW-23.
   T4	Status bar after Ctrl+C shows the exact context count where main shows the ~ estimate.	cosmetic	🟢 fixed on main: #141335 (d2d3fa1c1d2). Live retest pending: Ctrl+C during the first request in hermes shows ~N.

   Command layer (one gateway handler per command)

   ID	Finding	Severity	Status
   T7	Where the messaging handler and main's CLI disagree, the gateway behaviour now ships everywhere: /goal gate add is admin-only; /debug local uploads instead of rendering locally; /voice tts enables auto-TTS with voice off; bare /voice switches to voice-only replies; /reasoning show|hide and /footer save the per-platform key; /moa drops the cached agent so /refine after it finds nothing.	decision	🟢 decided and fixed: #141343 (50de19c13b6). Local sessions: the owner may add a /goal gate; /debug local never uploads (refused by name from a chat); /voice tts stays refused; /reasoning show|hide and /footer save display.platforms.cli.*, which every local surface reads first, and Desktop Settings writes both keys; a chat's /footer keeps the global key; /refine after /moa reads the session, with one review per session that the next turn cancels.
   T9	/voice playback: CLI/TUI don't read the gateway's TTS state; Ink's /voice on|off|status still uses the old sidecar call.	degraded	🟢 fixed on main: #141354 (18b79b2ba8e). Ink /voice on|off|status, wake, spoken and typed stops run on the gateway's per-session voice mode; a second session can't take over a live capture. Speech output stays off per CMD-2. Live retest pending: a real microphone. Typed stop on the shared gateway: TUI-26.

   ACP (Zed / VS Code / JetBrains)

   ID	Finding	Severity	Status
   ACP-7	Accept Edits / Don't Ask modes gone; set_session_mode and set_config_option raise.	broken	🟢 decided and fixed: #141338 (90b20222658). Gateway-owned per-session edit policy (ask / workspace_session / session) set through session.mutate, kept with the session, enforced by the gateway; sensitive paths always ask; forks start at ask; every snapshot (attach, reconnect, gap, /new, /undo) repaints the editor's mode. Live retest pending: Zed.

   Ink TUI and local Desktop

   ID	Finding	Severity	Status
   TUI-14	/browser connect|disconnect refused in Ink and Desktop.	deliberate	🟢 no longer refused: #141357 (eea83bb603c) scopes /browser connect|disconnect to the session's profile (CLI-9); Desktop's browser.manage wire is pinned by a test in #141336 (0bd184f3af3).
   TUI-16	Ink !cmd refused on non-local terminal backends.	deliberate	🟢 decided and shipped: #141336 (0bd184f3af3). The refusal stays and names the backend (field=terminal.backend); an older gateway gets a generic translated refusal.
   T10	Ink still refuses /config /export /import /snapshot /handoff /hatch /palette /timestamps /worktree /reload-mcp; /subscription, /topup and the /skills hub still use the old sidecar.	degraded	🟢 fixed on main: #141354 (18b79b2ba8e). /reload-mcp, /config, /timestamps and /save (via session.export, written in the user's cwd) run on the gateway; /export /import /snapshot /handoff /hatch /worktree /palette refuse with what to use instead. /subscription, /topup and the /skills hub already reach grant-checked gateway handlers, so nothing remains on a client sidecar. Live retest pending: /reload-mcp against a real MCP server.
   T11	Desktop still refuses or doesn't offer /config /export /import /snapshot /worktree /timestamps /palette /history /agents /rollback /save. File-writing commands need Electron-side writes because the gateway may be remote.	degraded	🟢 fixed on main: #141355 (b406d1860b9). /rollback, /agents, /config (owner projection only), /history run on gateway verbs; /save writes through Electron's dialog from session.export; /export and /import use the profile dialogs locally and refuse on a remote gateway; /timestamps and /palette are display toggles; /worktree and /snapshot refuse by name; all notices translated. Live retest pending: /save against a remote gateway; macOS and Windows dialogs.
   T15	/compress on a short session: token estimate lower than main's when the agent isn't resident.	cosmetic	🟢 fixed on main: #141335 (d2d3fa1c1d2). A cold no-op /compress prices the stored prompt and the tools[] pin, legacy name-list pins included.
   T22	Sessions created over the gateway API are platform LOCAL even from a remote client. Rename (e.g. CLIENT) touches stored rows.	decision	⚪ decided, no change: sessions created over the gateway API stay LOCAL; pinned by a test in #141336 (0bd184f3af3).

   Model and reasoning picks

   ID	Finding	Severity	Status
   MP-12	#139694: on a multi-profile gateway hosted in the default home, a /model switch in a non-default profile never commits, in the classic CLI and Desktop: the note appears, turns run on the config default, sessions.model never moves, and hermes -p X chat -m … --provider … loses its launch override. Default-profile switches commit.	broken	⚪ not reproduced on main 20acba03769 (package 12, #141321): a real multiplexed gateway with profile x committed /model through the classic CLI and session.mutate, and kept -m … --provider …; #139694 closed.
   MP-13	#139740: a Desktop model pick to a user-defined providers.<slug> endpoint always fails model_resolution_failed, while CLI and TUI /model to the same endpoint work. The session's frozen config used by session.mutate operation=model doesn't include the providers: entry when it was added after the session was created. Predates #106742, per @teknium1.	broken	🟢 fixed on main: #141337 (77b446b6e14), salvages #139746 (@KoNit-K). A pick resolves against the owning profile's current providers: (its current .env too), freezes only that entry and its key_env pointer, and replaces that entry's secrets whole; bypass sessions never import profile providers. Live retest pending: a real remote endpoint; a multiplexed secondary profile. Reorder after a pick: MP-17.

   Messaging and bots

   ID	Finding	Severity	Status
   MSG-15	#139136 (filed as a feature): in /topic mode a root-DM message no longer opens a fresh topic (v0.21 → v0.22, via the lobby-recovery rule).	—	🟢 decided and fixed: #141347 (6a67be16f9e), salvages #139188 (@KoNit-K). Opt-in telegram.extra.dm_topic_auto_transfer (off by default, read per bot): a root-DM message opens a fresh topic and the reply lands there unquoted; a failed topic create falls back to lobby recovery. Live retest pending: a real bot with Threaded Mode.

   Cron, Kanban and managed workers

   ID	Finding	Severity	Status
   KB-2	#139029: managed workers die with WorkerPersistenceError: synchronous_rpc_on_event_loop; a worker path reaches the synchronous persistence RPC from the event loop. The guard is correct; the caller moves to to_thread.	broken	🟢 fixed on main: #141334 (7e0966ace4f). Activity, label, aux-usage and async-delegation ledger writes reached from Relay callbacks run off its event loop; the guard is unchanged. Live retest pending: a Kanban managed worker on a Relay profile with a non-streaming provider.

   Dashboard and remote Desktop mutations

   ID	Finding	Severity	Status
   DB-7	#140455 (Desktop in URL, Cloud or SSH mode): a waiting approval or clarify prompt is lost after a reconnect: session.events.since is refused with replay_epoch: Extra inputs are not permitted by the standalone hermes serve contract.	broken	🟢 fixed on main: #141331 (fa2836d1ee9), salvages #140456 (@dokterdok). Live-retested by @dokterdok against a real hermes serve with the shared client in all four old/new pairs.
10. teknium1 commented on Oct 19, 2026

    

    teknium1

    on Oct 19, 2026

    Last edited by teknium1

    Collaborator

    More actions

    Package 13: the remaining open rows

    There are 34 open rows on the tracker. 14 of them are already covered by @kshitijk4poor's package 1 salvages:

    PR	Rows
    #140675	the /resume / /sessions switch subscribe (package 1 lane 1; no tracker row)
    #140676	CLI-16
    #140677	CLI-12
    #140678	CLI-30, CLI-31
    #141090	CLI-11, CLI-14
    #141091	CLI-10
    #141092	CLI-19
    #141093	CLI-22
    #141094	CLI-26
    #141095	CLI-27
    #141096	CLI-25
    #141105	T1
    #141110	CLI-15

    We are taking 17 of the other 20. GW-19, T12 and TUI-15 stay with their open contributor PRs (#140778 by @unsupportedpastels, #140382 by @yoniebans) and are not part of this package. Each lane reproduces its rows on main first. Each outside PR listed is reviewed and salvaged with cherry-pick -x if it's right, so its author keeps the commit.

    Lane	Rows	Salvage candidate
    Settlement notices	GW-20, GW-21, GW-22	none
    Browser scope edges	GW-23	none
    Classic CLI /queue	N5	none
    Classic CLI resume key	CLI-34	#140348 (@dokterdok)
    Session lists	TUI-8, RT-18	#139675 (@Hamakaze1s), #139031 (@KoNit-K)
    Live registry and deletes	TUI-21, TUI-19	#138817 (@686f6c61)
    Ink leftovers	TUI-17, N8, TUI-26	none
    Desktop repeated /undo / /retry after a lost reply	RT-40	#140465 (@rodricksz4h5)
    Windows Store ticket bridge	RT-21 (with RT-10's function)	#139174 (@austinpickett)
    Provider reorder after a pick	MP-17	none
    Kanban lost turns	KB-8	none

    Update, Oct 19 evening: package 13 is merged. The 14 rows it fixed are listed in Fixed on main: package 13. TUI-8, RT-18 and TUI-19 moved to #141573, and GW-19, T12 and TUI-15 stayed with their contributor PRs.
11. yoniebans commented on Oct 19, 2026

    

    yoniebans

    on Oct 19, 2026

    More actions

    T12 and TUI-15 fixed on main (package #139161, lane 5): #140382 (b4170e73b0), reviewed by @teknium1 and @kshitijk4poor. Rows for the Fixed on main table:

    Ink TUI and local Desktop

    ID	Finding	Severity	Status
    TUI-15	Desktop "edit after interrupt" becomes a plain resubmit (session.history refused).	degraded	🟢 fixed on main: #140382 (b4170e73b0); the shared gateway serves session.history with row ids
    T12	Desktop /undo and /retry during a running turn show the busy refusal instead of interrupting first.	degraded	🟢 fixed on main: #140382 (b4170e73b0); still refused while a reply runs, as before #106742, now with the stop-first text on Desktop and the Ink TUI, or the wait-for-compress text when another writer holds the compression lock

    The row text "instead of interrupting first" describes a behaviour main never had before #106742: the old backend also refused, with guidance. The fix restores that guidance rather than adding an interrupt.
12. yuzilongleif-collab commented on Oct 19, 2026

    

    yuzilongleif-collab

    on Oct 19, 2026

    More actions

    Managed Kanban construction failure: preserve the cause without replaying an unknown admission

    Reporting on this tracker as requested, with a small independent candidate rather than another overlapping log-boundary implementation.

    Fixed source and observed behavior

    Reproduced on official main at d4c6414728fda599e5fe886b89edf6b79ea0af84, Linux, Python 3.14.7, pytest 9.1.1, against isolated homes and a temporary SQLite board. All provider names and credential strings used in these tests are synthetic.

    The relevant chain is:

    1. agent/managed_worker.py:426–437 catches an agent-construction/resolution exception and calls record_start_failure.
    2. gateway/session_kanban.py:275–282 records the credential exit code but an empty last_output, so the board result loses the cause.
    3. hermes_cli/kanban_worker_client.py:75–85 raises on an unknown receipt before reading the exact attempt's result. A synthetic construction exception therefore never reaches that submitter's output.

    This is source-bound construction/receipt-path evidence, not a claim that every pre-first-turn failure shares this cause. In particular, a missing optional card skill is resolved after construction, in _run_task_turns; this patch does not repair that failure path.

    Minimal candidate

    • Render through the existing provider error formatter; choose the first nonempty line.
    • Force the existing secret redactor over the complete prefixed line before keeping its first 500 characters. This retains the leading cause and avoids splitting a secret before redaction.
    • On unknown, best-effort print only worker_result for the same task/run/claim; then raise the unchanged Kanban execution unknown; retained by owner error.
    • No second kanban.run, replay, automatic unblock, receipt settlement, state-machine change, credential/provider rerouting or retry-budget change. Existing exits 75 / 78 / 1 remain intact.

    The formatter's catch-all is intentional at this error-reporting boundary: a formatter failure must neither replace the startup failure nor log raw exception text. A fallback emits the exception class. The patch carries the repository's narrowly explained BLE001 comment waiver.

    Scope relative to current work

    • fix(gateway): key-less managed workers resolve their route key and keep one owner connection #139472 is merged and supplies the construction/credential-resolution boundary; it does not persist the cause text or read it on unknown.
    • Kanban cards accept skills the assignee profile cannot resolve — dispatched worker crash-loops at boot (fix ready) #140436 / fix(kanban): refuse card skills the assignee profile cannot resolve #140992 / fix(kanban): refuse slash-joined task skills at creation; degrade worker boot when skills don't resolve #139976 own unresolved card-skill authoring/preload behavior; all were open at the implementation/review checks. Their exact _run_task_turns delta is separate and is not duplicated here.
    • kanban: a worker that dies with a nonzero exit leaves no stderr anywhere — startup crashes are undiagnosable #88457 / fix(kanban): preserve stderr when workers crash during startup #88486 and fix(kanban): bound dead-worker diagnostic to the current invocation's log #129663 concern dispatcher stderr/log-boundary capture and attribution. This candidate does not change those modules or include fix(kanban): bound dead-worker diagnostic to the current invocation's log #129663's implementation. It only makes a construction cause available to the exact-attempt result and submitter output.
    • Preserve Kanban provider failures and enforce bounded retry recovery #114143 reports/classifies failed turn results and adjusts recovery budgets; it does not change this managed construction/result-reader pair.
    • [Feature]: Host-side persistence failures ('execution unknown; retained by owner' / owner_unavailable) burn the card retry budget — classify like the OWNER_WAIT_S host conditions #139612 concerns infrastructure-failure accounting; this candidate leaves that policy unchanged.

    Exact-symbol and semantic searches covered open and closed issues/PRs. No exact indexed implementation match for this bounded construction/result-reader delta was found; this is not proof that no unindexed branch exists. The tracker remains the appropriate reporting owner. The independent review checked later main b4170e73b0bbd74b47dc567c36166d749c069699, whose relevant inputs had not changed. The subsequent publication check observed 493208879a3248c6a4b844981d2c31c78ae00929: gateway/session_kanban.py now also has a separate Docker workspace/profile-binding change in _worker_workspace, outside this patch's construction-error/result-recording functions. That later complete head was not executed; the behavioral test results below remain bound to the fixed d4c64147 base.

    Actual tests

    Final identical tests on the pristine fixed base and candidate:

    Selection	Pristine base	Candidate
    test_kanban_start_failure_diagnostic.py	3 failed / 1 passed	4 passed
    test_kanban_start_failure_boundaries.py	9 failed / 4 passed	13 passed
    Existing result/credential exit tests	21 passed	21 passed
    Existing admission-wait tests	2 passed	2 passed

    The real construction wrapper/provider resolver, temporary SQLite persistence and real GatewayClient loop were exercised; transport was a deterministic in-memory fixture, not a live owner/model. Cases cover forced redaction, a synthetic secret crossing the output cut, long CJK first lines, multiline exclusion, exact task/run/claim mismatches, unknown with/without readable results, no replay and unchanged terminal exits.

    The native scripts/run_tests.sh entry was used per file with an explicitly disclosed infrastructure-only adaptation to retain isolated HOME/PM/cache/tmp roots through its environment reset, avoid whole-tree bytecode writes, and place its per-file scratch under the task root. This is not unmodified-runner or CI certification. The full reproduction note and separate harness delta accompany the candidate.

    After export, the final patch was applied to its pristine preimage and all four payload files matched byte-for-byte; its core diagnostic file then passed 4/4 again. Between the 40-case green and export, only the BLE001 explanatory comment changed; AST identity was verified. The 40-case run and final-artifact four-case rerun are separate evidence, not a full-PR/CI claim.

    Limits and proposed next step

    This does not fix old log attribution or promise that the dispatcher's existing 400-character tail preserves every long leading cause. Existing redaction is not universal PII removal. No production data/credentials, live provider requests, native Desktop render, installed-runtime change, hosted CI or full suite was used.

    The implementation runs above are author self-tests. A subsequent independent review fetched the same fixed official base and tested the final exported patch: the identical 17 new regression cases produced 12 failures / 5 passes on pristine production code and 17 passes with the patch; with 23 adjacent controls the exported candidate produced 40 passes, no failures/errors/skips. That review also checked patch replay and public-artifact identity; it reused the explicitly disclosed test-infrastructure adaptation and is not hosted-CI certification. Please consider the bounded patch below for maintainer pickup in this tracker. No competing PR or installed-runtime adoption is requested by this report.

    Complete independent patch (MIT)

    diff --git a/gateway/session_kanban.py b/gateway/session_kanban.py
    index 0eb71e59..d658feb8 100644
    --- a/gateway/session_kanban.py
    +++ b/gateway/session_kanban.py
    @@ -279,7 +279,21 @@ def record_start_failure(frame, exc):
         if context is None:
             return
         from hermes_cli.turn_exit import credential_failure_flags, turn_exit_code
    -    _record_worker_result(context, turn_exit_code(None, kanban_worker=True, **credential_failure_flags(exc)), '')
    +    _record_worker_result(context, turn_exit_code(None, kanban_worker=True, **credential_failure_flags(exc)),
    +                          _start_failure_text(exc))
    +
    +
    +def _start_failure_text(exc):
    +    """The cause a pre-turn death leaves on the board, as the one-shot worker printed it to its log
    +    (the rendered provider/model error, #46593): redacted, first line only, bounded like a reply."""
    +    from agent.redact import redact_sensitive_text
    +    from hermes_cli.runtime_provider import format_runtime_provider_error
    +    try:
    +        text = format_runtime_provider_error(exc)
    +    except Exception:  # health: allow BLE001 -- formatter failure must not replace the startup error or log raw exception text
    +        text = type(exc).__name__
    +    line = next((ln.strip() for ln in str(text).splitlines() if ln.strip()), type(exc).__name__)
    +    return redact_sensitive_text(f'Worker start failed: {line}', force=True)[:500]
     
     
     def _record_worker_result(context, code, last_output):
    diff --git a/hermes_cli/kanban_worker_client.py b/hermes_cli/kanban_worker_client.py
    index 95275888..386cee9b 100644
    --- a/hermes_cli/kanban_worker_client.py
    +++ b/hermes_cli/kanban_worker_client.py
    @@ -8,6 +8,7 @@ except ModuleNotFoundError as exc:
             raise
     
     import asyncio
    +import contextlib
     import os
     from pathlib import Path
     import sys
    @@ -75,9 +76,14 @@ async def _submit_claim(client, params, board_db):
         while receipt['status'] not in {'terminal', 'unknown'}:
             await asyncio.sleep(.2)
             receipt = await client.rpc('prompt.receipt', session_id=sid, admission_id=receipt['admission_id'])
    +    from gateway.session_kanban import worker_result
         if receipt['status'] == 'unknown':
    +        # The exact attempt's own words (e.g. why its agent could not start) still belong in this
    +        # log, where the crash diagnostic quotes them; the unknown verdict itself is unchanged.
    +        with contextlib.suppress(Exception):
    +            if text := worker_result(board_db, params).get('last_output'):
    +                print(text, flush=True)
             raise GatewayClientError('Kanban execution unknown; retained by owner')
    -    from gateway.session_kanban import worker_result
         result = worker_result(board_db, params)
         # The attempt's reply ends the log, as a one-shot worker's did: crash diagnostics quote it.
         if result.get('last_output'):
    diff --git a/tests/gateway/test_kanban_start_failure_boundaries.py b/tests/gateway/test_kanban_start_failure_boundaries.py
    new file mode 100644
    index 00000000..52dcbc9d
    --- /dev/null
    +++ b/tests/gateway/test_kanban_start_failure_boundaries.py
    @@ -0,0 +1,142 @@
    +"""Synthetic, no-network boundaries for managed startup diagnostics."""
    +import asyncio
    +import json
    +import os
    +
    +import pytest
    +
    +from gateway import session_kanban as sk
    +from hermes_cli import kanban_db as kb
    +from hermes_cli import kanban_worker_client as client_module
    +from hermes_cli.auth import AuthError, resolve_provider
    +from hermes_cli.auth_constants import CODEX_RATE_LIMITED_CODE
    +from hermes_cli.gateway_client import GatewayClient, GatewayClientError
    +from hermes_cli.kanban_db_connect import connect_closing
    +
    +
    +def bound(tmp_path, monkeypatch):
    +    monkeypatch.setenv("HERMES_HOME", str(tmp_path))
    +    for key in ("HERMES_KANBAN_TASK", "HERMES_KANBAN_BOARD", "HERMES_KANBAN_DB"):
    +        monkeypatch.delenv(key, raising=False)
    +    kb.init_db()
    +    path = kb.kanban_db_path()
    +    with connect_closing(path) as conn:
    +        task_id = kb.create_task(conn, title="synthetic startup", assignee="default")
    +        task = kb.claim_task(conn, task_id, claimer="fixture:owner")
    +        context = dict(db=str(path), board="default", task_id=task_id,
    +                       run_id=task.current_run_id, claim_lock=task.claim_lock)
    +        with kb.write_txn(conn):
    +            kb._append_event(conn, task_id, "worker_bound",
    +                             dict(pid=os.getpid(), claim_lock=task.claim_lock),
    +                             run_id=task.current_run_id)
    +    return path, context
    +
    +
    +def record(path, context, error):
    +    sk.record_start_failure({"policy": {"kanban_json": json.dumps(context)}}, error)
    +    return sk.worker_result(path, context)
    +
    +
    +def test_construct_agent_real_missing_provider_books_cause(tmp_path, monkeypatch):
    +    from agent.managed_worker import _construct_agent
    +    path, context = bound(tmp_path, monkeypatch)
    +    def factory(**kwargs):
    +        resolve_provider("synthetic-provider-not-installed")
    +    with pytest.raises(AuthError, match="Unknown provider"):
    +        _construct_agent({"policy": {"kanban_json": json.dumps(context)}}, factory, None, api_key="fixture")
    +    result = sk.worker_result(path, context)
    +    assert result["exit_code"] == 1
    +    assert "synthetic-provider-not-installed" in result["last_output"]
    +    assert result["last_output"].startswith("Worker start failed:")
    +
    +
    +def test_long_first_line_preserves_prefix_and_cause(tmp_path, monkeypatch):
    +    path, context = bound(tmp_path, monkeypatch)
    +    text = record(path, context, RuntimeError("SYNTHETIC_CAUSE: " + "界" * 1800 + "\nSECOND_LINE_SENTINEL"))["last_output"]
    +    assert text.startswith("Worker start failed: SYNTHETIC_CAUSE:")
    +    assert len(text) == 500
    +    assert "SECOND_LINE_SENTINEL" not in text
    +
    +
    +def test_force_redaction_before_truncation_even_when_disabled(tmp_path, monkeypatch):
    +    from agent import redact
    +    monkeypatch.setattr(redact, "_REDACT_ENABLED", False)
    +    path, context = bound(tmp_path, monkeypatch)
    +    synthetic = "sk-ant-" + "a" * 40
    +    text = record(path, context, RuntimeError("fixture " + synthetic + "\nSECOND_LINE_SENTINEL"))["last_output"]
    +    assert text.startswith("Worker start failed: fixture ")
    +    assert synthetic not in text
    +    assert "SECOND_LINE_SENTINEL" not in text
    +    assert len(text) <= 500
    +
    +
    +def test_secret_crossing_output_cut_is_redacted_before_cut(tmp_path, monkeypatch):
    +    from agent.redact import redact_sensitive_text
    +    path, context = bound(tmp_path, monkeypatch)
    +    synthetic = "sk-ant-" + "a" * 40
    +    first_line = "x" * 460 + synthetic + "z" * 200
    +    expected = redact_sensitive_text("Worker start failed: " + first_line, force=True)[:500]
    +    text = record(path, context, RuntimeError(first_line + "\nSECOND_LINE_SENTINEL"))["last_output"]
    +    assert text == expected
    +    assert synthetic not in text
    +    assert "SECOND_LINE_SENTINEL" not in text
    +
    +
    +@pytest.mark.parametrize("code,relogin,expected", [("invalid_provider", False, 1), ("unauthorized", True, 78), (CODEX_RATE_LIMITED_CODE, False, 75)])
    +def test_exit_mapping_unchanged(tmp_path, monkeypatch, code, relogin, expected):
    +    path, context = bound(tmp_path, monkeypatch)
    +    result = record(path, context, AuthError("Synthetic failure", code=code, relogin_required=relogin))
    +    assert result["exit_code"] == expected
    +    assert result["last_output"].startswith("Worker start failed:")
    +
    +
    +class Owner:
    +    def __init__(self, initial="unknown", final="unknown"):
    +        self.queue = asyncio.Queue()
    +        self.sent = []
    +        self.initial, self.final = initial, final
    +    async def send(self, raw):
    +        req = json.loads(raw)
    +        self.sent.append(req)
    +        receipt = {"status": self.initial if req["method"] == "kanban.run" else self.final, "admission_id": "a_fixture"}
    +        result = {"session_id": "s_fixture", "receipt": receipt} if req["method"] == "kanban.run" else receipt
    +        await self.queue.put(json.dumps({"id": req["id"], "result": result}))
    +    def __aiter__(self):
    +        return self
    +    async def __anext__(self):
    +        return await self.queue.get()
    +
    +
    +@pytest.mark.asyncio
    +@pytest.mark.parametrize("wrong", [None, "claim_lock", "run_id", "task_id"])
    +async def test_unknown_uses_exact_attempt_and_never_replays(tmp_path, monkeypatch, capsys, wrong):
    +    path, context = bound(tmp_path, monkeypatch)
    +    record(path, context, RuntimeError("CURRENT_SYNTHETIC_CAUSE"))
    +    params = {k: context[k] for k in ("board", "task_id", "run_id", "claim_lock")}
    +    if wrong:
    +        params[wrong] = params[wrong] + 1 if wrong == "run_id" else "other-fixture"
    +    owner = Owner(initial="queued")
    +    async with GatewayClient(owner) as client:
    +        with pytest.raises(GatewayClientError, match="execution unknown; retained by owner"):
    +            await client_module._submit_claim(client, params, path)
    +    assert [r["method"] for r in owner.sent] == ["kanban.run", "prompt.receipt"]
    +    text = capsys.readouterr().out
    +    assert ("CURRENT_SYNTHETIC_CAUSE" in text) == (wrong is None)
    +    with connect_closing(path) as conn:
    +        assert kb.get_task(conn, context["task_id"]).status == "running"
    +
    +
    +@pytest.mark.asyncio
    +async def test_terminal_prints_same_attempt_without_changing_exit(tmp_path, monkeypatch, capsys):
    +    path, context = bound(tmp_path, monkeypatch)
    +    record(path, context, RuntimeError("CURRENT_SYNTHETIC_CAUSE"))
    +    owner = Owner(initial="terminal")
    +    async with GatewayClient(owner) as client:
    +        assert await client_module._submit_claim(client, context, path) == 1
    +    assert "CURRENT_SYNTHETIC_CAUSE" in capsys.readouterr().out
    +    assert [r["method"] for r in owner.sent] == ["kanban.run"]
    +
    +
    +def test_non_kanban_start_failure_does_not_write(monkeypatch):
    +    monkeypatch.setattr(sk, "_record_worker_result", lambda *args: pytest.fail("unexpected write"))
    +    sk.record_start_failure({"policy": {}}, RuntimeError("synthetic"))
    diff --git a/tests/gateway/test_kanban_start_failure_diagnostic.py b/tests/gateway/test_kanban_start_failure_diagnostic.py
    new file mode 100644
    index 00000000..c42c4e79
    --- /dev/null
    +++ b/tests/gateway/test_kanban_start_failure_diagnostic.py
    @@ -0,0 +1,103 @@
    +"""A managed Kanban worker that dies before any turn leaves its cause where the board reads it.
    +
    +Agent construction (provider/credential resolution) runs inside the owner-managed worker. The
    +one-shot worker printed that error into the per-task log, which the dead-worker sweep quotes as
    +``worker_output``; the managed path booked only an exit code, and the submitter raised on the
    +``unknown`` receipt before reading the attempt's result, so the board showed "execution unknown"
    +plus whatever older text happened to sit in the append-mode log.
    +"""
    +import asyncio
    +import json
    +import os
    +
    +import pytest
    +
    +from gateway import session_kanban
    +from hermes_cli import kanban_db as kb
    +from hermes_cli import kanban_worker_client
    +from hermes_cli.gateway_client import GatewayClient, GatewayClientError
    +from hermes_cli.kanban_db_connect import connect_closing
    +
    +
    +def _bound_claim(tmp_path, monkeypatch):
    +    monkeypatch.setenv("HERMES_HOME", str(tmp_path))
    +    monkeypatch.delenv("HERMES_KANBAN_TASK", raising=False)
    +    kb.init_db()
    +    path = kb.kanban_db_path()
    +    with connect_closing(path) as conn:
    +        task_id = kb.create_task(conn, title="review fixture", assignee="default")
    +        task = kb.claim_task(conn, task_id, claimer="private-host:owner")
    +        assert task is not None
    +        context = {"db": str(path), "task_id": task_id, "board": "default",
    +                   "run_id": task.current_run_id, "claim_lock": task.claim_lock}
    +        with kb.write_txn(conn):
    +            kb._append_event(conn, task_id, "worker_bound",
    +                             {"pid": os.getpid(), "claim_lock": task.claim_lock},
    +                             run_id=task.current_run_id)
    +    return path, context
    +
    +
    +def _unknown_provider():
    +    from hermes_cli.auth import AuthError
    +    return AuthError("Unknown provider 'plugin-only-provider'.\n\nCheck 'hermes model' for available providers.",
    +                     code="invalid_provider")
    +
    +
    +def test_start_failure_books_its_cause_with_the_exit(tmp_path, monkeypatch):
    +    path, context = _bound_claim(tmp_path, monkeypatch)
    +    session_kanban.record_start_failure({"policy": {"kanban_json": json.dumps(context)}}, _unknown_provider())
    +    result = session_kanban.worker_result(path, context)
    +    assert result["exit_code"] == 1  # unchanged: not a re-login, not a quota wall
    +    assert result["last_output"] == "Worker start failed: Unknown provider 'plugin-only-provider'."
    +
    +
    +def test_start_failure_text_is_redacted_and_bounded(tmp_path, monkeypatch):
    +    path, context = _bound_claim(tmp_path, monkeypatch)
    +    secret = "sk-ant-" + "a" * 40
    +    session_kanban.record_start_failure({"policy": {"kanban_json": json.dumps(context)}},
    +                                        RuntimeError(f"bad key {secret} " + "x" * 2000))
    +    text = session_kanban.worker_result(path, context)["last_output"]
    +    assert secret not in text and len(text) <= 500
    +    assert text.startswith("Worker start failed: bad key ")
    +
    +
    +class _UnknownOwner:
    +    """Admits, then reports the admission ``unknown`` (the managed worker died before ``ready``)."""
    +
    +    def __init__(self):
    +        self.inbox, self.sent = asyncio.Queue(), []
    +
    +    async def send(self, raw):
    +        req = json.loads(raw)
    +        self.sent.append(req)
    +        await self.inbox.put(json.dumps({"id": req["id"], "result": {
    +            "session_id": "s1", "receipt": {"status": "unknown", "admission_id": "a1"}}}))
    +
    +    def __aiter__(self):
    +        return self
    +
    +    async def __anext__(self):
    +        return await self.inbox.get()
    +
    +
    +@pytest.mark.asyncio
    +async def test_unknown_receipt_still_prints_the_attempts_own_result(monkeypatch, tmp_path, capsys):
    +    monkeypatch.setattr(session_kanban, "worker_result",
    +                        lambda db, params: {"exit_code": 1, "last_output": "Worker start failed: Unknown provider 'x'."})
    +    params = {"board": "b", "task_id": "t1", "run_id": 1, "claim_lock": "lock"}
    +    async with GatewayClient(_UnknownOwner()) as client:
    +        with pytest.raises(GatewayClientError, match="execution unknown; retained by owner"):
    +            await kanban_worker_client._submit_claim(client, params, str(tmp_path / "kanban.db"))
    +    assert capsys.readouterr().out.strip() == "Worker start failed: Unknown provider 'x'."
    +
    +
    +@pytest.mark.asyncio
    +async def test_unknown_receipt_without_a_result_keeps_the_verdict(monkeypatch, tmp_path, capsys):
    +    def unreadable(db, params):
    +        raise OSError("board unavailable")
    +    monkeypatch.setattr(session_kanban, "worker_result", unreadable)
    +    params = {"board": "b", "task_id": "t1", "run_id": 1, "claim_lock": "lock"}
    +    async with GatewayClient(_UnknownOwner()) as client:
    +        with pytest.raises(GatewayClientError, match="execution unknown; retained by owner"):
    +            await kanban_worker_client._submit_claim(client, params, str(tmp_path / "kanban.db"))
    +    assert capsys.readouterr().out == ""
13. DarkoKuzmanovic commented on Oct 19, 2026

    

    DarkoKuzmanovic

    on Oct 19, 2026

    More actions

    Ink TUI process dock: every background process shows 0s · starting on the shared gateway

    Checked against main at 6e9b3e3b88 (running 5fc490e2b3). Surface: Ink TUI on the local shared gateway, Linux, single profile. I searched the open issues, PRs and this thread for started_epoch, uptime_seconds and "0s · starting" and found no existing report. #135062 covers the same dock but a different problem (ANSI in output_preview).

    Symptom: each row in the Processes dock reads ⚙ <command> · 0s · starting for the whole run. In my case a heartbeat had been running for over 4 hours (checked with ps) and still showed 0s. Rows of exited processes also show the wrong runtime.

    Cause: the client and the gateway disagree on field names.

    • gateway/session_ancillary.py process_snapshot() (the process.list handler on the shared gateway since bf94644) sends started_epoch, exited_at and output_tail.
    • ui-tui/src/app/processRoster.ts buildProcessRows() still reads uptime_seconds and output_preview, the fields the old tui_gateway _session_processes() / list_sessions() row carried. On the new payload both are undefined, so elapsedSeconds comes out as 0, and processVerdict() falls back to 'starting' because no output line is found.
    • The ProcessEntry contract in tui_gateway/contracts/tools_commands.py lists both sets of fields, so neither side's types catch the mismatch. The classic CLI dock reads the new fields and is not affected.

    Fix (running locally, about 4 lines): in buildProcessRows(), compute the runtime from started_epoch when present, as (exited_at || now) - started_epoch; this also keeps it ticking between polls. Fall back to output_tail for the last-line detail. Then add started_epoch and output_tail to the ProcessEntry interface. Test:

    const [live, done] = buildProcessRows([
      { session_id: 'gw_run', command: 'make', status: 'running', started_epoch: NOW - 300, output_tail: 'step 3\n' },
      { session_id: 'gw_done', command: 'make', status: 'exited', exit_code: 0, started_epoch: NOW - 50, exited_at: NOW - 10 }
    ], NOW * 1000)
    expect(live).toMatchObject({ elapsedSeconds: 300, detail: 'last: step 3' })
    expect(done).toMatchObject({ elapsedSeconds: 40, detail: 'exit 0 · 10s ago' })

    The other option is for process_snapshot() to also send uptime_seconds and output_preview. The client-side fix seems better because the runtime then counts up between polls, but either one works.
14. 686f6c61 commented on Oct 19, 2026

    

    686f6c61

    on Oct 19, 2026

    More actions

    Diagnosis matches main at 6e9b3e3b88: gateway/session_ancillary.py::process_snapshot() (process.list on the shared gateway) sends started_epoch, exited_at and output_tail, while ui-tui/src/app/processRoster.ts::buildProcessRows() still reads uptime_seconds and output_preview. Both come out undefined on the new payload, so elapsedSeconds is 0 and processVerdict() falls back to starting.

    Fix pushed as fix/tui-process-dock-elapsed (98267ccb), four lines in buildProcessRows() plus a test:

    • elapsedSeconds comes from started_epoch when it is present: (exited_at || now) - started_epoch, floored; the old uptime_seconds - sinceExitSeconds stays as the fallback for the session-local registry row, which is what the existing dock tests pin.
    • The last-line detail reads output_tail first and falls back to output_preview.
    • ProcessEntry declares both new fields. tui_gateway/contracts/tools_commands.py already lists them on both sides, so the contract file does not change.

    Client-side over the server-side alternative for the reason you gave: recomputing from started_epoch keeps the runtime advancing between the 1.5 s process.list polls instead of freezing at a value the server computed at poll time.

    Verification on that head: npx vitest run src/__tests__/processDock.test.tsx → 3 passed, and the new case fails without the source change (1 failed | 2 passed). npx tsc --noEmit -p tsconfig.json clean. The neighbouring dock tests (agentsDock, agentsCompact, agentsHydration, agentRoster) → 8 passed.

    Branch: https://github.com/686f6c61/hermes-agent/tree/fix/tui-process-dock-elapsed — gh pr create --base main --head 686f6c61:fix/tui-process-dock-elapsed --title "fix(tui): read the shared gateway's process row in the processes dock". My account is at the open-PR ceiling, so CreatePullRequest is refused even with --draft; the branch is the deliverable until there is a slot.
15. Sahilvishnaliya commented on Oct 19, 2026

    

    Sahilvishnaliya

    on Oct 19, 2026

    More actions

    F5 (RPCs past the frame cap) — candidate fix in #141707 (hermes_cli/gateway_client.py).

    Mechanism, for the record: the client delivers replies from its single reader loop, so an answer is only reachable while the reader reads. _read_on() stopped that reader at EVENT_OVERFLOW_FRAMES (16,384) / EVENT_OVERFLOW_BYTES (8 MiB) with an RPC pending — and the pending RPC is usually the one awaited by the task that drains events (session.resume after a gap, prompt.receipt). Because the owner attaches the re-attached subscription before it answers, a live turn's frames sit ahead of the answer; past the cap the reader parked for the consumer that was waiting on that answer, the 30 s stall budget expired, and the RPC failed as Gateway disconnected; turn outcome is unknown — a healthy socket and a still-running turn.

    #141707 keeps the reader reading toward the answer and declares what it can no longer hold: one session.replay_gap per session behind the held frames (the same notice the fanout sends on overflow, already handled by CLI and ACP by re-attaching and replaying from the store). Bounded at cap + one notice; a consumer that stalls with nothing awaited is still failed and bounded, unchanged.

    Distinct from #140388/CLI-35 (fixed by #140968 — the owner retiring a slow viewer's subscription, server → client), #132546 (LONG-handler pool saturation) and #141422 (sync compaction inside session.resume), and no open PR touches EVENT_OVERFLOW_FRAMES / EVENT_OVERFLOW_BYTES.

    Note on severity: this fails an in-flight RPC with a misleading reason and a resume workaround; the turn keeps running and persists, so I'd call it P1 degraded-RPC rather than P0-class, even though it was listed under this P0 tracker.
16. teknium1 commented on Oct 19, 2026

    

    teknium1

    on Oct 19, 2026

    Collaborator

    More actions

    Fixed on main: package 13

    The rows package 13 took (plan) and merged. Each PR went through an independent review round, Greptile, and the round-2 reviews from @unsupportedpastels and @dokterdok before merge. Every fix has a test that fails on the merge base and passes on the PR. Live retests that are still owed are listed in each PR body.

    ID	Fix	Merged
    GW-20	An automation turn that a gateway stop interrupts gets the neutral automation notice, never "Send your message again". If the agent already wrote its own failed-turn notice for that turn, the restart notice replaces it instead of stacking.	#141495 (53a32851071)
    GW-21	The restart/shutdown reason is also written to the transcript, in the settlement transaction and exactly once across retries, so a viewer that reconnects after the restart sees it.	#141495 (53a32851071)
    GW-22	A shutdown Stop that lands before a managed worker spawns, including during @ reference expansion, is latched: the worker never starts and the interrupted result carries the restart reason. A user Stop leaves no cause.	#141495 (53a32851071)
    GW-23	Per-profile browser scope edges. An unscoped thread on a multiplexed gateway resolves no live CDP endpoint. On macOS and Linux, a too-long per-profile harness socket dir moves to a checked private hbh-<hash> dir, measured on the resolved path. Each profile gets its own BH_TMP_DIR. A /browser connect during session creation or during browser_exec routing reroutes or reloads, so the old browser is never kept or handed out.	#141493 (650cbf30142)
    CLI-34	The classic CLI sends its launch key again on resume after a gateway restart. A keyed resume falls back to plain only on admission_conflict / permission_denied. A typed --api-key is held to the startup resume only. A cancelled reconnect cancels its session.resume (no 25 s exit hang). Salvages #140348 plus two fix commits by @dokterdok.	#141481 (3ea5a08fa19)
    N5	Classic CLI /queue while a turn runs, and a text-only queue-mode Enter, go into the gateway's durable FIFO. The prompt is adopted by admission id, survives an owner restart, runs once, and is painted after a reconnect. A slash command typed after it waits (bounded) until it starts. list / rm / edit / clear act on the session's shared queue; move explains it can't reorder owner-held rows.	#141501 (c0015464f0f)
    MP-17	A picked legacy custom_providers entry is recovered on cold start by its canonical name, so a reorder survives. An in-place rename (same index and URL) runs as on main. A deleted entry still refuses and never borrows another entry's key.	#141484 (7df8c462b8e)
    KB-8	Kanban card admissions carry a dispatcher-only marker, so a card attempt fenced unknown across owner epochs settles interrupted and is never replayed. Clients can't set the marker, and rows minted without it keep today's pause.	#141488 (e3277f17f30)
    N8	hermes --tui -w removes a clean worktree on quit again. A tree is kept when it is dirty, has unpushed commits, has a pending turn, has active /loop, heartbeat, goal or bound cron work, or another viewer still holds the session. A resume (--tui --resume, -c, classic --resume) recreates a missing launcher-recorded tree at the same path, so its turns run.	#141508 (f4ac0bea1ea)
    TUI-26	A typed stop phrase in a LOCAL voice chat on the shared gateway ends the voice chat. This works through prompt.submit and through busy steer/redirect, and a retry replays the stop. The new {voice_stopped: true} reply is handled by Ink, Desktop (both paths), ACP, the classic CLI, chat --attach and the sidecar forward. Messaging lanes are untouched.	#141508 (f4ac0bea1ea)
    TUI-17	Goal cards repaint the moment a tool changes the goal mid-turn (CLI, Ink, Desktop), with a per-session publish lock. Carries #139294 (co-author @patrykkopycinski, from #115779).	#141508 (f4ac0bea1ea)
    RT-40	A repeated Desktop /undo or /retry after a lost reply replays the same rewind, including the full cut of a lost /undo N and on named profiles after a redial. A refused rewind is dropped, not resent. Salvages #140465 (@rodricksz4h5).	#141492 (c9dce2c26f8)
    RT-21	Microsoft Store / bundled installs mint gateway tickets with the payload's own Python instead of the hermes.exe shim, and RT-10's remaining spaced .cmd override is quoted. The Windows attach E2E is green, but it doesn't stage a bundled payload (see CI-2, added below). Salvages #139174 (@KoNit-K, @austinpickett).	#141490 (afe27560b14)
    TUI-21	Needed no code: main already answers not_found for a live entry whose row is gone (0bf1ed99ce3, @dokterdok). #138817's test file passes on main and fails with the reported TypeError without that guard. The 30 s session.resume timeouts in the same Windows report are a different symptom. #138817 is closed as superseded.	0bf1ed99ce3

    Still open from package 13's scope: TUI-8, RT-18 and TUI-19, consolidated into @yoniebans's #141573. #141491 and #141498 are closed as superseded, and their closing notes list the commits to carry over. GW-19, T12 and TUI-15 stayed with their open contributor PRs (#140778, #140382).
17. sinisa222 commented on Oct 19, 2026

    

    sinisa222

    on Oct 19, 2026

    Last edited by sinisa222

    More actions

    Following up on the triage of #141163 (and its earlier, narrower duplicate #140913) into the "Gateway lifecycle and restart" section — it doesn't have a row ID there yet, so here are the build details and two further findings from the same machine.

    Build

    • main SHA: ccc6956350 (hermes --version: v0.22.2+1545.gccc6956 · stable), git install
    • Desktop: built from the same checkout (apps/desktop/release/mac-arm64/Hermes.app, bundle version 0.0.0)
    • macOS 26.5.2, Apple Silicon (M2 Max, 96 GB), Desktop (local), single profile plus six named profiles
    • Local runtime: llama.cpp router on 18434, Metal engine b11671 (the lockfile pin)

    Core problem (#141163) is unchanged on this SHA: after an update the Desktop app doesn't respawn its hermes serve backend, so the router it owns never starts. The workaround is still nohup hermes serve --skip-build & from a terminal; it doesn't work from a launchd agent (the router gets SIGINT right after binding).

    ---

    New 1 ‚Äî hermes update pins a new engine but never installs it

    After the update pm/lock.json pinned llamacpp-metal 11671, but binaries.installed_engine() kept returning b10964 (allow_outdated=True), and the router kept running on it. Nothing told me. It only surfaced because a catalog model with min_engine: b11370 was greyed out in Local Models with "update the engine first". Expected: the update installs the pinned engine, or says plainly that the running engine is older than the pin.

    New 2 — catalog MTP flag applied to a GGUF that has no MTP layers → model fails to load on b11671

    Qwen3.6-35B-A3B-UD-Q4_K_M.gguf from unsloth/Qwen3.6-35B-A3B-GGUF (22,134,528,992 bytes, qwen35moe.block_count = 40, no nextn_predict_layers key) has the same filename as the catalog variant, which comes from unsloth/Qwen3.6-35B-A3B-MTP-GGUF (22,663,387,424 bytes). find_entry_for_model() matches on the stem alone, so _runs_mtp() (presets.py:77) returns true and the preset gets spec-type = draft-mtp. On b10964 this was silently ignored; on b11671 the model refuses to start:

    W llama_init_from_model: context type MTP requested but model doesn't contain MTP layers
    E common_speculative_init_result: failed to create MTP context
    E srv    load_model: failed to create MTP context
    E srv  llama_server: exiting due to model loading error
    

    With the model as fallback_providers[1], this means the fallback chain itself fails right after the engine update. Expected: confirm the MTP layers from the GGUF header (or match the catalog on size/hash, not filename) before adding draft-mtp. Workaround: override spec-type = none after every preset regeneration.

    Related, smaller: there is no supported, durable way to cap a local model's window. window_overrides.json can only raise it (context_policy.py:153, target > decision.window), and presets.ini is rebuilt on every backend start. K2-Horizon-MoVA-36B-A4B was given its full 524,288-token training window and took 71.6 GiB RSS, which pushed this 96 GB machine to 7 % free and nearly full swap.
18. yoniebans commented on Oct 19, 2026

    

    yoniebans

    on Oct 19, 2026

    More actions

    TUI-19, TUI-8 and RT-18 fixed on main; TUI-21 already fixed (package #139161, lane 3): #141573 (cc6b1f299b), reviewed by @unsupportedpastels and @teknium1. Rows for the Fixed on main table:

    Ink TUI and local Desktop

    ID	Finding	Severity	Status
    TUI-8	Ink /resume hides stored messaging and cron sessions; resuming one → not_found.	degraded	🟢 fixed on main: #141573 (cc6b1f299b) lists them; resume landed earlier as 79f8d1f86c (@Hamakaze1s, #139675)
    TUI-19	A refused delete in the Ink picker (stale revision, read-only grant) reopens the closed session: the CAS-read session.resume attaches a viewer and nothing detaches it. Classic CLI /sessions delete has the same gap.	degraded	🟢 fixed on main: #141573 (cc6b1f299b); both clients hand the viewer back on refusal
    TUI-21	#138794: once hermes sessions delete removes a session that is still in the live registry, session.list raises TypeError in session_authority._handle.	broken	🟢 fixed on main: #140971 (0bf1ed99ce); diagnosis and regression probe by @686f6c61 (#138817)
    RT-18	#139017: Desktop treats messaging-channel sessions (WeChat) as local: their completions raise unread badges, and opening one fails with Session controls unavailable: stale_generation.	degraded	🟢 fixed on main: #141573 (cc6b1f299b), with @KoNit-K's Desktop commits from #139031

    TUI-21 does not reproduce at 8f5693af41: delete a live TUI session with hermes sessions delete <id> --yes, then session.list and session.active_list both answer and errors.log has no TypeError.

    RT-18 restores the control read as main had it before #106742: a messaging session's goal, loop and heartbeat stay readable after its chat moves on. A message sent from Desktop into a messaging session still lights the unread dot; replies from the chat itself do not.