Skip to content

Relay-mesh agents silently stop replying if mesh runtime drops after launch (no self-heal) #2062

Description

@micspiral

Summary

Relay-mesh (Buzz shared compute) agents silently stop replying if the mesh runtime drops at any point after app launch. No error surfaces to the user and nothing re-arms the runtime until an agent is manually (re)started.

Repro

  1. Launch desktop with a relay-mesh agent; mesh runtime comes up, ingress on 127.0.0.1:9337, bot replies fine.
  2. Leave the app running for an extended period (overnight), or otherwise have the embedded mesh runtime exit/wedge.
  3. Send the bot a message → no reply, no error.
  4. Manually start/create an agent → ensure_relay_mesh_for_record runs again, re-arms 9337, replies resume.

Observed live: 9337/3131 NOT listening while goose acp agent subprocesses were still running and healthy; curl 127.0.0.1:9337/v1/models → connection refused. A remote peer (M4) was serving the model the whole time, so compute availability was never the issue.

Root cause

ensure_relay_mesh_for_record (which brings up the local mesh client node / OpenAI ingress) is only invoked:

  • at launch — managed_agents/restore.rs (restore_managed_agents_on_launch)
  • on agent create/start — commands/agents.rs (start_local_agent_with_preflight)

There is no periodic re-arm and no liveness check tying the mesh runtime to agent message dispatch. If state.mesh_llm_runtime becomes None (runtime exited, or a bad restart left a zombie — cf. "timed out after 5s waiting for embedded mesh runtime thread to exit"), relay-mesh agents keep running but point at a dead ingress. The failure is invisible.

Impact

Any long-lived desktop session using shared-compute agents will eventually see bots go quiet with no indication why. This is the dominant "shared compute is broken" user-facing symptom even though discovery, transport, and remote serving are all healthy.

Suggested fix (options)

  1. On-demand re-arm: before dispatching a message to a relay-mesh agent (or in the agent reconcile loop), if mesh_llm_runtime.is_none() or the ingress is unreachable, call ensure_relay_mesh_for_record / ensure_client_node_for_model to bring it back.
  2. Liveness + surfaced error: health-check the runtime periodically; if it's down and can't be restored (e.g. no live serving member), surface a visible "shared compute offline" state on the agent instead of silent non-reply.
  3. Zombie guard: the mesh runtime thread to exit 5s timeout can leave a half-dead runtime — ensure a failed restart doesn't leave mesh_llm_runtime in a state that blocks re-arm.

Fix (1) alone resolves the reported symptom; (2) makes it debuggable; (3) prevents the wedge that motivated it.

Not this

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions