Linux: `service install` stops the agent and does not start it — watchdog did not recover (remote host goes offline)

[SemoTech]

Summary

On an already-enrolled Ubuntu 24.04 host, breeze-agent service install stopped breeze-agent.service and left it inactive. The running watchdog logged an IPC EOF and did not bring the agent back. The device went Offline until someone ran systemctl start breeze-agent out of band.

That is a stranding risk for any remote Linux box where the only management path is the agent (no SSH).

What we ran

Self-host API/web v0.110.0, host was already enrolled on 0.104.0. We staged the v0.110.0 Linux amd64 agent + watchdog binaries into /usr/local/bin/, then ran:

breeze-agent service install

BRT dropped (Device is not online). On the box immediately after:

• /usr/local/bin/breeze-agent printed v0.110.0
• breeze-agent.service = inactive (dead) — last systemctl stop at the moment of service install
• breeze-watchdog.service still active, but still the old 0.104.0 process (binary on disk replaced, process not restarted)
• Watchdog log: watchdog ipc: read loop error: ipc: read header: EOF
• No subsequent systemctl start from watchdog in the next ~minute

systemctl start breeze-agent + systemctl restart breeze-watchdog recovered the host (Online, agent 0.110.0).

Code (v0.110.0)

agent/internal/agentapp/service_cmd_linux.go service install:

1. systemctl stop breeze-agent (comment: “safe for upgrades”)
2. rewrite unit, daemon-reload, enable
3. print “installed and enabled”
4. if enrolled and not running → print “Next steps: sudo breeze-agent service start”
5. never systemctl start

breeze-watchdog service install is the same shape (stop + enable, “Start with: sudo breeze-watchdog service start”).

Docs still say re-running service install “reinstalls the agent binary and restarts the agent service (safe for upgrades)”. That restart does not happen.

Why watchdog did not save us

A clean systemctl stop is a successful exit. Watchdog saw IPC EOF and stayed put. It did not treat “unit enabled but inactive” as a recover-and-start condition — at least on the 0.104.0 watchdog process that was still running.

So: install stops the agent → remote session dies → watchdog does not start it → box is offline until console/SSH.

Expected

• If the host is already enrolled, service install should start (or restart) the agent before returning.
• Watchdog should recover an enabled-but-inactive agent after an upgrade/stop, not only after a crash.
• Official auto-update (promote / watchdog apply) must not share this “stop and wait for a human” path.

Ask

Please confirm the in-band agent auto-update path on Linux always starts the new agent without a human. If service install is also a supported upgrade method (docs say it is), it should start the service when enrolled.

We can retest a Linux canary after a fix. We will not promote the fleet until this stranding hole is closed.

[ToddHebebrand]

Confirmed — your read is exact, and the docs telling you this was safe are wrong.

service install never starts the unit: agent/internal/agentapp/service_cmd_linux.go:148-154 runs systemctl stop breeze-agent ("safe for upgrades" in the comment), then rewrites the unit, daemon-reload, enable (:178-200), and at :229 checks isSystemServiceRunning() — which is false because of its own stop — so it prints "Next steps: 1. Start" (:243-248) and exits. There is no systemctl start anywhere in that command. It is a hard omission, not a skipped branch. macOS has the same shape (service_cmd_darwin.go:124 unloads and never bootstraps), and scripts/install/install-linux.sh has the same agent hole. Windows is unaffected.

Why the watchdog did nothing: on graceful stop the agent sends a shutdown intent over IPC (internal/agentapp/main.go:580-588), which moves the watchdog from MONITORING to STANDBY (internal/watchdog/watchdog.go:36-39). STANDBY's transition table (:45-49) only accepts agent-recovered / standby-timeout / start-agent, so the subsequent process-gone signal is silently dropped. STANDBY times out after 30 minutes (internal/config/config.go:355) into FAILOVER, and the STANDBY-to-FAILOVER edge is the one edge that does not call ensureAgentStartedBeforeFailover (cmd/breeze-watchdog/main.go:762). So the box stays down indefinitely. The user_stop marker the agent writes (main.go:558-568) is read by nothing. Every graceful restart, including the auto-updater's, opens this same 30-minute unprotected window.

Watchdog still on 0.104: breeze-watchdog service install (cmd/breeze-watchdog/service_cmd_linux.go:98-104,139-145) also stops and only enables, so with a sibling present both units should have died. That yours survived on the old process most likely means the watchdog bootstrap step failed earlier (it warns at watchdog_bootstrap.go:266, typically when the sibling binary lookup or GitHub download fails on a self-host box). Inferred, not verified from your log.

Docs: agents/installation.mdx, advanced-install.mdx and self-host-migration.mdx all say service install "restarts the agent service (safe for upgrades)". That claim is false today and will be corrected. The path that is actually safe is the in-band update (internal/updater/restart_unix.go:30-38 does systemctl restart).

Fix: capture whether the unit was active before the stop and systemctl start after enable when it was (Linux, macOS, install script); watchdog install ends with a restart of its own unit; STANDBY gains transitions for agent-unhealthy and IPC-connected, user_stop standby bounded to a couple of minutes, and the STANDBY-to-FAILOVER edge starts the unit before failing over; Go tests assert the install argv contains a start. Labeled p1 / agent / medium. Until it ships, follow service install with systemctl restart breeze-agent breeze-watchdog on Linux upgrades, or prefer the in-band update.