Skip to content

docs(ops): correct the mechanism my own comment got wrong, and name the blast radius #319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jhfnetboy merged 1 commit into from
Sep 5, 2026
Merged

docs(ops): correct the mechanism my own comment got wrong, and name the blast radius #319

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion deploy/README-heartbeat.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Four launchd agents, sources version-controlled in `deploy/launchd/`:
| -------------------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `io.aastar.dvt-committee-health` | every 900 s | `committee-health.mjs` against the router; opens/updates a GitHub issue on any non-zero exit |
| `io.aastar.dvt-apply-rotation` | every 3600 s | `apply-verifier-rotation.mjs --broadcast`. **Its rotation is done** — applied externally at 2026-09-04T05:37:12Z, not by this job — so runs from here do not broadcast: they check the active verifier against the address the job is pinned to, print "already applied" and exit 0. Green means the observed state matched that pin (it fails loudly on a disarm, a different verifier, or an unpinned new rotation); it does **not** mean this job applied anything. Repoint the expected verifier before the next rotation |
| `io.aastar.dvt-tunnel-keepalive` | every 300 s | `tunnel-keepalive.sh`: probes the **public** `dvt{1,2,3}.aastar.io/health` and restarts the cloudflared container when they stop serving, then re-probes to confirm it worked |
| `io.aastar.dvt-tunnel-keepalive` | every 300 s | `tunnel-keepalive.sh`: probes the **public** `dvt{1,2,3}.aastar.io/health` and restarts the cloudflared container when they stop serving, then re-probes to confirm it worked. Refuses to restart while any node is unhealthy (exit 3) — compose destroys the tunnel _before_ checking `service_healthy`, so restarting there would turn a partial outage into a total one. Worth knowing: **one** unhealthy node blocks cloudflared from starting at all, so in that window the other two nodes have no public entry either |
| `io.aastar.dvt-committee-keeper` | `KeepAlive` (resident) | keeps `committee-keeper.mjs --watch` alive; restarts it if it dies |

The first two run `deploy/local-heartbeat.sh`, which is **independent of
Expand Down
18 changes: 15 additions & 3 deletions deploy/tunnel-keepalive.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,17 @@ internet_up() {
# therefore the condition that makes compose refuse to start. A 2-of-3 degradation would have been
# converted by this script into a 3-of-3 outage.
#
# Empty status (a container with no healthcheck defined) counts as NOT healthy: compose's
# service_healthy would not be satisfiable either, so fail-closed is the truthful direction.
# A container with no healthcheck counts as NOT healthy, which is the right direction (compose's
# service_healthy would not be satisfiable either). But note the MECHANISM, because the obvious
# reading is wrong: `.State.Health` is absent rather than empty on such a container, so the template
# does not return "" -- `docker inspect` FAILS with
# template parsing error: ... map has no entry for key "Health"
# and exits 1, and the `2>/dev/null` below swallows it. `st` ends up empty as a side effect of the
# command failing, not because the field is empty. Anyone who removes that redirect to debug
# something else will suddenly see template errors here; they are pre-existing, not a new fault.
# (Measured. Also measured: aastar-dvt:latest carries its own HEALTHCHECK, which compose inherits
# even when the service does not declare one, so reaching this state at all needs an explicit
# `healthcheck: {disable: true}`.)
nodes_healthy() {
local c st
for c in "${CONTAINERS[@]}"; do
Expand Down Expand Up @@ -148,7 +157,10 @@ if ! nodes_healthy; then
say "NODES DOWN or NOT HEALTHY: one or more of ${CONTAINERS[*]} is not reporting healthy. The \
tunnel is not the problem; restarting it would front an empty origin -- and compose would destroy \
the running tunnel and then refuse to recreate it, turning a partial outage into a total one. \
Start or repair the node stack first."
Start or repair the node stack first. NOTE: while ONE node is unhealthy, cloudflared cannot be \
started at all (compose gates it on all three being healthy), so if the tunnel also goes down in \
that window, the healthy nodes' public endpoints cannot come back either -- do not read 'the \
tunnel is not the problem' as 'the other two are fine publicly'."
exit 3
fi

Expand Down
Loading