grafana: page when OpenBao cannot unseal or a site fails outright, and label alerts by stage

[Peeja]

Written by Claude.

Part of FIL-1164.

Adds the two outage-class appliance rules that FIL-1164 still lacks. The third, Piri's proving deadline, is #163.

Rules

Both rules are severity = "critical". They sit in a new group, Forge Regions outages, evaluated every 60s. At the 300s the other appliance rules use, evaluation alone could use up FIL-1164's five-minute budget.

OpenBao cannot unseal. The node auto-unseals through the transit key at Central. When that fails, OpenBao logs failed to unseal core at WARN and retries every five seconds for as long as the failure lasts (command/server.go, runUnseal, at v2.6.2, the version the nodes pin). The only failure that loop treats as fatal is a failed declarative self-init, and the nodes initialise with bao operator init, so the rule matches only the WARN line. It counts it over a one-minute window, with for = 2m:

• A single failed attempt during a restart drops out of the window before two minutes of pending build up.
• A real failure keeps every window full, so it fires about three minutes after the first failure.

An OpenBao that was started but never initialised logs the same line, so a region whose provisioning stops between starting OpenBao and bao operator init fires it too; that node is sealed all the same.

It doesn't catch a node sealed by hand with bao operator seal. That logs vault is sealed once and doesn't retry, and every graceful restart logs the same line.

Appliance site is failing most requests. This is the outage tier of "Appliance 5xx rate too high", which stays as the warning. It fires when more than half of a site's requests get a 5xx for two minutes. It is split by host, so Ingot's site is judged on its own: Ingot exports no HTTP metrics, so Caddy's view of its site is its error rate. It uses a three-minute rate window, because Caddy is scraped every minute and rate() needs two samples, and links to the same Error rate panel.

It only fires once a site has served at least five 5xx responses in the window. The warning's floor of 0.1 requests per second would hide a quiet site that fails everything, and a busy one whose clients back off once it fails. Five errors in three minutes still keeps a single failed request on an idle site from paging. A site whose clients stop sending entirely has no errors to count, and this rule can't see it.

docs/observability.md's alert table has a row for each.

A stage label on every alert

The aim is for production critical alerts to page where staging's notify. An IRM route can only tell the two apart by a label the alert carries:

• Central's service rules already get stage from their queries.
• Every appliance rule now derives it from the appliance label, which is <stage>-<region> by construction. It is cut at the first hyphen, the way grafana: make an alert's dashboard link open on the stage that fired #199's dashboard links do.
• "Provision Lambda errors" derives it from its function name, fc-<stage>-provision.

An alert raised because a query failed (DatasourceError) or came back empty (DatasourceNoData) carries none of the query's labels, so it has no stage, and a route matching stage = "prod" won't see it. docs/observability.md now describes the routing and says IRM should page on a critical alert with no stage as well as on stage = "prod".

Adding a label changes every appliance alert instance's identity. Their state resets once, when this is applied.

Whichever of this and #163 merges second needs stage on #163's four rules too.

Not checked

• The OpenBao log line comes from reading the source, not from seeing it in staging's Loki. Before relying on the rule, run this query in Explore over a recent restart of the staging node's OpenBao: {service_name=~"appliance-.*-openbao"} |~ "unseal". It should show the unseal lines, and none of failed to unseal core while healthy.
• Whether Grafana drops an empty stage or keeps it as "" on a DatasourceError instance. Either way it doesn't match prod; the IRM route for "no stage" should be tested with a forced datasource error on a staging rule.
• The service_name the rule matches is inferred. It is appliance-<stage>-<region>-openbao, from the Alloy rule that builds names out of the Compose service, as dev's config does. Staging's Alloy is the host's own and lives outside these repositories.
• The stage label is unverified against live data. Check it after applying: the rules' instances in Grafana Alerting should show stage=staging, and not an empty value or the whole appliance name.
• The 50% threshold and the five-error gate are chosen, not derived, like the warning's 5%. FIL-1242 would set them.

Testing

• make check passes, including tofu fmt and the stage-picker check.
• tofu validate passes on the cached grafana provider 4.46.0.

✴️

🤖 Generated with Claude Code

https://claude.ai/code/session_01JCF8gyb8jNRmHmWqQmVhRJ

---

Generated by Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.