Alert when Piri stops receiving chain notifications

[Peeja]

Written by Claude.

Part of FIL-1383

Problem

On 2026-09-28, staging's Lotus stopped following the chain. Piri stayed up, answered its health check and kept sending metrics, but it stopped proving. Nothing alerted, and the problem surfaced more than a day later, when a deploy hung.

Piri exports no chain or proving metrics yet; FIL-1383 adds them. Until then, the one signal is in its logs. Curio's chain scheduler, which Piri runs, logs this while the head isn't moving:

no notifications received in 5m0s, resubscribing to ChainNotify

The text is at curio lib/chainsched/chain_sched.go:209, at the version Piri pins. It recurs about every ten minutes, and staging's Piri logged it throughout the incident.

Change

terraform/envs/grafana/alerts.tf: a fifth rule in the appliance group, "Piri has stopped receiving chain notifications". It's the first rule here that reads logs rather than metrics.

• Query A, a Loki instant query. With alert_stages = ["staging"], as terraform.tfvars sets it, it renders as:

  sum by (appliance, region, node) (
    count_over_time(
      {appliance=~"(staging)-.*", service_name=~"appliance-.*-piri"}
        |~ "no notifications received in .* resubscribing to ChainNotify"
      [15m]
    )
  )
  

  The (staging) group comes from local.stages, which joins alert_stages with |. Adding production would make it (staging|prod).
• Stage B: threshold > 0, the same shape as the other rules.
• for = 15m, so one line doesn't fire the alert by itself. It fires about twenty minutes after the head stops moving.
• no_data_state = OK: an empty result is the healthy state, as in the 5xx rule.
• severity = critical: a node that isn't proving is what matters most.
• Routing: team = forge, like every other rule here.
• runbook_url: the runbook's "When something is wrong" section, as the other appliance rules use.
• The stage filter uses alert_stages, as the other appliance rules do, so today it watches staging and not dev.

loki_datasource_uid: a new variable, set in terraform.tfvars like the Prometheus one. Its value, grafanacloud-logs, is confirmed from the data source's settings page URL.

Testing

• Against the incident: query A, run in Explore as a range query over the stuck window (2026-09-28 23:00 to 2026-09-29 21:40 UTC), returns a series with node="staging/eu-central-3". The stream carries appliance="staging-eu-central-3" and service_name="appliance-staging-eu-central-3-piri". That confirms the selector, the log text and the labels the summary uses.
• tofu fmt -check -recursive passes.
• tofu validate passes, with init -backend=false and the pinned grafana provider 4.46.0. The lock file is unchanged.

Notes

• Applies on merge if GRAFANA_APPLY_ENABLED is set.
• Staging's Piri is stopped at the moment, so this won't fire there until it runs again. Once it does, the alert should fire until the host's Lotus is fixed, which is correct.
• Not caught: a Lotus that's down outright logs a different error, so this rule doesn't catch it. Neither does a Lotus that keeps sending stale heads. Alert on Piri's chain head and PDP proving metrics #163's metric-based alerts cover both, once Export chain head and PDP proving metrics piri#152 is deployed, and this rule can go then.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JCF8gyb8jNRmHmWqQmVhRJ

✴️

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.