Alert on Piri's chain head and PDP proving metrics

[Peeja]

Written by Claude.

Part of FIL-1383

Uses the metrics from fil-forge/piri#152, and follows the alerts its docs settled on after five review rounds. It was stacked on #162, which has merged.

Problem

#162 catches a stuck chain head from a log line. It misses a Lotus that keeps sending stale heads, and it says nothing about proving itself. fil-forge/piri#152 adds gauges for Piri's chain head, each proof set's schedule and recorded proving failures, the proof sets Curio has given up on, and the last run of each proving task. This PR adds the alerts that use them.

The gauges are what Curio recorded in its database, not chain state, so none of these rules confirms that a proof landed. Each rule pages on one failure, and is named for what its metric actually shows.

Change

terraform/envs/grafana/alerts.tf: six rules in the "Forge Regions" group. They use a new piri_metric_matcher local (appliance=~"(<stages>)-.*", job="forge/piri", where <stages> is alert_stages joined with |, today staging|prod) and follow the Prometheus rules' existing shape: an instant query, then a threshold stage. Each carries team_name, component, severity and stage = local.appliance_stage_label, and groups by appliance, region, node, plus proof_set where the failure is a proof set's.

Rule	Severity	for	Fires when
Piri's chain head is stale	critical	5m	the last applied tipset is more than 300s old
Piri is reporting but has no chain head	critical	10m	target_info and piri_pdp_proofsets_unrecoverable are present, but no head gauge is
Piri's proving period has not advanced	critical	5m	a proof set's recorded challenge window has closed, against the head extrapolated from wall-clock time, and the set is not in a failure backoff
Piri proving is failing	critical	0m	increase(piri_pdp_proofset_consecutive_prove_failures[1h]) > 0 for a proof set
Piri proof set is unrecoverable	critical	0m	increase(piri_pdp_proofsets_unrecoverable[1h]) > 0
Piri's Prove task has not run in 1.5 proving periods	critical	10m	time since the last PDPv0_Prove run without a retryable error, over the longest proving period seen in a day, is > 1.5

Why each is shaped as it is:

• Head stale, 5m: a run of null rounds or a slow chain scheduler handler can leave five minutes between heads, so the head has to stay stale across two of the group's five-minute evaluations.
• No chain head, the extra gate: Piri reports the unrecoverable count on every collection, proof sets or none, and it does not depend on the chain. Requiring it keeps the rule quiet on a Piri from before chore(deps): Bump github.com/ethereum/go-ethereum from 1.17.5 to 1.17.6 #152, and covers a node with no proof sets.
• Not advanced: it catches scheduling that has stopped, such as a stuck head or the task engine not running. It does not catch a missed proof: Curio schedules the next period as soon as the window closes, whether or not the proof landed. A set in a failure backoff is left out with unless … next_prove_attempt_epoch > <extrapolated head>, since Curio holds it back on purpose and "proving is failing" has already paged. 5m is enough, since a healthy set is past its window for only a minute or two.
• Proving failing, increase over 1h: only a successful prove send resets the count, so the level stays up for up to a proving period after any revert, and indefinitely if proving is then disabled. That makes it a graph, not a page. increase rather than delta, so a reset followed by a new failure within the hour still fires. The window is the debounce, so for is 0m. During a long backoff it pages after each failure rather than throughout.
• Unrecoverable, increase over 1h: Piri doesn't run Curio's data set deletion, so the count never goes down on its own and the level would page forever. A set already unrecoverable when the metric first arrives is never paged for, and removing a row by hand while others remain reads as a reset and fires once.
• Prove task, critical: the metric is the last run without a retryable error, which includes runs that woke too late or had their proof rejected. So it shows the task is running, not that proofs land. It adds a Prove task that keeps failing with a retryable error, or never runs, while scheduling carries on. Critical, as chore(deps): Bump github.com/ethereum/go-ethereum from 1.17.5 to 1.17.6 #152's table recommends. The period is read over a day, so a node whose last set goes unrecoverable loses it before the threshold, and this rule doesn't repeat that page.
• no_data_state = OK on all six: an empty result is healthy, or means a Piri older than the metrics. The two increase rules evaluate to 0 on a healthy set, so they are empty only when there is nothing to watch.
• exec_err_state = Error, like the other appliance metric rules.

Every group_left/group_right carries () on the same line. On the next line, ( is parsed as a label list, which breaks the whole rule (piri#152's fourth round).

docs/observability.md: the six rules are in the alert table.

This PR also deletes the Loki rule from #162, "Piri has stopped receiving chain notifications", with its matcher and docs row. These rules replace it, and running both would page twice for a stuck Lotus. So the PR must merge only once every stage the rules watch runs a Piri that exports these gauges: until then, the Loki rule is the only thing watching the chain there.

Draft until

fil-forge/piri#152 is deployed to the alerting stages, today staging and prod. Prod has no nodes yet, so staging is the one to check. Before these rules run, check in Explore that:

• the metrics arrive under these names, for example piri_chain_head_timestamp_seconds{job="forge/piri"} and piri_pdp_proofsets_unrecoverable{job="forge/piri"};
• staging's Piri series carry the labels the rules select and group by: count by (node, appliance, region) (target_info{job="forge/piri"}) should show staging/eu-central-3, staging-eu-central-3 and eu-central-3;
• piri_pdp_proofsets_unrecoverable reads 0, since a set that is already unrecoverable won't be paged for.

Testing

• tofu fmt -check -recursive and tofu validate pass, with the pinned grafana provider, after merging main.
• scripts/check-stage-picker.sh, scripts/normalise-dashboard.sh --check and git diff --check pass.
• All six expressions, with the matcher filled in, parse in Prometheus v0.314's PromQL parser. The file's other fourteen also parse, with placeholder matchers.
• In v0.314's test engine, with synthetic series:
  • "not advanced" fires for a set past its window;
  • it leaves out a set whose backoff deadline is ahead of the head, and takes it back once the head passes;
  • increase fires on a rise, a reset and a new rise, where delta gives 0;
  • a flat count gives 0.
• Not run: any rule against real series, and Grafana's own validation, which only happens on apply.

Notes

• Not caught: a proof Curio sent that then failed on chain, and a Prove task that woke after its window closed. Curio records both as successful runs and leaves the failure count alone. Catching them needs "last proven" from chain state, which chore(deps): Bump github.com/ethereum/go-ethereum from 1.17.5 to 1.17.6 #152 leaves as follow-up.
• A Piri that stops reporting entirely isn't caught by any rule in this file, because an empty result is healthy for all of them. A separate absent(target_info{…}) rule would cover it. I haven't written one, because it would need to know which nodes are meant to exist.
• A node added later must run a Piri that includes Export chain head and PDP proving metrics piri#152. One that predates it exports none of these gauges, so no rule here fires for it, and the Loki rule that would have caught a stuck head is gone.
• Staging's Alloy now runs Select staging's Forge telemetry by service.namespace infra-nodes#113's namespace-based configuration, which keeps forge/.+ jobs.
• The 30-second epoch is hard-coded, which is right for Calibration and mainnet.

✴️

🤖 Generated with Claude Code

https://claude.ai/code/session_01JCF8gyb8jNRmHmWqQmVhRJ

---

Generated by Claude Code

[Peeja]

Claude reviewer says… (1st round, ce80842)

Nothing blocking. The six expressions parse, match fil-forge/piri#152's final alerts, and bring back none of the bugs fixed there. Two should-fix findings: one failure gets only a warning until well after a proof is missed, and the Loki rule this PR keeps would double-page with the new ones.

Verified:

• tofu fmt -check -recursive, tofu validate, scripts/check-stage-picker.sh and git diff --check pass.
• All six expressions parse in Prometheus v0.314.0's parser, with the matcher filled in.
• Each matches chore(deps): Bump github.com/ethereum/go-ethereum from 1.17.5 to 1.17.6 #152's monitoring.md:
  • increase rather than delta;
  • the backoff exclusion against the extrapolated head;
  • group_left()/group_right() with () on the same line.
• The metric names survive OTLP translation unchanged.
• appliance, region and node survive every aggregation, so the stage template resolves.
• Labels, no_data_state/exec_err_state, the missing dashboard link and for = 0m all follow the neighbouring rules.
• At the group's 300s interval, for = 5m fires on the second evaluation and 0m on the first.
• In promqltest, a Piri with target_info but no head fires "no chain head", while "not advanced" returns empty for the same node.
• Not run: anything against live data, and Grafana's apply-time validation.

1. should-fix: "Piri is reporting but has no chain head" is a warning, and its comment says the proving rules page if the failure lasts long enough (alerts.tf ~1525-1540). They don't:

   • "Not advanced" needs the head series, so it returns nothing.
   • "Head stale" has no series either.
   • "Proving failing" can't fire, because nothing is sent.
   • The Loki rule doesn't fire when Lotus is down outright.

   So a Piri against an unreachable Lotus gets only a warning until the Prove-task rule fires, about 36h later, while a proof is missed within about a day.

2. should-fix: the Loki rule "Piri has stopped receiving chain notifications" stays (~765-793). Merging at the draft gate runs it alongside the new rules, so a stuck Lotus raises two critical alerts. Its comment "Nothing else Piri exports says so" also stops being true.

3. nit: "for is 5m, the shortest pending period a group evaluated every five minutes can give". 0m is shorter, so it should say the shortest non-zero one (~1600-1610).

4. nit: "loses the period series within the day, before the threshold" holds only while 1.5 proving periods is longer than a day (~1915-1920).

5. note: one cause can raise several pages, for example a stuck head pages "head stale", then "not advanced", then "Prove task". That's chore(deps): Bump github.com/ethereum/go-ethereum from 1.17.5 to 1.17.6 #152's design, and worth a sentence in the header comment.

6. nit: the new docs/observability.md table rows don't line up with the table's columns.

✴️ Claude Code session

---

Generated by Claude Code

[Peeja]

Claude author says…

The fixes are in a89fbb4.

1. Fixed. "Piri is reporting but has no chain head" is now critical, like "chain head is stale", which has the same cause. Its comment now says why: none of the proving rules can fire for a node with no head, so the only later page would come after a proof was missed.
2. Fixed by deleting the Loki rule "Piri has stopped receiving chain notifications", along with its now-unused piri_log_matcher and its docs row.
   • Why: these rules replace it, and the header comment now says that running both would page twice.
   • When to merge: the PR body now says it must merge only once every stage the rules watch runs a Piri that exports the gauges, because until then the Loki rule is all that watches the chain there.
3. Fixed: "the shortest non-zero pending period".
4. Fixed. The comment now gives the assumption: the Prove-task rule lapses before its threshold only while 1.5 proving periods is longer than a day, as with mainnet's 2880 epochs.
5. Added to the header comment: one cause can raise more than one of these over time, and each names the failure as it has become by then.
6. Fixed. The docs/observability.md table is realigned, including the rows that predate this PR.

Checks:

• tofu fmt -check -recursive, tofu validate on terraform/envs/grafana, scripts/check-stage-picker.sh and git diff --check pass.
• The PromQL is unchanged since round 1's parse check.

Round 1 had should-fix findings, so a round 2 follows.

✴️ Claude Code session

---

Generated by Claude Code

[Peeja]

Claude reviewer says… (2nd round, a89fbb4)

There's nothing blocking and nothing that needs a should-fix: only comment nits and one note on the merge gate.

Verified:

• tofu fmt, tofu validate, scripts/check-stage-picker.sh and git diff --check pass.
• All six expressions parse with the matcher filled in.
• Apart from comments and the deletion, a89fbb4's only change is warning → critical on "reporting but has no chain head".
• Nothing in infra-central or infra-nodes' docs/RUNBOOK.md still refers to the deleted Loki rule or piri_log_matcher.
• The docs table realignment changed only whitespace in existing rows.
• Coverage holds for nodes that export the gauges. piri_chain_head_timestamp_seconds is the tipset's own timestamp, so a Lotus that resubscribes with a stuck tipset still trips "chain head is stale".
• The no-head rule's claim holds: of the other rules, only the Prove-task rule can still fire for a Piri with no head.
• The metric names, backoff exclusion, increase window and thresholds match Export chain head and PDP proving metrics piri#152's final monitoring.md.

Findings:

1. nit (alerts.tf, "proving period has not advanced"): it still says for is "the shortest a five-minute group gives"; 0m is shorter. Use the "shortest non-zero pending period" wording that the chain-head rule now has.
2. nit (alerts.tf, chain-head rule): one comment line runs to 85 columns.
3. nit (alerts.tf, postgres_log_matcher/openbao_log_matcher): "same scheme" referred to the deleted piri_log_matcher comment, the one that explained how container log streams are named. It now follows piri_metric_matcher, which uses a different scheme.
4. nit (alerts.tf, Piri group header): "The proving failures page (critical)" undersells it, since the chain-head rules page too.
5. note (PR body): the merge gate is right for today. But once this merges, a node added later that runs a Piri older than chore(deps): Bump github.com/ethereum/go-ethereum from 1.17.5 to 1.17.6 #152 has no chain or proving coverage at all, where the Loki rule used to cover it. Worth a line in Notes.

✴️

---

Generated by Claude Code

[Peeja]

Claude author says…

The fixes are in 2e4e8e8.

1. Both five-minute pending periods now say "the shortest non-zero pending period a five-minute group gives".
2. Rewrapped.
3. The explanation of container log stream naming (Alloy names each stream appliance-<stage>-<region>-<service> from the Compose service and stamps appliance, region and node) now sits on postgres_log_matcher. OpenBao's "same scheme" refers back to it again.
4. The header now says all of these rules page, and why.
5. Added under Notes: a node added later must run a Piri that includes Export chain head and PDP proving metrics piri#152.

Checks: tofu fmt -check -recursive, scripts/check-stage-picker.sh and git diff --check pass. The commit changes only comments, so I didn't run the PromQL checks again.

This round found only nits, so there won't be another. The PR stays a draft until its gate is met.

✴️

---

Generated by Claude Code