Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
d3da683
Alert on Piri's chain head and PDP proving gauges
claude Sep 30, 2026
03d0a63
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Sep 30, 2026
de03aad
grafana: don't count or name the Piri rules in their comments
claude Sep 30, 2026
8dadbb8
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Sep 30, 2026
511c352
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Sep 30, 2026
730c73f
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Oct 5, 2026
c996e1b
List the Piri metric rules in the observability alert table
claude Oct 5, 2026
462944f
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Oct 5, 2026
615791e
Label the Piri metric rules team_name, as #189 did the rest
claude Oct 5, 2026
aa702f1
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Oct 6, 2026
22d379b
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Oct 7, 2026
42eca63
Merge main, and give the Piri rules a stage label
claude Oct 8, 2026
2d59df4
Merge remote-tracking branch 'origin/main' into claude/happy-bell-cfh…
claude Oct 9, 2026
ce80842
grafana: align the Piri proving rules with piri#152's reviewed semantics
claude Oct 9, 2026
a89fbb4
grafana: page on a Piri with no head, and retire the Loki stopgap
claude Oct 9, 2026
2e4e8e8
grafana: tidy the Piri alert comments
claude Oct 9, 2026
32dd110
Merge branch 'main' into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Oct 12, 2026
6b450b4
Merge branch 'main' into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Oct 12, 2026
e290397
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Oct 13, 2026
71db9bf
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Oct 13, 2026
9ae8285
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Oct 13, 2026
11aef1b
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Oct 13, 2026
5e4c2c2
Merge main into claude/happy-bell-cfh5qb-piri-proving-alerts
claude Oct 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 21 additions & 16 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,22 +84,27 @@ The rules watch staging and prod, the stages listed in `alert_stages` in
`fc-<stage>` names, and an appliance rule matches the `<stage>-<region>`
appliance label.

| Rule | Group | Source | Ticket |
| ----------------------------------- | ------------------------- | ----------------- | -------- |
| Service has no healthy hosts | Forge Central | ALB, CloudWatch | FIL-1151 |
| Service 5xx errors | Forge Central | ALB, CloudWatch | FIL-1207 |
| Service is failing most requests | Forge Central | ALB, CloudWatch | FIL-1207 |
| Provision Lambda errors | Forge Central | Lambda, CloudWatch| FIL-1151 |
| Pandora VPN tunnel down | Forge Central | VPN, CloudWatch, prod account | FIL-1436 |
| Pandora VPN both tunnels down | Forge Central | VPN, CloudWatch, prod account | FIL-1436 |
| Appliance has stopped reporting | Forge Regions | deploy stamp | FIL-1163 |
| Appliance free disk space below 40% | Forge Regions | node exporter | FIL-1209 |
| Piri has stopped receiving chain notifications | Forge Regions | Piri's logs, Loki | FIL-1383 |
| Postgres is refusing connections | Forge Regions | Postgres's logs, Loki | FIL-1163 |
| Appliance 5xx rate too high | Forge Regions | Caddy | FIL-1163 |
| Appliance container is not running | Forge Regions containers | cAdvisor, staging | FIL-1163 |
| OpenBao cannot unseal | Forge Regions outages | OpenBao's logs, Loki | FIL-1164 |
| Appliance site is failing most requests | Forge Regions outages | Caddy | FIL-1164 |
| Rule | Group | Source | Ticket |
| ---------------------------------------------------- | ------------------------ | ----------------------------- | -------- |
| Service has no healthy hosts | Forge Central | ALB, CloudWatch | FIL-1151 |
| Service 5xx errors | Forge Central | ALB, CloudWatch | FIL-1207 |
| Service is failing most requests | Forge Central | ALB, CloudWatch | FIL-1207 |
| Provision Lambda errors | Forge Central | Lambda, CloudWatch | FIL-1151 |
| Pandora VPN tunnel down | Forge Central | VPN, CloudWatch, prod account | FIL-1436 |
| Pandora VPN both tunnels down | Forge Central | VPN, CloudWatch, prod account | FIL-1436 |
| Appliance has stopped reporting | Forge Regions | deploy stamp | FIL-1163 |
| Appliance free disk space below 40% | Forge Regions | node exporter | FIL-1209 |
| Piri's chain head is stale | Forge Regions | Piri's metrics | FIL-1383 |
| Piri is reporting but has no chain head | Forge Regions | Piri's metrics | FIL-1383 |
| Piri's proving period has not advanced | Forge Regions | Piri's metrics | FIL-1383 |
| Piri proving is failing | Forge Regions | Piri's metrics | FIL-1383 |
| Piri proof set is unrecoverable | Forge Regions | Piri's metrics | FIL-1383 |
| Piri's Prove task has not run in 1.5 proving periods | Forge Regions | Piri's metrics | FIL-1383 |
| Postgres is refusing connections | Forge Regions | Postgres's logs, Loki | FIL-1163 |
| Appliance 5xx rate too high | Forge Regions | Caddy | FIL-1163 |
| Appliance container is not running | Forge Regions containers | cAdvisor, staging | FIL-1163 |
| OpenBao cannot unseal | Forge Regions outages | OpenBao's logs, Loki | FIL-1164 |
| Appliance site is failing most requests | Forge Regions outages | Caddy | FIL-1164 |

## Logs

Expand Down
Loading
Loading