Skip to content

feat(observability): add prometheus alert rules with slo burn-rate - #260

Merged
tiana-code merged 1 commit into
mainfrom
feat/E-07-alert-rules
Jun 19, 2026
Merged

feat(observability): add prometheus alert rules with slo burn-rate#260
tiana-code merged 1 commit into
mainfrom
feat/E-07-alert-rules

Conversation

@tiana-code

Copy link
Copy Markdown
Owner

Adds Prometheus alerting rules as code, the final E-07 slice (#252).

What

  • deploy/observability/alerts.yml with 5 rules across P0-P3:
    • InstanceDown (P0) - up == 0.
    • ErrorBudgetBurnFast (P1) / ErrorBudgetBurnSlow (P2) - multi-window multi-burn-rate on the 5xx ratio against a 99% availability SLO (14.4x over 5m+1h; 6x over 30m+6h). Uses http_server_requests_seconds_count only (no fragile fixed-le bucket dependency); NaN-safe when idle.
    • HighRequestLatencyP95 (P2) - histogram_quantile(0.95, ...) > 0.5s.
    • TargetFlapping (P3) - changes(up[15m]) >= 4.
  • prometheus.yml references the rules via rule_files; the profiled Prometheus mounts alerts.yml.
  • A new CI workflow validates the rules with promtool (separate check config and check rules steps) via the pinned prom/prometheus:v2.55.1 image, triggered on PR and push for deploy/observability/**.

Notes

Alertmanager/notification routing is out of scope - the rules load into Prometheus and are visible at /alerts.

Gate chain

  • critic: GO-WITH-CHANGES (separate check-rules step + dual PR/push triggers applied).
  • security-auditor (opus): PASS - no secrets, no Actions injection (static env + $PWD only), §5.3 clean.
  • code-reviewer: APPROVED - PromQL verified (and-match by job, le in by-clause, correct burn multipliers, real metrics).
  • evaluator: PASS (0.895 >= 0.80).
  • promtool unavailable locally (Docker down); the new CI workflow is the binding gate. compose-smoke unaffected (prometheus profile-gated).

Closes #252

Add P0-P3 alerting rules loaded by the observability-profile
Prometheus: an instance-down page, multi-window multi-burn-rate alerts
on the 5xx error budget (fast and slow), a p95 request-latency alert,
and a target-flapping warning. The rules reference the metrics the
services already emit and are validated in CI by promtool (check config
and check rules) against the pinned Prometheus image.

Closes #252
@tiana-code tiana-code added this to the v0.2.0 milestone Jun 19, 2026
@tiana-code
tiana-code merged commit 897ef87 into main Jun 19, 2026
9 checks passed
@tiana-code
tiana-code deleted the feat/E-07-alert-rules branch June 19, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

F-07.8 Prometheus alert rules as-code

1 participant