feat(observability): alerting rules for critical business signals

[chitrank2050]

Problem

Dashboards require someone watching them. Alerts notify the team when something goes wrong — before customers notice.

Proposed solution

Prometheus alerting rules (exported to Alertmanager or Grafana Alerts):

1. DLQ growth: meterplex_dead_letter_total increases by >10 in 5 minutes → warning
2. Consumer lag: meterplex_kafka_consumer_lag > 1000 for >5 minutes → warning
3. Outbox backlog: meterplex_outbox_pending_count > 500 for >5 minutes → critical
4. Invoice generation failure: meterplex_invoices_generated_total{status="failed"} increases → critical
5. Payment failure spike: meterplex_payments_total{status="failed"} rate > 20% of total → warning
6. API error rate: http_requests_total{status=~"5.."} rate > 1% → warning
7. Auth brute force: http_requests_total{path="/api/v1/auth/login",status="401"} rate > 50/min → critical
8. Entitlement check latency: meterplex_entitlement_check_duration_seconds p99 > 100ms → warning

API design (if applicable)

N/A — Prometheus/Grafana alerting configuration files.

Alternatives considered

• Application-level alerting — rejected. Monitoring infrastructure should handle alerting, not the application.
• PagerDuty integration — future enhancement. Start with Slack/email notifications.

Additional context

Alert rules stored in prometheus/alerts/ directory, version-controlled alongside the application.