feat(Phase-7): Observability and production readiness

[chitrank2050]

Overview

This is the tracking issue for Phase 7. The goal is to prove Meterplex can run in production — structured metrics, distributed tracing, alerting on critical business signals, and operational dashboards.

Phases 1-6 build the features. Phase 7 proves you can debug them when they break at 3am.

Success Criteria

• Prometheus metrics exposed at /metrics endpoint
• OpenTelemetry tracing on all HTTP requests, Kafka consumer messages, and database queries
• Structured JSON logs with request/event correlation IDs
• Grafana dashboards for API latency, Kafka consumer lag, outbox backlog, billing cycle health
• Alerts configured for: DLQ growth, publisher lag, invoice generation failures, webhook failures, auth brute force
• One API request traceable end-to-end: HTTP → DB writes → Kafka → consumer → aggregate → Redis
• Health check endpoint reports all dependency statuses (Postgres, Redis, Kafka)
• Load test baseline with k6 documenting throughput and latency percentiles

Tracking / Sub-tasks

• Step 1: Prometheus metrics — HTTP request duration, status codes, active connections (feat(observability): Prometheus metrics for HTTP requests #169)
• Step 2: Business metrics — usage events ingested, invoices generated, payments processed, DLQ size (feat(observability): business metrics for billing pipeline health #170)
• Step 3: OpenTelemetry tracing — HTTP spans, Kafka spans, Prisma spans (feat(observability): distributed tracing with OpenTelemetry #171)
• Step 4: Structured logging upgrade — JSON format, trace IDs in every log line (feat(observability): structured JSON logging with trace IDs #172)
• Step 5: Grafana dashboards — API golden signals, Kafka consumer health, billing cycle (feat(observability): Grafana dashboards for API and pipeline health #173)
• Step 6: Alerting rules — DLQ growth, consumer lag, invoice failures, auth brute force (feat(observability): alerting rules for critical business signals #174)
• Step 7: Health check enhancement — Kafka, Redis, Postgres status with latency (feat(observability): enhanced health check with dependency latency #175)
• Step 8: k6 load test suite — auth, usage ingestion, entitlement check, invoice generation (feat(observability): k6 load test suite for critical paths #176)
• Step 9: Performance baseline documentation — throughput, latency p50/p95/p99, bottlenecks (docs(observability): performance baseline and bottleneck analysis #177)
• Step 10: Finalize Documentation & Bruno Collection (docs(phase-7): finalize documentation and Bruno collection #178)

Resources

• Architecture Log: docs/phases/phase-7.md (to be created)
• Architecture Doc: docs/architecture/phase-7-observability.md (to be created)

Why this phase matters

The question isn't "does it work?" — phases 1-6 proved that. The question is "can you tell when it stops working?" and "can you figure out why?"

A system without observability is a system you can't operate. You can't debug what you can't see. You can't scale what you can't measure. You can't promise uptime without alerts.

This is the difference between "I built a billing system" and "I can run a billing system in production."

Note: This issue represents the "Phase 7" milestone. Individual PRs should reference the specific sub-tasks above.