Skip to content

feat(observability): business metrics for billing pipeline health #170

Description

@chitrank2050

Problem

HTTP metrics tell you the API is responding. They don't tell you the billing pipeline is healthy. We need metrics for business-critical operations.

Proposed solution

Add custom Prometheus metrics:

  1. meterplex_usage_events_ingested_total — counter by tenant, feature, status (accepted/rejected/duplicate)
  2. meterplex_usage_events_validated_total — counter by result (valid/rejected/dead_letter)
  3. meterplex_usage_events_aggregated_total — counter
  4. meterplex_outbox_pending_count — gauge (current PENDING outbox rows)
  5. meterplex_outbox_publish_duration_seconds — histogram
  6. meterplex_dead_letter_total — counter by stage (validation/aggregation/publishing)
  7. meterplex_invoices_generated_total — counter by type (full/prorated)
  8. meterplex_payments_total — counter by status (succeeded/failed)
  9. meterplex_entitlement_check_duration_seconds — histogram (the hot path)
  10. meterplex_kafka_consumer_lag — gauge by consumer group

API design (if applicable)

Exposed via the same /metrics endpoint. No separate API.

Alternatives considered

  • Log-based metrics — rejected. Structured logs are good for debugging but bad for real-time dashboards. Prometheus metrics are purpose-built for monitoring.

Additional context

Instrument at the service level, not the controller level. The producer/consumer services emit metrics directly.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions