Problem
When a usage event takes 5 seconds to process, where's the bottleneck? Is it Postgres? Kafka? Redis? The validation logic? Without tracing, debugging distributed systems is guesswork.
Proposed solution
Integrate OpenTelemetry SDK:
- Auto-instrument: HTTP (incoming + outgoing), Prisma/pg, Redis (ioredis), kafkajs
- Manual spans for business logic: outbox publish, validation checks, aggregation upsert, invoice generation
- Export to Jaeger (local dev) or OTLP collector (production)
- Trace context propagation: HTTP → Kafka message headers → consumer spans
- Correlation ID from request middleware linked to trace ID
API design (if applicable)
No API change. Traces exported to collector backend.
Alternatives considered
- Datadog APM — rejected as the only option. Vendor lock-in. OpenTelemetry is vendor-neutral — export to Datadog, Jaeger, Grafana Tempo, or any OTLP-compatible backend.
- Manual logging with timestamps — rejected. Not scalable. Can't visualize call graphs or identify parallelism.
Additional context
Add Jaeger to docker-compose.yml for local development. UI at http://localhost:16686.
Problem
When a usage event takes 5 seconds to process, where's the bottleneck? Is it Postgres? Kafka? Redis? The validation logic? Without tracing, debugging distributed systems is guesswork.
Proposed solution
Integrate OpenTelemetry SDK:
API design (if applicable)
No API change. Traces exported to collector backend.
Alternatives considered
Additional context
Add Jaeger to
docker-compose.ymlfor local development. UI athttp://localhost:16686.