feat(observability): full observability stack with OTel, Prometheus, Grafana, Loki, Tempo

[Mohith1612]

Summary

• Replaces standard logging with structured JSON (structlog) with automatic trace/span ID injection
• Adds OpenTelemetry distributed tracing across the API→worker async boundary using OTel Links
• Instruments enqueue/dequeue/worker loop with Prometheus metrics (counters, histograms, gauges) exposed via a dedicated metrics endpoint
• Ships a full sidecar observability stack: OTel Collector, Prometheus, Tempo, Loki, Promtail, and Grafana with 5 pre-built dashboards and 7 alert rules
• Updates CI deploy script to run the observability compose alongside the main stack

Metrics added

redis_command_duration, enqueue, retry_drain, jobs_failed, jobs_retried, jobs_active, retry_backoff, worker_loop_iterations, queue_depth, db_pool_size

Grafana dashboards

1. System health
2. Worker performance
3. Queue behavior
4. Infrastructure (DB pool, Redis)
5. Trace/log explorer (Tempo ↔ Loki correlation)

Test plan

• CI passes (lint, typecheck, tests, docker build)
• After merge: deploy-backend job succeeds and VM health check passes
• On VM: docker compose -f docker-compose.yml -f docker-compose.observability.yml ps shows all 10 services healthy
• Enqueue a job via POST /api/v1/jobs, verify trace appears in Grafana → Explore → Tempo
• Verify structured JSON logs appear in Grafana → Explore → Loki
• Verify Prometheus targets page shows api, worker, and otel-collector all UP

[cloudflare-workers-and-pages]

Deploying job-queue-system with    Cloudflare Pages

Latest commit:	377fb15
Status:	✅  Deploy successful!
Preview URL:	https://dc227034.job-queue-system.pages.dev
Branch Preview URL:	https://feat-observability.job-queue-system.pages.dev

View logs