Skip to content

Add OpenTelemetry trace context propagation across Express middleware and Horizon listener in new src/observability/tracing.ts #654

Description

@1nonlypiece

📌 Description

The backend already emits Prometheus metrics and correlation IDs, but there is
no distributed tracing: a slow create_vault request that fans out into a
Soroban submission, an ETL enqueue, and a webhook delivery cannot be followed as
a single trace. Without spans it is hard to attribute latency in the Horizon →
event-processor → vault-store pipeline.

This issue wires OpenTelemetry tracing with W3C traceparent propagation across
the HTTP layer and into the background pipeline.

Goal: a single trace links an inbound request to its downstream Soroban,
job-queue, and webhook spans.

🎯 Requirements and Context

  • src/observability/tracing.ts initializes a tracer with an OTLP exporter
    configured from env (OTEL_EXPORTER_OTLP_ENDPOINT), disabled when unset.
  • An Express middleware extracts/injects traceparent and reuses the existing
    correlation ID as a span attribute.
  • Spans are created around Soroban submission (soroban.ts), job handlers
    (jobs/handlers.ts), and webhook delivery.
  • No-op tracer in test/dev so the suite runs without a collector.

🛠️ Suggested Execution

1. Create a branch

git checkout -b feature/otel-tracing

2. Implement changes

  • Add src/observability/tracing.ts and a tracing middleware.
  • Instrument the three downstream call sites with spans.
  • Validate OTEL_* vars in src/config/env.ts.
  • Document in docs/operations-metrics.md.

3. Test and commit

  • Add src/tests/tracing.test.ts using an in-memory span exporter to assert
    parent/child linkage and correlation-ID attribute propagation.
  • Run bun test src/tests/tracing.test.ts.

Example commit message

feat: OpenTelemetry trace propagation across HTTP and background pipeline

✅ Guidelines

  • Minimum 95% test coverage on new/changed lines.
  • No-op when no collector configured; document the env contract.
  • Timeframe: 96 hours.

🏷️ Labels

type-feature · area-backend · type-enhancement · MAYBE REWARDED · GRANTFOX OSS · OFFICIAL CAMPAIGN

💬 Community & Support

Metadata

Metadata

Assignees

Labels

GRANTFOX OSSGrantFox open-source campaignMAYBE REWARDEDEligible for GrantFox rewardOFFICIAL CAMPAIGNOfficial GrantFox campaign issueStellar WaveIssues in the Stellar wave programarea-backendBackend / API worktype-enhancementImprovement to existing functionalitytype-featureNew feature work
No fields configured for Feature.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions