Skip to content

feat(server): export low-cardinality HTTP request metrics via OTLP #1560

Description

@hpliStartAgain

Why do you need it?

The lifecycle Server already has an OpenTelemetry MeterProvider and OTLP HTTP exporter, but the only instrument on main is opensandbox.sandbox.create.duration, populated by SDK reports sent to POST /v1/metrics/events.

As a result, direct REST clients and requests that fail before a lifecycle operation is reached do not produce Server-side request telemetry. Operators cannot derive basic Server request rate, status-code error rate, or route latency from the existing OTLP stream.

This is adjacent to, but intentionally separate from, #1408 / #1412: those cover lifecycle business operations on mutating handlers. It is also a narrower metrics-only follow-up to the closed, unmerged #1185 proposal; this issue does not add tracing or Controller instrumentation.

A minimal reproduction on current main is to enable [otel], send any authenticated or unauthenticated request to the Server, and inspect the exported metrics. The SDK-reported creation histogram may be present, but no generic HTTP request metric is emitted.

How could it be?

Reuse the existing Server OTLP pipeline and add a thin ASGI middleware that records one histogram:

Metric Type Unit Attributes
server.http.request.duration Histogram ms http_method, http_route, http_status_code

The histogram sample count provides request volume/QPS, while the status-code attribute provides error-rate breakdowns.

The middleware should:

  • cover successful responses, authentication failures, validation errors, unmatched routes, and unhandled exceptions;
  • use the matched route template, never the raw URL path, with unknown when no route template is available;
  • keep attributes low-cardinality and exclude sandbox IDs, tenant IDs, API keys, request/response bodies, and query strings;
  • remain a no-op when the existing [otel].enabled setting is false;
  • ensure metric recording or exporter failures never change the HTTP response.

Other related information

This should be additive and reuse the current opentelemetry-api, SDK, and OTLP HTTP exporter dependencies. It should not add a /metrics endpoint, a new listener or port, an authentication bypass, a tracing provider, or new configuration.

Suggested tests cover route-template normalization and 2xx, 401, 404, 422, and 500 status semantics, plus the disabled and recorder-failure paths.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions