feat(server): export HTTP request metrics via OTLP

[hpliStartAgain]

Problem

The lifecycle Server already owns an OpenTelemetry MeterProvider and OTLP HTTP exporter, but main only exports the SDK-reported opensandbox.sandbox.create.duration histogram. Direct REST clients, authentication failures, validation errors, unmatched routes, and other Server HTTP traffic therefore have no generic request-rate, error-rate, or latency signal in the Server OTLP stream.

This is deliberately separate from #1408 / #1412, which instrument lifecycle business operations, and narrower than the closed, unmerged #1185 Server+Controller metrics/traces proposal.

Minimal reproduction

1. Enable [otel] for the Server and point it at an OTLP metrics collector.
2. Send an authenticated request such as GET /v1/sandboxes, then also send a request that returns 401, 404, or 422.
3. Inspect the exported Server metrics.

Before this change, the collector may receive SDK-reported creation latency, but it receives no generic HTTP request metric for those requests.

Root cause

The existing Server OTEL integration creates only the SDK creation-duration instrument. The FastAPI application has no request metrics middleware and does not record matched route templates or response status codes.

Change

• add a thin pure-ASGI middleware around the complete user middleware stack;
• export one server.http.request.duration histogram in milliseconds through the existing Server MeterProvider;
• attach only http_method, matched http_route, and http_status_code;
• use http_route=unknown when routing never resolves, including early auth failures;
• record 500 for unhandled exceptions before re-raising them unchanged;
• keep recording as a no-op when [otel].enabled = false and contain recorder failures;
• document the Server metric inventory and route-cardinality/privacy contract.

The histogram count provides request volume/QPS, status-code dimensions provide error rate, and buckets provide latency percentiles without a separate request counter.

Tests

• uv run pytest tests/test_http_metrics.py tests/test_metrics_api.py -q — 17 passed
• uv run pytest -q — 1407 passed
• uv run ruff check — passed
• targeted Pyright for the changed middleware, OTEL helper, and regression tests — 0 errors
• corepack pnpm@9.15.0 docs:build — passed
• ./scripts/verify-license.sh — passed
• in-memory OpenTelemetry reader — collected server.http.request.duration with the expected unit, count, and bounded attributes

Full-repository uv run pyright is not currently a green baseline on main (1386 existing errors); the files and tests changed here pass the targeted check.

Compatibility and security impact

This is additive. It changes no OpenAPI contract, SDK, CLI, CRD, Helm value, listener, port, or authentication behavior, and adds no dependency or configuration key. With OTEL disabled, request behavior is unchanged apart from a no-op middleware call.

Raw paths, query strings, sandbox IDs, tenant IDs, API keys, and request/response bodies are never metric attributes. Route templates are application-defined and bounded. Metric recording and exporter failures cannot change an HTTP response.

Existing Issue

Fixes #1560.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 537187897f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9caa8acb2f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".