server: no self-instrumentation — the only exported metric is a client-reported number, and no failure is ever counted

[ferponse]

Summary

The lifecycle server exports exactly one metric, and it does not measure the server. sandbox.create.duration is a number the SDK reports to POST /metrics/events; the server only forwards it to OTLP. Nothing the server itself does — provisioning, deleting, pausing, talking to Kubernetes or Docker — is instrumented, and no failure of any kind is counted.

Evidence

server/opensandbox_server/integrations/otel/metrics.py defines a single instrument:

def _histogram_from_provider(provider: MeterProvider):
    return provider.get_meter("opensandbox.server").create_histogram(
        name=_CREATE_DURATION_HISTOGRAM_NAME,
        unit=_CREATE_DURATION_UNIT,
        description=_CREATE_DURATION_DESCRIPTION,
    )

and its only caller is the ingest endpoint:

server/opensandbox_server/api/metrics.py:67:  record_sandbox_create_duration(
server/opensandbox_server/api/metrics.py:49:  @router.post("/metrics/events", ...)

with attributes sdk.language, sdk.version, success — i.e. the shape of a client report, measured client-side, including network and SDK overhead.

Two consequences:

1. A deployment whose clients do not call /metrics/events exports nothing at all. This is not hypothetical: it is what happens with any integration that talks to the REST API directly rather than through an instrumented SDK.
2. Even when clients do report, the number describes the client's experience, not the server's work, so it cannot be used to tell a slow server from a slow network.

What is missing

SandboxErrorCodes in services/constants.py is a ready-made taxonomy of ~40 failure modes — image pull failed, execd start failed, pod ready timeout, invalid volume backend, pool not found — and not one of them is ever counted. When creation starts failing in production, the server's metrics are silent; the signal exists only in logs.

Also absent: any count of lifecycle operations, any server-measured duration, any gauge of live sandboxes.

Note POST /sandboxes returns 202 Accepted and provisions asynchronously, so "create duration" has two distinct meanings — time to schedule, and time to ready. Only the first is measurable at the API boundary; the second needs instrumentation in the provisioning path. Worth deciding deliberately rather than by accident.

Suggested direction

Server-measured, additive, no change to the existing SDK-reported histogram:

sandbox.operation.duration{operation, outcome}          histogram, ms
sandbox.operation.total{operation, outcome, error_code}  counter

operation from a closed set of lifecycle verbs, outcome in success/error, error_code only present on failure and drawn from SandboxErrorCodes. Bounded cardinality, and it makes the existing error taxonomy visible without inventing a new one.

I have a PR for the API-boundary half of this. The time-to-ready half is a design question I would rather not answer unilaterally — happy to follow whatever you prefer.

[ferponse]

PR for the API-boundary half sent in #1412. The time-to-ready half is left open there on purpose — it needs a decision about where the readiness boundary sits, and I would rather follow your preference.