feat(vm): export driver traces over OTLP (#2564)

Continue distributed traces across the gateway-to-driver process boundary and
export VM driver spans to the same OTLP/gRPC collector. The driver reports as
the distinct openshell-driver-vm service.

Updated the gateway architecture and configuration reference with a generic
external-driver forwarding contract.

Instrumented:

- Every RemoteComputeDriver RPC injects the active W3C trace context into
  tonic metadata. Managed VM readiness and runtime initialization give startup
  capability probes stable parent operations rather than isolated root spans.
- A tonic service layer creates fixed, low-cardinality server spans for every
  ComputeDriver RPC. New handlers inherit tracing automatically; failures record
  OpenTelemetry error status and the gRPC status code.
- Background provisioning remains attached to CreateSandbox after the RPC
  returns without extending the RPC span lifetime.
- Provisioning records image preparation, bootstrap image resolution, overlay
  preparation, lifecycle configuration, pre-launch hooks, guest preparation,
  and launcher spawn as child spans.
- VM startup reconciliation roots one trace for the persisted-sandbox scan,
  with per-sandbox restore and provision operations beneath it. The root remains
  open until all spawned restore tasks finish.
- Delete cleanup records its own child operation.

Design notes:

- The gateway forwards its configured OTLP endpoint to managed external
  drivers. SDK `OTEL_*` variables continue to own sampling, batching, limits,
  headers, and transport tuning.
- The VM driver has its own tracer provider and service resource so trace
  backends preserve the service boundary.
- RPC operation names come from an explicit method mapping, keeping cardinality
  bounded without parsing the protobuf descriptor set at runtime.
- Propagation uses a remote SpanContext for spawned provisioning. This keeps
  one trace while allowing the CreateSandbox server span to finish when the
  RPC response is sent.
- Startup restoration is independent of gateway requests. It begins at the VM
  driver reconciliation span rather than attaching to an unrelated RPC.
- Existing tracing events remain on the logging path. The OpenTelemetry layer
  exports spans only and excludes the SDK exporter callsites to avoid recursive
  traces.
- Export configuration failures do not prevent the driver from serving, and
  buffered spans are drained during graceful shutdown.
- Trace fields identify drivers, sandboxes, images, lifecycle phases, and gRPC
  outcomes without recording credentials, sandbox tokens, or request query
  parameters.

Refs #2507

Signed-off-by: Kris Hicks <khicks@nvidia.com>

Author: krishicks

Cargo.lock

@@ -637,17 +637,21 @@ export: the table's presence is the on-switch, and `OTEL_EXPORTER_OTLP_ENDPOINT`
 is ignored so enablement has a single source. TOML decides whether and where
 to export; the SDK's `OTEL_*` variables tune how. Transport is OTLP over gRPC
 only. Shared provider, resource, and tracing-layer construction lives in
-`openshell-otel`.
-
-Span emission requires no per-handler instrumentation. The `tower_http`
-`TraceLayer` in `multiplex.rs` opens a span per inbound request, and that span
-continues incoming W3C trace context when present or starts a new trace
-otherwise. It is named for the RPC and carries the request ID that also appears
-in the gateway's logs — the identifier that lets an operator pivot between a
-trace and its log lines. Store and compute-driver spans become children of the
-request span. Reconciliation, provider refresh, and driver-watch loops create
-their own operation spans because they have no inbound request to provide a
-parent. gRPC status is recorded when response trailers arrive.
+`openshell-otel`, along with shared HTTP/tonic trace-context propagation and
+gRPC failure recording.
+
+The `tower_http` `TraceLayer` in `multiplex.rs` opens a span per inbound request,
+and that span continues incoming W3C trace context when present or starts a new
+trace otherwise. It is named for the RPC and carries the request ID that also
+appears in the gateway's logs — the identifier that lets an operator pivot
+between a trace and its log lines. Store and compute-driver spans become
+children of the request span. Reconciliation, provider refresh, and
+driver-watch loops create their own operation spans because they have no
+inbound request to provide a parent. gRPC status is recorded when response
+trailers arrive.
+
+The gateway forwards OTLP configuration and W3C trace context to managed
+external drivers. Each driver exports under its own service name.
 
 Two invariants shape the failure behavior. Telemetry is diagnostic, so no OTLP
 failure stops the gateway from serving: a malformed endpoint is logged at

architecture/gateway.md

@@ -20,12 +20,15 @@ path = "src/main.rs"
 
 [dependencies]
 openshell-core = { path = "../openshell-core", default-features = false }
+openshell-otel = { path = "../openshell-otel" }
 openshell-policy = { path = "../openshell-policy" }
 openshell-vfio = { path = "../openshell-vfio" }
 
 bollard = { version = "0.20", features = ["ssh"] }
 tokio = { workspace = true }
 tonic = { workspace = true, features = ["transport"] }
+tower-http = { workspace = true }
+http = { workspace = true }
 prost = { workspace = true }
 prost-types = { workspace = true }
 futures = { workspace = true }
@@ -34,6 +37,9 @@ nix = { workspace = true }
 clap = { workspace = true }
 tracing = { workspace = true }
 tracing-subscriber = { workspace = true }
+opentelemetry = { workspace = true }
+opentelemetry_sdk = { workspace = true }
+tracing-opentelemetry = { workspace = true }
 miette = { workspace = true }
 url = { workspace = true }
 serde = { workspace = true }
@@ -56,6 +62,9 @@ telemetry = ["openshell-core/telemetry"]
 
 [dev-dependencies]
 temp-env = "0.3"
+tempfile = "3"
+opentelemetry_sdk = { workspace = true, features = ["testing"] }
+opentelemetry-proto = { version = "0.32", default-features = false, features = ["gen-tonic", "trace"] }
 
 # smol-rs/polling drives the BSD/macOS parent-death detection in
 # procguard via kqueue's EVFILT_PROC / NOTE_EXIT filter. We could use

crates/openshell-driver-vm/Cargo.toml

@@ -208,7 +208,7 @@ RUST_LOG=openshell_server=debug,openshell_driver_vm=debug \
   mise run gateway:vm
 ```
 
-The VM guest's serial console is appended to `<state-dir>/<sandbox-id>/console.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `<state-dir>/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions, removes same-owner stale sockets, and the gateway removes the socket on clean shutdown via `ManagedDriverProcess::drop`. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development.
+The VM guest's serial console is appended to `<state-dir>/<sandbox-id>/console.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `<state-dir>/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions and removes same-owner stale sockets. On clean shutdown, the gateway sends the managed driver `SIGTERM`, waits up to five seconds for it to flush telemetry and exit, then force-kills it if necessary and removes the socket. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development.
 
 ## Host-side nftables rules