Standardize proxy + vMCP metrics to stacklok.*, delete legacy twins

docs/observability.md

@@ -188,37 +188,31 @@ For VirtualMCPServer telemetry, see the
 ### MCP Proxy Metrics
 
 These metrics are emitted by the telemetry middleware (`pkg/telemetry/middleware.go`)
-for each MCP server proxy.
+for each MCP server proxy. Metric and label names follow the shared
+`stacklok.*`/OTel semantic-convention vocabulary (Metrics Standardization RFC);
+see the [Telemetry Migration Guide](./telemetry-migration-guide.md) for the
+mapping from the legacy `toolhive_mcp_*` names these replace.
 
-#### `toolhive_mcp_requests` (Counter)
+#### `stacklok.toolhive.proxy.active_connections` (UpDownCounter)
 
-Total number of MCP requests processed.
+Number of currently active MCP connections. Prometheus exposes this as
+`stacklok_toolhive_proxy_active_connections`.
 
 | Attribute | Type | Description |
 |-----------|------|-------------|
-| `method` | string | HTTP method (`POST`, `GET`) |
-| `status_code` | string | HTTP status code (`200`, `500`) |
-| `status` | string | `"success"` or `"error"` (error if status >= 400) |
-| `mcp_method` | string | MCP method name (`tools/call`, `resources/read`, etc.) |
-| `mcp_resource_id` | string | Tool name, resource URI, or prompt name |
-| `server` | string | MCP server name |
-| `transport` | string | Backend transport type (`stdio`, `sse`, `streamable-http`) |
-
-> **Note**: SSE connection establishment events also increment this counter
-> with `mcp_method="sse_connection"` and do not include `mcp_resource_id`.
-
-#### `toolhive_mcp_request_duration` (Histogram, seconds)
-
-Duration of MCP requests. Uses default histogram bucket boundaries.
-
-**Attributes**: Same as `toolhive_mcp_requests`.
+| `mcp_server` | string | MCP server name |
+| `transport` | string | Backend transport type |
+| `connection_type` | string | `"sse"` (only present for SSE connections) |
 
 #### `mcp.server.operation.duration` (Histogram, seconds)
 
 Duration of MCP server operations per the
 [OTEL MCP semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/mcp.md).
+Recorded only for requests with a resolvable MCP method (`tools/call`,
+`resources/read`, etc.) — GET (SSE stream open) and DELETE (session
+termination) requests carry no MCP method and are not recorded here.
 
-**Bucket boundaries**: `[0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 30, 60, 120, 300]`
+**Bucket boundaries** (`coremetrics.BucketsMCPProxy()`): `[0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300]`
 
 | Attribute | Type | Condition | Description |
 |-----------|------|-----------|-------------|
@@ -232,25 +226,65 @@ Duration of MCP server operations per the
 | `gen_ai.tool.name` | string | For `tools/call` | Tool name |
 | `gen_ai.prompt.name` | string | For `prompts/get` | Prompt name |
 
-#### `toolhive_mcp_tool_calls` (Counter)
+> **Note**: `mcp_resource_id` (tool name, resource URI, or prompt name) is
+> deliberately omitted from this metric's attributes to bound cardinality; it
+> feeds `gen_ai.tool.name`/`gen_ai.prompt.name` instead, which are themselves
+> only present for the method kinds that have a name to report.
 
-Total number of MCP tool invocations (only recorded for `tools/call` requests).
+#### `http.server.request.duration` (Histogram, seconds)
 
-| Attribute | Type | Description |
-|-----------|------|-------------|
-| `server` | string | MCP server name |
-| `tool` | string | Tool name |
-| `status` | string | `"success"` or `"error"` |
+Duration of every HTTP request the middleware handles, per the
+[OTEL HTTP semantic conventions](https://opentelemetry.io/docs/specs/semconv/http/).
+Recorded for every request — including SSE-open GETs and session-delete
+DELETEs that carry no MCP method — so transport-level coverage doesn't depend
+on a resolvable MCP method the way `mcp.server.operation.duration` does. For
+SSE connections, recorded once the connection closes (not per-chunk).
 
-#### `toolhive_mcp_active_connections` (UpDownCounter)
+**Bucket boundaries** (`coremetrics.BucketsFastHTTP()`): `[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]`
 
-Number of currently active MCP connections.
+| Attribute | Type | Condition | Description |
+|-----------|------|-----------|-------------|
+| `http.request.method` | string | Always | HTTP method (`GET`, `POST`, `DELETE`) |
+| `http.response.status_code` | int | Always | HTTP status code |
+| `error.type` | string | On HTTP 5xx | HTTP status code as string |
+
+#### `stacklok.build_info` (Gauge)
+
+Registered once per process via the shared `coremetrics.RegisterBuildInfo`
+helper (`pkg/telemetry/middleware.go`). Always reports `1`, carrying version
+metadata as attributes.
 
 | Attribute | Type | Description |
 |-----------|------|-------------|
-| `server` | string | MCP server name |
-| `transport` | string | Backend transport type |
-| `connection_type` | string | `"sse"` (only present for SSE connections) |
+| `component` | string | Always `"toolhive"`. A bare `component` label distinct from the promoted `stacklok_component` below — it exists so this metric's identity doesn't collide with the D8 constant label. |
+| `version` | string | ToolHive version |
+| `commit` | string | Build commit SHA |
+
+Like every other metric ToolHive emits, `stacklok.build_info` also carries
+`stacklok_component`/`stacklok_product` as constant labels — see
+[D8 Ownership Labels](#d8-ownership-labels). Those two are universal, not
+specific to this metric.
+
+### D8 Ownership Labels
+
+Every metric emitted by ToolHive carries `stacklok_component="toolhive"` and
+`stacklok_product="stacklok-platform"`. These are set as OTel resource
+attributes, applied as the last OTel resource detector so they cannot be
+overridden by `--otel-custom-attributes` or `OTEL_RESOURCE_ATTRIBUTES` (OTel
+resource merge is last-detector-wins). See `pkg/telemetry/providers/providers.go`.
+
+The Prometheus exporter is configured with `WithResourceAsConstantLabels` to
+promote exactly these two resource attributes onto every exported series as
+constant labels (`pkg/telemetry/providers/prometheus/prometheus.go`) — so in
+scraped Prometheus output they appear as per-series labels on every metric,
+not only on a separate `target_info`/`stacklok_build_info` line.
+
+The Go/process runtime metrics (`go_*`, `process_*`) are native Prometheus
+collectors registered directly on the raw registry rather than through the
+OTel SDK, so `WithResourceAsConstantLabels` does not reach them. They carry
+the same two labels via a separate `prometheus.WrapRegistererWith` applied
+only to the runtime-metrics registerer, so the D8 labels are consistently
+present on every series, including runtime/process metrics.
 
 ### Rate Limit Metrics
 
@@ -259,7 +293,7 @@ and VirtualMCPServer. Prometheus appends `_total` to counter names. The latency
 histogram is exported with the `_seconds` unit suffix and the standard
 `_bucket`, `_sum`, and `_count` series suffixes.
 
-#### `toolhive_rate_limit_decisions` (Counter)
+#### `stacklok.toolhive.ratelimit.decisions` (Counter)
 
 Total number of rate limit bucket decisions. An allowed request increments once
 for every applicable bucket. A rejected request increments only for the first
@@ -269,30 +303,30 @@ do not increment this counter.
 | Attribute | Type | Description |
 |-----------|------|-------------|
 | `namespace` | string | Kubernetes namespace associated with the server |
-| `server` | string | MCPServer or VirtualMCPServer name |
+| `mcp_server` | string | MCPServer or VirtualMCPServer name |
 | `decision` | string | `"allowed"` or `"rejected"` |
 | `scope` | string | `"shared"` or `"per_user"` |
 | `operation_type` | string | `"server"` or `"tool"` |
 
-#### `toolhive_rate_limit_redis_errors` (Counter)
+#### `stacklok.toolhive.ratelimit.redis_errors` (Counter)
 
 Total number of Redis errors encountered while checking rate limits.
 
 | Attribute | Type | Description |
 |-----------|------|-------------|
 | `namespace` | string | Kubernetes namespace associated with the server |
-| `server` | string | MCPServer or VirtualMCPServer name |
+| `mcp_server` | string | MCPServer or VirtualMCPServer name |
 | `error_type` | string | `"timeout"`, `"connection"`, `"auth"`, or `"other"` |
 
-#### `toolhive_rate_limit_check_latency` (Histogram, seconds)
+#### `stacklok.toolhive.ratelimit.check_latency` (Histogram, seconds)
 
 Duration of each attempted atomic Redis Lua rate limit check, including failed
 checks.
 
 | Attribute | Type | Description |
 |-----------|------|-------------|
 | `namespace` | string | Kubernetes namespace associated with the server |
-| `server` | string | MCPServer or VirtualMCPServer name |
+| `mcp_server` | string | MCPServer or VirtualMCPServer name |
 
 ## Span Attributes
 
@@ -399,7 +433,9 @@ Only variables explicitly listed in the configuration are captured.
 
 **Custom resource attributes** (`--otel-custom-attributes` or
 `OTEL_RESOURCE_ATTRIBUTES`): Key-value pairs added as OTEL resource attributes
-to all telemetry signals.
+to all telemetry signals. `stacklok.component`/`stacklok.product` are
+reserved and cannot be set this way — see
+[D8 Ownership Labels](#d8-ownership-labels).
 
 ### SSE Connection Attributes
 

docs/operator/virtualmcpserver-observability.md

@@ -24,91 +24,78 @@ The vMCP uses a decorator pattern to wrap backend clients and workflow executors
 with telemetry instrumentation. This approach provides consistent metrics and
 tracing without modifying the core business logic.
 
-The implementation of both metrics and traces can be found in `pkg/vmcp/server/telemetry.go`.
+The implementation of backend metrics and traces can be found in
+`pkg/vmcp/internal/backendtelemetry/backendtelemetry.go`; composite tool
+(workflow) telemetry is in `pkg/vmcp/core/core_telemetry.go` and
+`pkg/vmcp/server/sessionmanager/factory.go`.
 
 ## Metrics
 
-### Backend Metrics
-
-Backend metrics track requests to individual backend MCP servers.
+Metric and label names follow the shared `stacklok.*`/OTel semantic-convention
+vocabulary (Metrics Standardization RFC); see the
+[Telemetry Migration Guide](../telemetry-migration-guide.md) for the mapping
+from the legacy `toolhive_vmcp_*` names these replace.
 
-#### `toolhive_vmcp_backends_discovered` (Gauge)
+### Backend Metrics
 
-Number of backends discovered. Recorded once at startup.
+Backend metrics track requests to individual backend MCP servers
+(`pkg/vmcp/internal/backendtelemetry/backendtelemetry.go`).
 
-#### `toolhive_vmcp_backend_requests` (Counter)
+#### `stacklok.vmcp.mcp_server.health` (Gauge)
 
-Total number of requests sent to backend MCP servers.
+Per-backend health, re-derived from the live backend registry on every
+collection so a backend removed at runtime (e.g. via `list_changed`) stops
+being reported instead of leaving a stale series behind. Emits one point per
+`(mcp_server, state)` pair — the observed state reports `1`, the other `0`.
 
 | Attribute | Type | Description |
 |-----------|------|-------------|
-| `target.workload_id` | string | Backend workload ID |
-| `target.workload_name` | string | Backend workload name |
-| `target.base_url` | string | Backend base URL |
-| `target.transport_type` | string | Backend transport type (`stdio`, `sse`, `streamable-http`) |
-| `action` | string | Internal action name (`call_tool`, `read_resource`, `get_prompt`, `list_capabilities`) |
-| `mcp.method.name` | string | MCP method name (`tools/call`, `resources/read`, `prompts/get`, `list_capabilities`) |
-
-Method-specific attributes (added in addition to the above):
-
-| Attribute | Method | Description |
-|-----------|--------|-------------|
-| `tool_name` | `call_tool` | Tool name (ToolHive-specific) |
-| `gen_ai.tool.name` | `call_tool` | Tool name (OTEL MCP semconv) |
-| `resource_uri` | `read_resource` | Resource URI (ToolHive-specific) |
-| `mcp.resource.uri` | `read_resource` | Resource URI (OTEL MCP semconv) |
-| `prompt_name` | `get_prompt` | Prompt name (ToolHive-specific) |
-| `gen_ai.prompt.name` | `get_prompt` | Prompt name (OTEL MCP semconv) |
-
-#### `toolhive_vmcp_backend_errors` (Counter)
-
-Total number of errors from backend MCP servers.
-
-**Attributes**: Same as `toolhive_vmcp_backend_requests`.
-
-#### `toolhive_vmcp_backend_requests_duration` (Histogram, seconds)
-
-Duration of requests to backend MCP servers. Uses default histogram bucket
-boundaries.
-
-**Attributes**: Same as `toolhive_vmcp_backend_requests`.
+| `mcp_server` | string | Backend workload name |
+| `state` | string | One of `"healthy"`, `"degraded"`, `"unhealthy"`, `"unknown"`, `"unauthenticated"` |
 
 #### `mcp.client.operation.duration` (Histogram, seconds)
 
 Duration of MCP client operations per the
 [OTEL MCP semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/mcp.md).
 
-**Bucket boundaries**: `[0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 30, 60, 120, 300]`
+**Bucket boundaries** (`coremetrics.BucketsMCPProxy()`): `[0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300]`
 
 | Attribute | Type | Condition | Description |
 |-----------|------|-----------|-------------|
 | `mcp.method.name` | string | Always | MCP method name |
+| `mcp_server` | string | Always | Backend workload name |
 | `network.transport` | string | Always | `"tcp"` or `"pipe"` |
 | `error.type` | string | On error | Go error type (e.g., `*url.Error`) |
 
-### Workflow Metrics
+Backend operation spans carry additional ToolHive-specific and OTEL MCP
+semconv attributes (`target.workload_id`, `target.workload_name`,
+`target.base_url`, `target.transport_type`, `action`, `gen_ai.tool.name`,
+`mcp.resource.uri`, `gen_ai.prompt.name`) — see
+[Backend Operation Spans](#backend-operation-spans) below.
+
+### Composite Tool (Workflow) Metrics
 
-Workflow metrics track composite tool workflow executions.
+Composite tool metrics track workflow executions
+(`pkg/vmcp/core/core_telemetry.go`, `pkg/vmcp/server/sessionmanager/factory.go`).
 
-#### `toolhive_vmcp_workflow_executions` (Counter)
+#### `stacklok.vmcp.composite_tool.executions` (Counter)
 
-Total number of workflow executions.
+Total number of composite tool workflow executions, split by outcome.
 
 | Attribute | Type | Description |
 |-----------|------|-------------|
-| `workflow.name` | string | Workflow name |
+| `composite_tool` | string | Composite tool (workflow) name |
+| `outcome` | string | `"success"` or `"error"` |
 
-#### `toolhive_vmcp_workflow_errors` (Counter)
+#### `stacklok.vmcp.composite_tool.duration` (Histogram, seconds)
 
-Total number of workflow execution errors.
+Duration of composite tool workflow executions, recorded regardless of outcome.
 
-**Attributes**: Same as `toolhive_vmcp_workflow_executions`.
+**Bucket boundaries** (`coremetrics.BucketsMCPProxy()`): `[0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300]`
 
-#### `toolhive_vmcp_workflow_duration` (Histogram, seconds)
-
-Duration of workflow executions.
-
-**Attributes**: Same as `toolhive_vmcp_workflow_executions`.
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `composite_tool` | string | Composite tool (workflow) name |
 
 ## Distributed Tracing