Standardize proxy + vMCP metrics to stacklok.*, delete legacy twins

[glageju]

Summary

The proxy and vMCP used toolhive_mcp_* / toolhive_vmcp_* metric names, emitted six legacy metrics alongside their emerging-semconv replacements, and diverged from the platform label vocabulary. This finishes the semconv migration and adopts the shared toolhive-core vocabulary, per the Metrics Standardization RFC.

• Proxy (pkg/telemetry/): toolhive_mcp_active_connections → stacklok.toolhive.proxy.active_connections; build_info via coremetrics.RegisterBuildInfo; added http.server.request.duration (OTel HTTP semconv) so deleting the legacy MCP twins doesn't drop transport-level coverage; D8 ownership labels (stacklok_component=toolhive, stacklok_product=stacklok-platform) promoted to every series via WithResourceAsConstantLabels and hardened as the final OTEL resource detector so a CLI custom attribute or OTEL_RESOURCE_ATTRIBUTES entry for those keys cannot override the frozen values.
• vMCP (pkg/vmcp/): toolhive_vmcp_workflow_* → stacklok.vmcp.composite_tool.*; backend mcp_server.health gauge rename to a live per-(mcp_server, state) gauge, keyed by workload ID for consistency with the backend registry and health-status provider; optimizer renames + token_savings.
• ⚠️ Breaking: six legacy semconv twins deleted (toolhive_mcp_requests, toolhive_mcp_request_duration, toolhive_mcp_tool_calls, toolhive_vmcp_backend_requests, toolhive_vmcp_backend_errors, toolhive_vmcp_backend_requests_duration). Their semconv replacements (mcp.server.operation.duration, mcp.client.operation.duration, http.server.request.duration) are kept.
• Bumps the toolhive-core dependency to v0.0.34, the release carrying telemetry/metrics (adopts core label keys, outcome values, and bucket presets). The transitive AWS-SDK and grpc bumps in go.mod come from toolhive-core's own requirements.
• Updates integration and e2e tests (test/integration/vmcp/, test/e2e/) that asserted the deleted/renamed metric names and old label scheme (mcp_method→mcp_method_name, server→mcp_server, status→HTTP status code) to assert the semconv replacements.
• Updates the Grafana example dashboards and docs/telemetry-migration-guide.md for the renamed/deleted metrics, including a complete old→new metric and label mapping and a corrected MCP Request Rate vs. total HTTP request-rate query split.
• Enables the Grafana dashboard sidecar in the example otel dev stack (examples/otel/prometheus-stack-values.yaml) so the updated example dashboards can be auto-provisioned into a local Kind cluster via a labeled ConfigMap, for validating the renamed metrics land correctly end-to-end.
• Extends the rename to pkg/ratelimit/observability.go, missed by the original pass: toolhive_rate_limit_decisions/_redis_errors/_check_latency → stacklok.toolhive.ratelimit.{decisions,redis_errors,check_latency}, with the migration guide and docs/observability.md updated to match.
• Renames toolhive_vmcp_backend_revision_reclassifications (added independently on main after this branch forked) to stacklok.vmcp.backend.revision_reclassifications so it doesn't reintroduce the legacy naming scheme this PR retires.
• Clarifies docs/telemetry-migration-guide.md's Backward Compatibility section: span attributes and metrics follow different policies (dual-emission vs. hard cutover) for different reasons — the guide previously implied one uniform strategy.

Type of change

• New feature

Test plan

• Unit tests (task test)
• Linting (task lint-fix)
• Manual testing (describe below)

Verified the shipped standalone path (GOWORK=off, published toolhive-core v0.0.34, no workspace):

• GOWORK=off go build ./... — clean.
• GOWORK=off go test ./pkg/telemetry/... ./pkg/vmcp/... — all packages pass, including the D8 label-promotion assertion in pkg/telemetry/buildinfo_test.go (stacklok_component="toolhive" / stacklok_product="stacklok-platform" promoted to per-series labels) and the updated TestVMCPServer_Telemetry_CompositeToolMetrics integration test.
• task test (full suite) is green.
• The e2e telemetry tests (test/e2e/) were updated for the new metric/label vocabulary and compile clean (go vet); they require a full task test-e2e run (containers) to exercise end-to-end.

API Compatibility

• This PR does not break the v1beta1 API, OR the api-break-allowed label is applied and the migration guidance is described above.

Does this introduce a user-facing change?

Yes. Prometheus metric names emitted by the proxy, vMCP, and rate limiting change to the stacklok.* namespace, and six legacy toolhive_* twins are removed. Their OTel-semconv replacements cover the same signals; docs/telemetry-migration-guide.md maps old → new. A customer scraping the old names loses those series (RFC D13 accepts this: the deleted twins are not scraped in a default shipped deployment).

Special notes for reviewers

• Depends on toolhive-core v0.0.34 (the shared-vocabulary release); this PR's go.mod pins it.
• toolhive-core v0.0.34 trims the Component* value roster a pre-release build referenced; toolhive defines its own stacklok.component = "toolhive" constant (pkg/telemetry/middleware.go, pkg/telemetry/providers/providers.go), asserted by pkg/telemetry/buildinfo_test.go.
• pkg/vmcp/internal/backendtelemetry/backendtelemetry.go's health gauge resolves each backend's status with a three-tier precedence — a live health.StatusProvider first, then request-outcome tracking, then the registry's discovery-time snapshot — matching the same precedence filterHealthyBackends uses for capability aggregation. core.New unregisters the gauge's callback on every construction-failure path after registration succeeds, and on Close(), so a partially-constructed coreVMCP never leaks a callback against the shared meter provider.

Links

• RFC: Stacklok Platform Metrics Standardization (D1, D2, D5, D6, D8, D11; §3.5 twin deletion).
• Telemetry migration guide — old→new metric/label mapping and upgrade steps.
• Telemetry: retire legacy span attribute dual-emission (useLegacyAttributes) #6072 — follow-up to retire the separate useLegacyAttributes span-attribute dual-emission flag (out of scope here; metrics/labels in this PR use a hard cutover with no such flag).

Ref: https://github.com/stacklok/stacklok-enterprise-platform/issues/1380
Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.57798% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.28%. Comparing base (57e681f) to head (b27b63e).

Files with missing lines	Patch %	Lines
pkg/telemetry/middleware.go	86.95%	5 Missing and 1 partial ⚠️
pkg/vmcp/core/core_vmcp.go	88.88%	2 Missing and 1 partial ⚠️
...vmcp/internal/backendtelemetry/backendtelemetry.go	95.38%	3 Missing ⚠️
pkg/telemetry/providers/providers.go	90.90%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5956      +/-   ##
==========================================
+ Coverage   72.24%   72.28%   +0.03%     
==========================================
  Files         722      722              
  Lines       75125    75152      +27     
==========================================
+ Hits        54277    54321      +44     
+ Misses      16985    16981       -4     
+ Partials     3863     3850      -13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[glageju]

Review Summary

Reviewed the metrics-vocabulary migration end to end: the proxy/vMCP rename to stacklok.*, the six-metric deletion, D8 ownership-label hardening, and the vMCP backend health-gauge rework.

Findings addressed in 045ccad

• mcp_server label missing from mcp.client.operation.duration's documentation in two places (docs/operator/virtualmcpserver-observability.md, docs/telemetry-migration-guide.md)
• currentHealthStatus's doc comment (pkg/vmcp/internal/backendtelemetry/backendtelemetry.go) described the wrong divergence condition from filterHealthyBackends — corrected to state it occurs when provider == nil and record() has an entry, not the "provider set but not yet tracking" case it originally claimed
• Grafana sidecar searchNamespace: ALL (examples/otel/prometheus-stack-values.yaml) had no RBAC note or ConfigMap setup steps — scoped to the monitoring namespace with a comment on the tradeoff, and added the kubectl create configmap/label steps to examples/otel/README.md
• D8 ownership-label wiring in providers_strategy.go had no test through its actual UnifiedMeterStrategy.CreateMeterProvider construction path — added TestUnifiedMeterStrategy_PrometheusOwnershipLabels
• toolhive_vmcp_backend_revision_reclassifications's rename was missing from the migration guide's mapping table

Verified, not bugs

• Using OTel dotted attribute-key constants (stacklok.component) as Prometheus label-map keys in providers_strategy.go does not panic MustRegister as initially suspected — prometheus/common's UTF8Validation default (active in this repo's pinned v0.67.5) accepts dots and escapes them to underscore form at scrape time. Confirmed by reproducing the exact registration + scrape against this repo's pinned dependencies.
• The toolhive-core v0.0.34 pin in go.mod is not visible as a diff hunk because it already landed via an earlier merge from main — the branch builds and compiles cleanly.

Overall

Callback lifecycle (register → unregister-on-every-construction-failure-path → idempotent Close()), the D8 label-override hardening, and the six metric deletions/renames are all correctly implemented and backed by tests that exercise the actual failure paths, not just the happy path — including a regression test that scrapes a live Prometheus handler to prove no gauge series leak after a construction failure.

---

Generated with Claude Code