Dual-emit legacy metric names behind --otel-use-legacy-metrics

[glageju]

Summary

Stacked on #5956. Base is gautam/metrics-std-toolhive, so this diff shows only the dual-emission layer. Review #5956 first — and the two must ship together: #5956 alone is the hard cutover this PR exists to avoid.

#5956 renames ToolHive's metrics to the stacklok.* vocabulary. Review blocked it on the same point twice: a rename empties a dashboard panel exactly the way a deletion does, and the renamed metrics have no overlap window — the old name stops and the new one starts in the same release, so there is no moment where a user can run both queries side by side and confirm they agree.

Span attributes already have machinery for exactly this: dual emission behind --otel-use-legacy-attributes, with #6072 tracking its retirement. Metrics went with a hard cutover instead, and the asymmetry wasn't justified. This PR closes it — metrics get the same treatment: a flag, a window, and a documented path.

After this PR, upgrading #5956 breaks nothing by default. 23 of the 24 pre-rename metric names are still emitted, under their original label keys. The exception is toolhive_vmcp_backends_discovered, whose replacement is a live health gauge rather than a fire-once count, so there is nothing to reproduce — that one panel has to be migrated rather than waited out.

• Adds the dual-emit seam (pkg/telemetry/legacymetrics.go): helpers that return a working instrument when enabled and a no-op otherwise, so the record sites stay branch-free instead of growing an if each. An instrument-creation failure also yields a no-op — a legacy alias must never be the reason a process fails to start.
• Threads UseLegacyMetrics end to end — CLI flag, config file, operator CRD, thv serve, and vMCP — defaulting to true at every layer.
• Emits legacy aliases for 23 of the 24 old names across proxy, rate-limit, and vMCP metrics, including the six §3.5 semconv twins that Standardize proxy + vMCP metrics to stacklok.*, delete legacy twins #5956 deletes outright. Those cost nothing new to keep: they already coexist with their semconv replacements today.
• Aliases keep their original label keys. This matters more than it looks: the rate-limit metrics change both name and label (toolhive_rate_limit_decisions{server=…} → stacklok_toolhive_ratelimit_decisions_total{mcp_server=…}), so those dashboards break twice over. An alias emitting the old name with the new label key would be a second breaking change, not a compatibility path — so where the key vocabularies diverge, the legacy attribute set is built separately rather than shared.
• Un-merges the outcome-labelled counters. Standardize proxy + vMCP metrics to stacklok.*, delete legacy twins #5956 folds four _errors/_not_found counters into an outcome label (RFC D4). The aliases fan back out to the original separate counters, and find_tool deliberately keeps its legacy counter incrementing before the call — the original counted attempts, not completions, so it's reproduced as-was rather than silently changing meaning during the overlap window.
• Reframes the migration guide from a one-shot cutover into a deprecation schedule, with the old → new mapping and the removal plan.

Type of change

• New feature

Test plan

• Unit tests (task test) — 259 packages, race detector on
• Linting (task lint-fix) — 0 issues
• Manual testing (described below)

Verified against a running proxy with --otel-enable-prometheus-metrics-path:

• Default (flag on) — both name families present in /metrics.
• --otel-use-legacy-metrics=false — zero ^toolhive_ series; new names still present.
• Label compatibility — toolhive_rate_limit_decisions_total still carries server="…", not mcp_server="…". This is the check that would catch the "breaks twice over" regression, and an unanchored assertion can't catch it: server="x" is a substring of mcp_server="x".
• Un-merge — a not-found call_tool increments both the new outcome="not_found" series and the legacy _not_found counter.

The un-merge is now also pinned by TestTelemetryOptimizer_LegacyUnmergeFansOutToSeparateCounters, verified load-bearing by sabotage: wiring not_found onto the errors counter fails it, and so does renaming tool_name to mcp_tool_name — the regression an unanchored assertion cannot catch. The negative legacy-absence assertions now set the flag explicitly rather than leaning on the Go zero value; flipping it fails all five, so they can actually fail.

API Compatibility

• This PR does not break the v1beta1 API, OR the api-break-allowed label is applied and the migration guidance is described above.

What this PR changes. One optional field, MCPTelemetryOTelConfig.UseLegacyMetrics, with +kubebuilder:default=true. Purely additive — no field is removed, and existing CRs are unaffected. Both it and the adjacent useLegacyAttributes are *bool with omitempty, so an unset key stays distinguishable from an explicit false rather than resting on admission having stamped the default. The generated CRD schema is byte-identical either way — *bool + omitempty + a kubebuilder default emits the same OpenAPI as a plain bool — so there is no schema churn and no stored CR is rewritten. This PR introduces no API break, and no migration is required for it.

Why the label is applied anyway. The check fails for a pre-existing reason unrelated to this diff, so the three points above have no break of this PR's to describe:

1. What is changing: nothing, in this PR. The reported removals are status.referencingWorkloads and status.referenceCount across six config CRDs (mcpauthzconfigs, mcpexternalauthconfigs, mcpoidcconfigs, mcptelemetryconfigs, mcptoolconfigs, mcpwebhookconfigs).
2. Why it is unavoidable here: the check baselines against the latest release tag (v0.41.0), and Remove referencingWorkloads/referenceCount from config CRD statuses #5631 removed those fields after that tag was cut. Every PR touching deploy/charts/operator-crds/files/crds/** inherits the errors; this PR touches that path only to add the field above. referencingWorkloads appears zero times on this branch and zero times on main, and twice per affected CRD in v0.41.0.
3. Migration path: none from this PR. Remove referencingWorkloads/referenceCount from config CRD statuses #5631 owns the removal and its own migration guidance.

The label should be removed once the next release moves the baseline past #5631, at which point the check goes green on its own for every PR.

Changes

File	Change
pkg/telemetry/legacymetrics.go	New. The three dual-emit helpers
pkg/telemetry/config.go	Config.UseLegacyMetrics, default, (*Provider).UseLegacyMetrics() accessor
pkg/telemetry/middleware.go	Proxy legacy aliases, including the SSE-path mcp_method="sse_connection" series
pkg/telemetry/serve.go	Flag default on thv serve
cmd/thv/app/run_flags.go	--otel-use-legacy-metrics; resolveOptionalBool helper; passes the resolved toggles to WithTelemetryConfigFromFlags
pkg/config/config.go	OpenTelemetryConfig.UseLegacyMetrics
pkg/runner/{config,telemetry_config,config_builder,middleware}.go	Threading; nil → true; ReadJSON defaulting for configs persisted before the toggles existed
pkg/ratelimit/{observability,limiter,middleware}.go	Rate-limit aliases keeping the server label
pkg/vmcp/core/core_telemetry.go	Composite-tool aliases
pkg/vmcp/internal/backendtelemetry/backendtelemetry.go	Backend aliases; revision_reclassifications alias; legacy attribute set snapshotted from commonAttrs
pkg/vmcp/server/sessionmanager/factory.go	9 optimizer aliases built once outside the per-session closure; outcome un-merge
pkg/vmcp/{cli/serve,ratelimit/factory/limiter}.go	vMCP threading
cmd/thv-operator/pkg/spectoconfig/telemetry.go	Reads both legacy toggles outside the openTelemetry.enabled guard, as *bool so unset ≠ false
pkg/vmcp/config/{defaults,yaml_loader}.go	Load-time defaulting so an omitted YAML key means true, not false
cmd/thv-operator/**, deploy/charts/**, docs/{cli,operator,server}/**	CRD field + regenerated
docs/telemetry-migration-guide.md, docs/observability.md	Deprecation schedule

Does this introduce a user-facing change?

Yes, and it makes #5956's change non-breaking by default.

New flag --otel-use-legacy-metrics (also use-legacy-metrics in config, spec.otel.useLegacyMetrics on the CRD), defaulting to true. While on, ToolHive emits both the new stacklok.* metric names and the pre-rename toolhive_* names with their original label keys, so existing dashboards and alerts keep working across the upgrade.

Migrate at your own pace: run the old and new queries side by side, confirm they agree, then set the flag to false to drop the legacy series. The flag and the legacy names will be removed in a later minor — see docs/telemetry-migration-guide.md for the old → new mapping and the schedule.

Dual emission roughly doubles series count for the metrics involved while enabled; histograms are the expensive case.

Special notes for reviewers

On RFC D13 ("no dual-emit window"). Reviewers on #5956 raised this and I agreed: a
dual-emit window is necessary for ToolHive OSS, so that users' existing dashboards don't
break the moment they upgrade. D13 argued the legacy twins aren't scraped in a default
shipped deployment — but this repo ships four Grafana dashboards under
examples/otel/grafana-dashboards/, and D13's own closing sentence anticipates exactly
this case: "If a scraped consumer is identified before the rename lands, revisit this
decision for that surface."

The RFC is unedited here; if D13 should now read as allowing the window, that's a
one-line amendment rather than a code change.

Seven config paths reach the dual-emission layer with the wrong value. Each
defeats the compatibility window on a real deployment, and all seven are fixed
here rather than left for a follow-up, since a flag that silently ignores you is
worse than no flag:

1. thv run re-derived telemetry from raw flag values after resolving them, so a
   config-file use-legacy-metrics: false was flipped back to true in the persisted
   RunConfig — and again in rate-limit middleware, giving a half-disabled state.
2. The operator read both toggles inside the openTelemetry.enabled guard, so a
   Prometheus-only MCPTelemetryConfig fell to the Go zero value and dropped every
   toolhive_* series on upgrade. Prometheus scraping is where a legacy dashboard is
   most likely to exist, so this defeated the feature exactly where it mattered most.
   An explicit useLegacyMetrics: true was discarded the same way.
3. Standalone vMCP unmarshalled a plain bool straight from YAML with no defaulting
   layer, so any telemetry: block omitting the key disabled dual emission silently.
4. toolhive_mcp_requests was incremented from two sites pre-rename; the SSE branch
   returns early, so the mcp_method="sse_connection" series read zero.
5. Persisted RunConfig has no defaulting, so a runconfig.json written before these
   toggles existed decodes both as the Go zero value false — dropping every toolhive_*
   series on the first restart after upgrade, for exactly the pre-upgrade workloads the
   window exists to protect. Live on thv restart, thv run --from-config, and
   thv-proxyrunner. ReadJSON re-probes the raw JSON to tell an absent key from an
   explicit false.
6. The toggles are persisted twice, and the copy that configures the provider is the
   nested one. Runner.Run skips PopulateMiddlewareConfigs whenever
   middleware_configs is already populated, and CreateMiddleware builds the provider
   from the parameters inside that entry rather than from the top-level
   telemetry_config. Both the CLI builder and the operator persist it, so healing the
   top-level block alone leaves the record sites reading false. The re-probe covers the
   telemetry and rate-limit entries too, matching on middleware type and leaving
   parameters it cannot decode untouched.
7. Rate-limit params carried use_legacy_metrics as omitempty, which erased an
   explicit false on the way out and made it indistinguishable from a config predating
   the field — so the re-probe in 6 would turn emission back on against the user's wishes.
   The option is dropped so the value always round-trips.

Scope note. Fixes 2, 3, 5, and 6 also correct the same pre-existing bugs for
useLegacyAttributes. The two toggles sit on adjacent lines and share the
defaulting path, so the fix is symmetric and lands together rather than
splitting a two-line change across PRs.

Deliberate design choices, in case they look wrong at a glance:

• Aliases carry the old label keys, not the new ones. Looks like an inconsistency in the diff; it's the whole point. See the rate-limit case above.
• find_tool's legacy counter increments before the call. Also looks inconsistent next to the new outcome counter, which increments after. The original counted attempts, so reproducing it faithfully means keeping that placement.
• legacymetrics.go takes the metric name as a parameter and contains no name literals, so the seam has no opinion about what's being aliased.
• token_savings is the one row where the rename is nearly invisible. Both the new metric and its alias carry unit %, so they export as stacklok_vmcp_optimizer_token_savings_percent and toolhive_vmcp_optimizer_token_savings_percent — the new Prometheus name differs from the old only in prefix, and the exporter does not double the _percent suffix on the alias. Settled rather than assumed: confirmed against a live Prometheus exporter, and the guard is otlptranslator's !strings.HasSuffix(name, mainUnitSuffix) check in metric_namer.go. The metric name still changed, so the query does need migrating — only the suffix coincides.

Still open: the removal release is unnamed. The guide's deprecation timeline says "current release / future release / later release" rather than literal versions, matching how useLegacyAttributes is already documented. Two things depend on settling that version, and I'd rather settle it before shipping either: pinning actual release numbers in the timeline, and the startup deprecation warning raised in review. A warning that says "a future release" gives users nothing to plan against, so I left it out pending a number. Happy to add both here.

Known gap, not fixed here (pre-existing, and orthogonal to dual emission): stacklok.build_info exports as stacklok_build_info_ratio, because coremetrics.RegisterBuildInfo sets metric.WithUnit("1") and the Prometheus translator appends a unit suffix. _ratio is wrong for an identity gauge, and it means a dashboard joining on stacklok_build_info returns empty. The durable fix is upstream in toolhive-core. Happy to file it, or fold it in here if you'd rather it not reach customer dashboards at all.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 85.50296% with 49 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.72%. Comparing base (d4e44a2) to head (ac8862a).

Files with missing lines	Patch %	Lines
pkg/runner/config.go	65.38%	9 Missing and 9 partials ⚠️
pkg/telemetry/legacymetrics.go	55.55%	9 Missing and 3 partials ⚠️
pkg/vmcp/cli/serve.go	0.00%	8 Missing ⚠️
pkg/runner/middleware.go	77.77%	1 Missing and 1 partial ⚠️
pkg/telemetry/middleware.go	97.18%	1 Missing and 1 partial ⚠️
pkg/telemetry/serve.go	50.00%	2 Missing ⚠️
pkg/vmcp/config/defaults.go	80.00%	1 Missing and 1 partial ⚠️
pkg/vmcp/server/sessionmanager/factory.go	96.77%	2 Missing ⚠️
pkg/telemetry/config.go	83.33%	1 Missing ⚠️

Additional details and impacted files

@@                       Coverage Diff                       @@
##           gautam/metrics-std-toolhive    #6168      +/-   ##
===============================================================
+ Coverage                        72.64%   72.72%   +0.08%     
===============================================================
  Files                              736      737       +1     
  Lines                            76447    76748     +301     
===============================================================
+ Hits                             55533    55816     +283     
+ Misses                           16997    16985      -12     
- Partials                          3917     3947      +30     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.