feat: add per-request trace IDs and system-error metrics

[charleswool]

Description

Improves log correlation and error observability for the provider's request path.

Now rebased directly onto main — the metrics wiring it depended on landed with #2848, and the now-empty #2847 was closed. This PR is a single commit against main.

Change

• Per-request trace IDs: attach a generated trace ID to each verify/mutate request context via logger.InitContext. It propagates to all context-aware downstream logging — executor, verifiers, auth providers, policy providers — so a single request's logs correlate. (Reading the trace ID from an incoming request header is supported by internal/logger but requires InitLogConfig to register the header names, which this binary does not call today; that can be wired up in a follow-up.)
• Context-aware handler logging: the handler cache-warning logs now use the context logger, so they carry the trace ID too.
• System-error metric: count handler/processing failures (executor validate/resolve errors, reference parse errors) with metrics.ReportSystemError, using stable, low-cardinality category labels (verify_artifact, mutate_parse_reference, mutate_resolve_reference) so Prometheus label cardinality stays bounded. The full error text is still returned to the caller in the response item.

Further metric-coverage expansion (outcome labels on request histograms, cache hit/miss and registry/downstream counters) is left as a follow-up.

#2873 builds on this branch and adds request-outcome logging.

Testing

• go build ./..., go vet, package tests, and golangci-lint pass. ReportSystemError is nil-guarded so it is a no-op when metrics are disabled.

[codecov]

Codecov Report

❌ Patch coverage is 42.85714% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.86%. Comparing base (6ead8b6) to head (224baea).

Files with missing lines	Patch %	Lines
internal/httpserver/handlers.go	60.00%	2 Missing ⚠️
internal/httpserver/server.go	0.00%	2 Missing ⚠️

❌ Your patch check has failed because the patch coverage (42.85%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2849      +/-   ##
==========================================
- Coverage   76.89%   76.86%   -0.04%     
==========================================
  Files          90       90              
  Lines        4276     4279       +3     
==========================================
+ Hits         3288     3289       +1     
- Misses        831      832       +1     
- Partials      157      158       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds request-scoped observability to the Gatekeeper provider request path by initializing per-request logging context (trace IDs) and expanding metrics reporting for request durations and handler/system errors.

Changes:

• Initialize request context in verify/mutate HTTP handlers via logger.InitContext to enable per-request trace IDs in downstream context-aware logs.
• Add request duration reporting for verify/mutate and report handler/system errors via metrics.ReportSystemError.
• Wire metrics exporter initialization into the Gatekeeper provider binary behind a new --metrics-port flag, with accompanying test updates.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
internal/httpserver/server.go	Initializes request context for verify/mutate handlers to propagate trace IDs.
internal/httpserver/handlers.go	Adds request-duration metrics, context-aware cache warning logs, and system-error metrics reporting.
cmd/ratify-gatekeeper-provider/main.go	Adds metrics exporter initialization via --metrics-port.
cmd/ratify-gatekeeper-provider/main_test.go	Updates tests for metrics init wiring and default flag parsing.

Comments suppressed due to low confidence (2)

internal/httpserver/handlers.go:122

• metrics.ReportSystemError records the provided string as a metric attribute. Using the full parse error message (item.Error) will produce unbounded label cardinality (different user inputs / error details). Use a stable category (the detailed error is already returned to the caller via item.Error).

		metrics.ReportSystemError(ctx, item.Error)

internal/httpserver/handlers.go:159

• metrics.ReportSystemError uses the string as a metric label; sending err.Error() here can produce high-cardinality metrics (downstream error text often varies). Use a stable category string instead and keep the full error in logs/response payloads.

		metrics.ReportSystemError(ctx, err.Error())

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[charleswool]

Correct - InitLogConfig is not called in this binary, so trace IDs are generated per request, not read from incoming headers. I have fixed the PR description to say generated; header-based propagation can be wired later by calling InitLogConfig with the header names.

[charleswool]

Good catch - switched ReportSystemError to stable category labels (verify_artifact, mutate_parse_reference, mutate_resolve_reference) to keep Prometheus label cardinality bounded. The full error text still goes into the response item returned to the caller.

[charleswool]

Addressed in the metrics PR this is stacked on (#2847): the non-fatal case now uses a sentinel error and asserts via errors.Is that startRatify does not return the metrics-init error. This branch was rebuilt on that updated commit.