HDDS-16019. Publish httpfs gateway metrics to Prometheus via /prom endpoint

[arunk-kumar]

What changes were proposed in this pull request?
HttpFSServerWebServer builds an HttpServer2 directly rather than extending BaseHttpServer, so it had no /prom endpoint and could not be scraped by Prometheus. This meant httpfs metrics were invisible to any Prometheus-based monitoring, and httpfs was absent from Grafana dashboards that group by component.

Changes:

Extract the Prometheus servlet wiring from BaseHttpServer into a new public static helper method addPrometheusEndpoint(HttpServer2, ConfigurationSource, String). This lives in the same package as BaseHttpServer so it retains access to the package-private PROMETHEUS_SINK constant and HttpServer2.getWebAppContext().

Refactor BaseHttpServer to call the helper — no behaviour change for existing services.

Wire the helper into HttpFSServerWebServer: call addPrometheusEndpoint in the constructor (after HttpServer2 is built), register the returned sink with DefaultMetricsSystem in start(), and unregister in stop(). Gated by hdds.prometheus.endpoint.enabled (default: true), matching the behaviour of other services.

The security model is preserved: if hdds.prometheus.endpoint.token is set, /prom is added as an internal servlet (token-based auth, SPNEGO bypassed); otherwise it is a regular servlet protected by the server's auth filter.

This is a prerequisite for HDDS-15858 (add httpfs to the ZDU Rolling Upgrade Grafana dashboard), which will add the httpfs scrape target and build-info metrics on top.

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16019

Prereq for: HDDS-15858
Epic: HDDS-14496 (Zero Downtime Upgrade)

How was this patch tested?
Local verification against upstream/master:

mvn -pl :ozone-httpfsgateway,:hdds-server-framework -am install -DskipTests -DskipShade -DskipRecon -DskipDocs — builds clean.
mvn -pl :ozone-httpfsgateway,:hdds-server-framework checkstyle:check — 0 violations on both modules.
mvn -pl :ozone-httpfsgateway,:hdds-server-framework apache-rat:check — 0 unapproved licences.
mvn -pl :ozone-httpfsgateway,:hdds-server-framework test — 573 tests, 0 failures (1 pre-existing skip in TestNetworkTopologyImpl).
TestBaseHttpServer and TestPrometheusServletAuthorization in hdds-server-framework
pass without modification, confirming the refactor does not change existing behaviour.

[arunk-kumar]

Verified end-to-end in the docker compose environment with monitoring.yaml included.

1. /prom endpoint on httpfs (port 14000) returns Prometheus-format metrics:

# TYPE httpfsserver_bytes_read counter
httpfsserver_bytes_read{sessionid="null",context="httpfs",hostname="0f6476c9d1f2"} 0
# TYPE http_server2_metrics_http_server_thread_count gauge
http_server2_metrics_http_server_thread_count{context="HttpServer2",server_name="webhdfs"} 8

2. Prometheus target health — httpfs scraped successfully:

"scrapeUrl": "http://httpfs:14000/prom?user.name=hadoop",
"health": "up",
"lastError": "",
"lastScrape": "2026-08-04T08:05:03.987196928Z"

3. httpfs up metric in Prometheus (value = 1):

{"__name__": "up", "component": "httpfs", "instance": "httpfs:14000", "job": "ozone-httpfs"} = 1

4. httpfs-specific metric queryable from Prometheus:

{"__name__": "httpfsserver_bytes_read", "component": "httpfs", "instance": "httpfs:14000"} = 0

Note on prometheus.yml: HttpFSAuthenticationFilter requires a user identity even in non-secure mode, so httpfs is configured as a separate ozone-httpfs job with params: user.name: [hadoop]. Other services don't need this because their /prom endpoints bypass the auth filter.

On acceptance tests: the existing prometheus_check.py smoketest validates all running Prometheus targets generically — it will automatically cover httpfs once this PR merges, with no changes to the test needed.

[chihsuan]

Thanks for the patch! @arunk-kumar I left two inline comments, mainly around consistency and lifecycle handling. Otherwise, it works in the default setup 👍

[chihsuan]

Would it make sense to omit this call? The Prometheus object is registered as a MetricsSink, while unregisterSource() only removes metrics sources, so it may not have any effect

[chihsuan]

I found that with hdds.prometheus.endpoint.token set, httpfs rejects the bearer-token request that s3g accepts. Same token:

curl -H 'Authorization: Bearer testtoken' http://httpfs:14000/prom  -> 401
curl -H 'Authorization: Bearer testtoken' http://s3g:19878/prom     -> 200

Is this difference intentional, or should HttpFS be updated for consistency?

[arunk-kumar]

@chihsuan Thanks for the review!

Issue 1 (unregisterSource): Fixed — removed the unregisterSource("prometheus") call from stop(). BaseHttpServer.stop() doesn't call it either; httpServer.stop() handles cleanup. Fix is in the latest commit.

Issue 2 (bearer token vs S3G): You're right that the behavior differs. The root cause is that HttpFSAuthenticationFilter intercepts all requests (including /prom) before PrometheusServlet gets a chance to check the bearer token — unlike S3G which doesn't have that filter. This seems like a pre-existing inconsistency rather than something introduced by this PR. Happy to file a follow-up JIRA to track aligning the bearer token behavior across services, if that would be useful.