[improve][monitor] Restore the BookKeeper Prometheus stats provider on the Prometheus Java client 1.x

[lhotari]

Motivation

io.prometheus:simpleclient 0.16.0 is the final release of the Prometheus Java client 0.x line. Pulsar
is still on it everywhere, and one of the things pinning us there is that the server distribution ships
org.apache.bookkeeper.stats:prometheus-metrics-provider, whose POM has hard compile dependencies on
simpleclient, simpleclient_hotspot and simpleclient_servlet.

Pulsar used to own this provider: it lived at jetty-upgrade/bookkeeper-prometheus-metrics-provider
until it was removed in 982eac3 ([feat] Upgrade BookKeeper dependency to 4.18.0, #25886), which
also repointed conf/bookkeeper.conf at BookKeeper's own provider. This PR restores it, rebuilt on the
Prometheus Java client 1.x, as the first concrete step off simpleclient.

Modifications

The module is restored at the repository root as bookkeeper-prometheus-metrics-provider rather than
under jetty-upgrade/, since it is no longer related to the Jetty upgrade work. The Gradle project name
:pulsar-bookkeeper-prometheus-metrics-provider and the
org.apache.pulsar.metrics.prometheus.bookkeeper package are unchanged from before the removal.

Wiring:

• conf/bookkeeper.conf — statsProviderClass points back at
  org.apache.pulsar.metrics.prometheus.bookkeeper.PrometheusMetricsProvider.
• distribution/server — ships this module instead of
  org.apache.bookkeeper.stats:prometheus-metrics-provider.
• settings.gradle.kts, gradle/libs.versions.toml — new module plus a prometheus-client-java version
  reference (1.8.0) and the four artifacts it needs.

API migration inside the module:

simpleclient 0.16	Prometheus Java client 1.8
CollectorRegistry	io.prometheus.metrics.model.registry.PrometheusRegistry
hotspot.{Standard,MemoryPools,GarbageCollector,Thread}Exports	io.prometheus.metrics.instrumentation.jvm.JvmMetrics
Gauge.build(...).create().setChild(...)	io.prometheus.metrics.core.metrics.GaugeWithCallback
Collector.sanitizeMetricName	PrometheusNaming.sanitizeMetricName
TextFormat.CONTENT_TYPE_004	PrometheusTextFormatWriter.CONTENT_TYPE

PrometheusTextFormat.writeMetricsCollectedByPrometheusClient now delegates to the client library's own
PrometheusTextFormatWriter instead of hand-rolling the rendering, so every snapshot type is emitted
exactly as the exposition format specifies. The hand-written gauge/counter/opStat output that this module
has always produced for BookKeeper's own stats is unchanged.

Note that prometheus-metrics-* 1.8.0 was already on the server classpath via
opentelemetry-exporter-prometheus, so this adds no new dependency family — only -core and
-instrumentation-jvm are newly bundled.

⚠️ Breaking: bookie JVM metric names change

The Prometheus Java client 1.x renamed several JVM metrics relative to the simpleclient hotspot exports.
Since conf/bookkeeper.conf currently selects BookKeeper's simpleclient-based provider, bookies emit the
old names today, and this change switches them:

before	after
jvm_memory_bytes_used	jvm_memory_used_bytes
jvm_memory_bytes_committed	jvm_memory_committed_bytes
jvm_memory_bytes_init	jvm_memory_init_bytes
jvm_memory_bytes_max	jvm_memory_max_bytes
jvm_memory_pool_bytes_used	jvm_memory_pool_used_bytes
jvm_memory_pool_bytes_committed	jvm_memory_pool_committed_bytes
jvm_memory_pool_bytes_init	jvm_memory_pool_init_bytes
jvm_memory_pool_bytes_max	jvm_memory_pool_max_bytes

jvm_gc_collection_seconds, jvm_threads_*, process_* and the Netty-tracked
jvm_memory_direct_bytes_used / jvm_memory_direct_bytes_max keep their names. Dashboards and alerts
on the renamed bookie JVM metrics need updating, so this should be called out in the release notes.

Two smaller output differences also come from the client library:

• metrics with no labels are written as name value rather than simpleclient's name{} value;
• a new jvm_runtime_info metric is exposed, whose label values contain spaces and + characters.

The restored tests are updated for both, including relaxing the test's own "hacky parsing" regex, whose
trailing-label-value character class could not represent a value like 21.0.11+10-LTS.

This does not remove simpleclient

To set expectations: simpleclient stays in the distribution after this change. 64 files across
pulsar-broker, pulsar-broker-common, pulsar-proxy, pulsar-metadata, managed-ledger,
pulsar-transaction, pulsar-common, pulsar-functions and the vendored ZooKeeper provider still use
it, and PrometheusMetricsGeneratorUtils scrapes CollectorRegistry.defaultRegistry. This module is the
first of four independent units; the ZooKeeper provider is the natural next one, and the
pulsar-functions instance metrics stack is what would unlock upgrading prometheus-jmx to 1.x.

Verifying this change

• Make sure that the change passes the CI checks.

This change is already covered by existing tests — the module's restored TestPrometheusFormatter and
TestPrometheusMetricsProvider, which pin the exposition text format and the direct-memory gauges.

Verified locally:

• ./gradlew :pulsar-bookkeeper-prometheus-metrics-provider:test — 11 tests, all green
• ./gradlew quickCheck sanityCheck
• ./gradlew checkBinaryLicense — confirms the BookKeeper provider jar is gone from the server
  distribution and the client_java jars that replaced it are accounted for
• Inspected the built apache-pulsar-*-bin.tar.gz to confirm it ships
  pulsar-bookkeeper-prometheus-metrics-provider and no longer ships
  org.apache.bookkeeper.stats-prometheus-metrics-provider

Does this pull request potentially affect one of the following parts:

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

conf/bookkeeper.conf's statsProviderClass default changes, and the bookie JVM metric names change as
described above.

[void-ptr974]

Thanks for working on this. I noticed a couple of compatibility details and wanted to share them for consideration.

First, around PrometheusNaming.sanitizeMetricName(...): I noticed it behaves differently from the previous Collector.sanitizeMetricName(...). BookKeeper metric names such as add-entry, read-cache-hits, or ledgers-count may keep the hyphen when they are written by the custom text formatter.

Since this provider still writes those BookKeeper metrics manually in the legacy Prometheus text format, preserving the previous underscore escaping behavior would help avoid compatibility surprises. A small regression test with a hyphenated metric name would also make the intended behavior explicit.

Second, I noticed deployment/terraform-ansible/templates/bookkeeper.conf still references org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider. Since this PR removes org.apache.bookkeeper.stats:prometheus-metrics-provider from the server distribution, this template likely needs to be updated to org.apache.pulsar.metrics.prometheus.bookkeeper.PrometheusMetricsProvider as well.