[FLINK-39546][s3] Improve observability in flink-s3-fs-native by exposing operation-level S3 metrics by Samrat002 · Pull Request #28427 · apache/flink

Samrat002 · 2026-06-13T12:12:04Z

What is the purpose of the change

flink-s3-fs-native currently emits no metrics. When a job's checkpoints, savepoints, or sinks go through it, operators have no visibility into how Flink is actually talking to S3: request volume, latency, throttling, or retries, which makes diagnosing slow or failing checkpoints largely guesswork.

This change makes the native S3 filesystem report operation-level S3 metrics into Flink's metric system. It does so by bridging the AWS SDK's built-in metrics SPI into Flink Counter/Histogram instruments, so every completed S3 API call is counted, timed, and classified.

More details on : FLIP-576

Brief change log

flink-core

Add MetricsAware (@PublicEvolving, org.apache.flink.core.plugin): a FileSystemFactory implements it to be handed a MetricGroup.
Add FileSystem.attachMetrics(MetricGroup) (@Internal): creates a filesystem child group and forwards it to every registered MetricsAware factory; resilient to a misbehaving factory and idempotent.
PluginFileSystemFactory now implements MetricsAware and forwards setMetricGroup to the wrapped inner factory under the plugin classloader. Without this, plugin-loaded filesystems (the normal deployment mode) would silently never receive the group.

flink-runtime

ClusterEntrypoint (JobManager) and TaskManagerRunner (TaskManager) call FileSystem.attachMetrics(processMetricGroup) during startup. The ClusterEntrypoint service-init order was adjusted so this runs before HA/blob services cache filesystem clients, otherwise those early clients would be created without a metric group.

flink-s3-fs-native

NativeS3FileSystemFactory / NativeS3AFileSystemFactory implement MetricsAware and tag metrics with a filesystem_type label set to the scheme (s3 vs s3a), so the two stay distinguishable.
AwsSdkMetricBridge implements software.amazon.awssdk.metrics.MetricPublisher and translates each MetricCollection into Flink metrics: api_call_count (labels: op, status_class), api_call_duration_ms (histogram, label op), throttle_count (label op), retry_count (labels: op, reason).
S3MetricHistogram: a bounded sliding-window histogram backing the duration metric.
S3ClientProvider registers the publisher on the sync/async clients.
New config options: s3.metrics.enabled (off by default), s3.metrics.allowlist, s3.metrics.histogram.window-size.

Verifying this change

This change added tests and can be verified as follows.

Automated tests

AwsSdkMetricBridgeTest — translation of SDK records to Flink metrics; status_class classification (2xx/4xx/5xx/throttled); retry attribution; allowlist behavior (explicit list, * wildcard, empty → defaults).
S3MetricHistogramTest — sliding-window statistics.
NativeS3FileSystemFactoryMetricsTest — the filesystem_type label resolves to s3 / s3a per factory.
FileSystemAttachMetricsTest (flink-core) — attachMetrics unwraps PluginFileSystemFactory to reach the real factory, skips non-MetricsAware factories, survives a throwing factory, and is idempotent.
NativeS3MetricsEmissionITCase — MinIO via Testcontainers; real GET/HEAD/LIST round trips, asserting the counters/histograms are readable back through a real MetricRegistry (MetricListener). Auto-skips without Docker.

Manual end-to-end against real AWS S3

I ran a standalone cluster built from this branch with s3.metrics.enabled: true and the SLF4J reporter, and submitted a large-state streaming job checkpointing to s3://<bucket>/checkpoints (HashMap backend, filesystem checkpoint storage, 10 s interval). The native plugin loaded (Plugin loader ... s3-fs-native), built its client via the SDK default credential chain, and wrote real checkpoint objects to S3. The reporter then showed the metrics on both the TaskManager (data-plane writes) and the JobManager (checkpoint coordination / multipart):

# TaskManager
...filesystem.filesystem_type.s3.op.PutObject.status_class.2xx.api_call_count: 31
...filesystem.filesystem_type.s3.op.PutObject.api_call_duration_ms: count=31, min=343, max=8329, mean=3383.2, p99=8329.0
...filesystem.filesystem_type.s3.op.ListObjectsV2.status_class.2xx.api_call_count: 31
...filesystem.filesystem_type.s3.op.ListObjectsV2.reason.other.retry_count: 1
...filesystem.filesystem_type.s3.op.HeadObject.status_class.4xx.api_call_count: 25

# JobManager
...filesystem.filesystem_type.s3.op.CreateMultipartUpload.status_class.2xx.api_call_count: 25
...filesystem.filesystem_type.s3.op.UploadPart.status_class.2xx.api_call_count: 25
...filesystem.filesystem_type.s3.op.CompleteMultipartUpload.status_class.2xx.api_call_count: 25
...filesystem.filesystem_type.s3.op.DeleteObject.status_class.2xx.api_call_count: 48
...filesystem.filesystem_type.s3.op.HeadObject.status_class.2xx.api_call_count: 48

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: yes, native-s3-fs

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented?
this change introduces the feature, followup documentation is up next.

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

flinkbot · 2026-06-13T12:18:21Z

CI report:

e5c8bce Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

gaborgsomogyi

Thanks for the efforts! Left some high level questions

gaborgsomogyi · 2026-06-22T09:50:51Z

+                    .defaultValues(
+                            "api_call_count",
+                            "api_call_duration_ms",
+                            "throttle_count",
+                            "retry_count",
+                            "iops")


Just for my own understanding. I want to use a new metric called foo. Do I need to go to s3.metrics.allowlist definition, check the default value and then I need to set: s3.metrics.allowlist=ORIGINAL_DEFAULT,foo?

Yes. The allowlist replaces the default list, so retaining the defaults while adding foo requires ORIGINAL_DEFAULT,foo; * enables every metric emitted by the plugin. The shared option description now states the replacement behavior explicitly.

gaborgsomogyi · 2026-06-22T09:57:13Z

+        AwsSdkMetricBridge bridge = this.metricBridge;
+        if (bridge == null) {
+            synchronized (this) {
+                bridge = this.metricBridge;
+                if (bridge == null) {
+                    bridge =
+                            new AwsSdkMetricBridge(
+                                    metrics,
+                                    config.get(METRICS_ALLOWLIST),
+                                    config.get(METRICS_HISTOGRAM_WINDOW_SIZE));
+                    this.metricBridge = bridge;
+                }
+            }
+        }
+        return bridge;


metricBridge write is inside the the sync block but read is outside + not transient. I've a feeling that we have random read here without super well founded consideration. Maybe some guardedby would be good as always. Not if that fits for synchronized but end-to-end what protects what and when would be good.

Good catch. I removed the lock-free double-checked path in 47babe4. setMetricGroup(...) and bridge resolution now synchronize on the same monitor, and pluginMetrics, attachedMetricGroup, and metricBridge are all @GuardedBy("this"). transient is unnecessary because the factory is not serializable.

gaborgsomogyi · 2026-06-22T09:59:39Z

+            metricRegistry = createMetricRegistry(configuration, pluginManager, rpcSystem);
+
+            final RpcService metricQueryServiceRpcService =
+                    MetricUtils.startRemoteMetricsRpcService(
+                            configuration,
+                            commonRpcService.getAddress(),
+                            configuration.get(JobManagerOptions.BIND_HOST),
+                            rpcSystem);
+            metricRegistry.startQueryService(metricQueryServiceRpcService, null);
+
+            final String hostname = RpcUtils.getHostname(commonRpcService);
+
+            processMetricGroup =
+                    MetricUtils.instantiateProcessMetricGroup(
+                            metricRegistry,
+                            hostname,
+                            ConfigurationUtils.getSystemResourceMetricsProbingInterval(
+                                    configuration));
+
+            // Second-phase init for file system plugins that opt into metrics (e.g.
+            // flink-s3-fs-native): hand them the process-level metric group before any file system
+            // is used. This must run ahead of the HA services and BlobServer below, because those
+            // may open external file systems (e.g. S3 HA/blob storage), creating them first would
+            // cache metric-less file system clients for the rest of the process lifetime. See
+            // FileSystem#attachMetrics and MetricsAware.


Not yet, checked in-depth. Is this just a move as-is or changed too?

The metric-registry, query-service, and process-group block is moved as-is. The functional addition is FileSystem.attachMetrics(processMetricGroup) immediately after the process group exists, so filesystem clients created by later startup services receive the metric scope.

gaborgsomogyi · 2026-06-22T10:15:38Z

+        // Second-phase init for file system plugins that opt into metrics (e.g.
+        // flink-s3-fs-native): hand them the process-level metric group now that the
+        // MetricRegistry exists. See FileSystem#attachMetrics and MetricsAware.
+        FileSystem.attachMetrics(taskManagerMetricGroup.f0);


Now we're solving that for filesystems which is good. How do we think that a non-FS plugin can be used in metrics system. We don't need a full featuring plan but knowing this is a good measure how this solution fits here.

MetricsAware is intentionally only the opt-in contract; it does not add global plugin discovery. A non-filesystem plugin family would use the same pattern by forwarding setMetricGroup(...) through its wrappers and invoking it at the runtime point where that family already owns an appropriate metric group. The filesystem attachment is the first concrete integration of that pattern.

Izeren

Hi @Samrat002, thank you for the change. I have left clarification questions.

Overall, it looks good, but I would like you to extract more cloud agnostic code from native FS. This way, the shims we will need to do to repeat this in Azure, GCS will be really thin. Another concern I have is regrading duplicating non-trivial code to avoid dependency footprint. It seems to me that would be worth extracting lightweight dependency rather than duplicating the code.

Izeren · 2026-06-23T13:00:35Z

+        // Second-phase init for file system plugins that opt into metrics (e.g.
+        // flink-s3-fs-native): hand them the process-level metric group now that the
+        // MetricRegistry exists. See FileSystem#attachMetrics and MetricsAware.
+        FileSystem.attachMetrics(taskManagerMetricGroup.f0);


Izeren

Hi @Samrat002, I went through some of it today, and I can see that there are many comments that I have left before that were not replied. Could you please provide replies in threads before I take a second pass.

Izeren

Thank you @Samrat002, high level LGTM. But I would like some brief explanation regarding backwards compatibility of the change:

-import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
+import org.apache.flink.metrics.SlidingWindowHistogram;

My original suggestion was rather "move" DescriptiveStatistics to metrics-core and make both runtime-core and native-fs import it from metrics-core (which is supposedly lighter dependency to take).

In current implementation, there is a behavioural change of DescriptiveStatisticsHistogram + we still bring in custom implementation which is otherwise unjustified.

Samrat002 · 2026-07-28T03:54:53Z

Thanks @Izeren . I've decoupled this. DescriptiveStatisticsHistogram is now reverted to its original commons-math3 implementation, so there's no behavioural or backwards-compat change to the runtime histogram.

On reusing it from metrics-core, the blocker is that DescriptiveStatisticsHistogramStatistics pulls in commons-math3 (Percentile, StandardDeviation, …). Moving it to flink-metrics-core would force commons-math3 onto the leanest, most widely-depended module (every reporter/connector), which felt like a worse trade than a small dependency-free histogram. And since the S3 plugin only depends on flink-runtime in test scope, its main code can't reference DescriptiveStatisticsHistogram anyway. org.apache.flink.metrics is parent-first for plugins and already on the classpath, so SlidingWindowHistogram lives there for the plugin to use. If we later want to unify the runtime histogram onto a shared sliding-window base,

I'd propose doing that as its own JIRA with proper percentile-compat testing rather than as a side effect here. WDUT?

Izeren · 2026-07-30T16:53:54Z

Thanks @Izeren . I've decoupled this

Thank you @Samrat002, I agree, lets do improvement of histogram implementation separately. This PR is big enough

…sing operation-level S3 metrics

Samrat002 · 2026-08-03T14:33:49Z

@gaborgsomogyi PTAL whenever time

gaborgsomogyi reviewed Jun 22, 2026

View reviewed changes

Izeren reviewed Jun 23, 2026

View reviewed changes

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Jun 23, 2026

Samrat002 force-pushed the metrics-system branch 3 times, most recently from ff9f59f to f027133 Compare July 10, 2026 04:57

Izeren reviewed Jul 15, 2026

View reviewed changes

Samrat002 force-pushed the metrics-system branch from f027133 to 7bc4048 Compare July 19, 2026 10:21

Samrat002 requested review from Izeren and gaborgsomogyi July 19, 2026 16:39

Samrat002 force-pushed the metrics-system branch 2 times, most recently from 288c21d to 540295f Compare July 21, 2026 06:26

Izeren reviewed Jul 22, 2026

View reviewed changes

Samrat002 requested a review from Izeren July 28, 2026 03:55

Samrat002 force-pushed the metrics-system branch from 540295f to 998b32b Compare July 28, 2026 04:30

Izeren approved these changes Jul 30, 2026

View reviewed changes

Samrat002 added 2 commits August 2, 2026 11:40

[FLINK-39546][s3] Improve observability in flink-s3-fs-native by expo…

f62a9bd

…sing operation-level S3 metrics

[FLINK-39546][s3] Extract cloud-neutral filesystem metrics

e5c8bce

Samrat002 force-pushed the metrics-system branch from 998b32b to e5c8bce Compare August 2, 2026 07:48

Uh oh!

Conversation

Samrat002 commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

gaborgsomogyi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Izeren left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Izeren left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Izeren left a comment

Choose a reason for hiding this comment

Uh oh!

Samrat002 commented Jul 28, 2026

Uh oh!

Izeren commented Jul 30, 2026

Uh oh!

Samrat002 commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Samrat002 commented Jun 13, 2026 •

edited

Loading

flinkbot commented Jun 13, 2026 •

edited

Loading