HDDS-15552. Ratis events should not be published as metrics by jojochuang · Pull Request #10523 · apache/ozone

jojochuang · 2026-06-15T20:36:27Z

What changes were proposed in this pull request?

HDDS-15552. Ratis events should not be published as metrics

Please describe your PR in detail:

Treat getRatisEvents() as a normal method in the Metrics classes. Drop the Metrics annotation.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15552

How was this patch tested?

Copilot

Pull request overview

This PR addresses HDDS-15552 by stopping Ratis state machine events from being exported via the Hadoop Metrics2 system and instead exposing them via the existing *Info,component=ServerRuntime JMX (MXBean) endpoints used for service runtime information.

Changes:

Removed the @Metric annotation from getRatisEvents() in OM/SCM metrics classes so events are no longer published as metrics.
Added getRatisEvents() to OMMXBean/SCMMXBean and implemented it in OzoneManager/StorageContainerManager by delegating to their metrics instances.
Updated OM/SCM web UIs (and an SCM integration test) to read RatisEvents from the *Info,component=ServerRuntime MXBeans instead of tag.RatisEvents from the metrics beans.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/ozoneManager.js	Switch OM UI Ratis events query from `OMMetrics` to `OzoneManagerInfo` MXBean.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java	Expose `RatisEvents` via OM MXBean by delegating to `OMMetrics`.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMMXBean.java	Add `getRatisEvents()` attribute to OM MXBean contract.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMMetrics.java	Stop exporting Ratis events as a Metrics2 metric by removing `@Metric`.
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMMXBean.java	Assert SCM MXBean exposes `RatisEvents`.
hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm.js	Switch SCM UI Ratis events query from `SCMMetrics` to `StorageContainerManagerInfo` MXBean.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java	Expose `RatisEvents` via SCM MXBean by delegating to `SCMMetrics`.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMMXBean.java	Add `getRatisEvents()` attribute to SCM MXBean contract.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/metrics/SCMMetrics.java	Stop exporting Ratis events as a Metrics2 metric by removing `@Metric`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

                    var metrics = result.data.beans[0];
-                    var rawEvents = metrics['tag.RatisEvents'] ? metrics['tag.RatisEvents'].split('\n') : [];
+                    var rawEvents = (metrics && metrics['RatisEvents']) ? metrics['RatisEvents'].split('\n') : [];


                    var metrics = result.data.beans[0];
-                  var rawEvents = metrics['tag.RatisEvents'] ? metrics['tag.RatisEvents'].split('\n') : [];
+                    var rawEvents = (metrics && metrics['RatisEvents']) ? metrics['RatisEvents'].split('\n') : [];


   */
  String getHostname();
+
+  String getRatisEvents();


   */
  String getHostname();
+
+  String getRatisEvents();


amaliujia · 2026-06-19T03:18:08Z

+  @Override
+  public String getRatisEvents() {
+    return metrics != null ? metrics.getRatisEvents() : "";
+  }


This looks like a legit suggestion

For SCM it is simple, because TestSCMMXBean already exists, but there is no test for OMMXBean currently. So this could be a follow-up.

Created HDDS-15761.

+    String ratisEvents = (String) mbs.getAttribute(bean, "RatisEvents");
+    assertEquals(scm.getMetrics().getRatisEvents(), ratisEvents);


errose28

Thanks for the quick fix @jojochuang. To ensure this is no longer being published to Prometheus we should implement this from the Jira as well:

Additionally to verify this change, we should add an acceptance test call to GET http://:9090/api/v1/targets and ensure that health=up for each component to prevent future regressions like this.

This should be a small addition to any existing acceptance test that uses Prometheus.

Change-Id: I06634562bad09e1cf308b3d2f9ec93d4b6c078fe

adoroszlai · 2026-06-24T13:46:47Z

Key delete passthrough                                                | FAIL |
4 != 0

failed here and in first attempt in fork. Unfortunately, check artifact is not available due to:

The path for one of the files in artifact is not valid: /ozone/ozone-prometheus-1_runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH
.stack. Contains the following character:  Double quote "

Originally reported it as follow-up (HDDS-15660), but looks like it would be better to fix it right in this PR. Please filter prometheus here:

ozone/hadoop-ozone/dist/src/main/compose/testlib.sh

Lines 290 to 299 in d1905e1

    
           ## @description Create stack dump of each java process in each container 
        
           create_stack_dumps() { 
        
             local c pid procname 
        
             for c in $(docker-compose ps | cut -f1 -d' ' | grep -e datanode -e om -e recon -e s3g -e scm); do 
        
               while read -r pid procname; do 
        
                 echo "jstack $pid > ${RESULT_DIR}/${c}_${procname}.stack" 
        
                 docker exec "${c}" bash -c "jstack $pid" > "${RESULT_DIR}/${c}_${procname}.stack" 
        
               done < <(docker exec "${c}" bash -c "jps | grep -v Jps" || true) 
        
             done 
        
           }

like:

diff --git hadoop-ozone/dist/src/main/compose/testlib.sh hadoop-ozone/dist/src/main/compose/testlib.sh
index 903fdad7de..3c31de9b48 100755
--- hadoop-ozone/dist/src/main/compose/testlib.sh
+++ hadoop-ozone/dist/src/main/compose/testlib.sh
@@ -290,7 +290,7 @@ reorder_om_nodes() {
 ## @description Create stack dump of each java process in each container
 create_stack_dumps() {
   local c pid procname
-  for c in $(docker-compose ps | cut -f1 -d' ' | grep -e datanode -e om -e recon -e s3g -e scm); do
+  for c in $(docker-compose ps | cut -f1 -d' ' | grep -e datanode -e om -e recon -e s3g -e scm | grep -v -e prometheus); do
     while read -r pid procname; do
       echo "jstack $pid > ${RESULT_DIR}/${c}_${procname}.stack"
       docker exec "${c}" bash -c "jstack $pid" > "${RESULT_DIR}/${c}_${procname}.stack"

With that fix, we can get an artifact for the test run if it fails again, which may help determine if the failure is related to this change or not.

Change-Id: I3845a01fbd97f6e51106d64037d00cc2d4fc9073

adoroszlai · 2026-06-24T19:35:50Z

Thanks @jojochuang for fixing the stack dump problem, now we get acceptance-unsecure artifact. log.html shows that test failure is caused by unexpected log in ozone sh key list output that jq cannot parse as JSON:

$ ozone sh key list o3://om:9862/21377-with-del/bfso
[ {
  "volumeName" : "21377-with-del",
  "bucketName" : "bfso",
  "name" : ".Trash/",
   ...
} ]
Jun 24, 2026 6:33:48 PM io.opentelemetry.sdk.common.internal.ThrottlingLogger doLog
SEVERE: Failed to export spans. The request could not be executed.
java.io.InterruptedIOException: executor rejected
	at okhttp3.internal.connection.RealCall$AsyncCall.failRejected$okhttp(RealCall.kt:563)
	at okhttp3.internal.connection.RealCall$AsyncCall.executeOn(RealCall.kt:554)
	at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.kt:252)
	at okhttp3.Dispatcher.promoteAndExecute$default(Dispatcher.kt:163)
	at okhttp3.Dispatcher.finished$okhttp(Dispatcher.kt:269)
	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:597)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.internal.connection.RealCall$AsyncCall@611166bd rejected from java.util.concurrent.ThreadPoolExecutor@73163d48[Running, pool size = 5, active threads = 5, queued tasks = 0, completed tasks = 5]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2081)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1376)
	at okhttp3.internal.connection.RealCall$AsyncCall.executeOn(RealCall.kt:551)
	... 7 more

parse error: Invalid numeric literal at line 70, column 4

The problems is intermittent, passed in PR run, failed in fork run.

Change-Id: Ie72dd58955082f77262aa380c6e4e8e35c87c30d

Change-Id: Ib3a9a17b0459ea1bea679ebde8543d304342e485

jojochuang · 2026-06-24T23:23:06Z

Ignoring the opentelemetry error by removing stderr output.

adoroszlai · 2026-06-25T08:07:59Z

Ignoring the opentelemetry error by removing stderr output.

Users running these commands would also see the errors, I don't think the test should just suppress them.

adoroszlai · 2026-06-25T08:29:00Z

@sravani-revuri could you please help check the problem described in #10523 (comment)?

sravani-revuri · 2026-06-26T06:15:06Z

From seeing the logs the this seems to be the main cause of the error:

Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.internal.connection.RealCall$AsyncCall@611166bd rejected from java.util.concurrent.ThreadPoolExecutor@73163d48[Running, pool size = 5, active threads = 5, queued tasks = 0, completed tasks = 5]

Issue:

the exporter isn't able to send data to the collector (jaeger) and drops spans because of all the threads currently being occupied.
there could be 2 causes for this.

jaeger container is down . hence the exporter isn't able to export the spans to the collector and therefore they're being dropped. (ruling out because logs don't show anything that suggest this)
exporter isn't able to keep up with the load of spans being sent to it.

The error is seen across om and scm logs too.

multiple commands are being run and the sampling level has been set to 100% causing everything to be recorded.
This might cause intermittent failures between runs which depends on the burst of spans for that specific run.
Also this code currently uses the simplespanprocessor which sends data to the exporter as and when span.end() is called. changing to batchspanprocessing to batch spans into fewer exports reduces this .

@adoroszlai.

adoroszlai · 2026-06-26T07:10:36Z

Thanks @sravani-revuri for checking. You are right, the change to BatchSpanProcessor fixes the problem:

2/10 passed before HDDS-15579: https://github.com/adoroszlai/ozone/actions/runs/28154222769
10/10 passed after HDDS-15579: https://github.com/adoroszlai/ozone/actions/runs/28221759430

So I think we have two options, depending on how much work is left in #10569:

getting HDDS-15579. Replace SimpleSpanProcessor with BatchSpanProcessor #10569 merged before this
splitting the compose and robot changes from this PR to a follow-up

errose28 · 2026-06-26T16:55:27Z

I'm fine with splitting the acceptance test changes to a follow up as long as we manually check that Prometheus is able to pull from all nodes before merging this. I suspect the acceptance criteria can be done with a simpler mechanism that the python script but I don't want to block the Prometheus integration on this.

…-lib.robot" This reverts commit fc5d81b.

…bot" This reverts commit 6ea9137.

jojochuang · 2026-07-06T21:50:57Z

I can't self approve it but LGTM thanks @adoroszlai

adoroszlai · 2026-07-07T06:40:32Z

Thanks @jojochuang for the patch, @amaliujia, @errose28 for the review.

* master: (519 commits) HDDS-14544. OM DB Insights: Duplicate API calls triggered when changing limit selector (apache#10677). HDDS-15587. [Recon] Show 0 for offline DN pending deletion instead of -1 (apache#10585). HDDS-15521. StreamBlockInputStream fails with TimeoutIOException without retry or failover. (apache#10479) HDDS-15170. Add mock-based unit tests for DataStream write path (apache#10230) HDDS-15552. Ratis events should not be published as metrics (apache#10523) HDDS-15746. Bump kerby to 2.1.2 (apache#10666) HDDS-15579. Replace SimpleSpanProcessor with BatchSpanProcessor (apache#10569) HDDS-15747. Address review comments for HDDS-15083 (apache#10669) HDDS-15732. Some ozone commands ignore OZONE_MODULE_ACCESS_ARGS (apache#10655) HDDS-15605. Fix flaky testContainerExclusionWithClosedContainerException (apache#10621) HDDS-11855. Fix flaky TestContainerBalancerDatanodeNodeLimit#checkIterationResultException (apache#10667) HDDS-15741. Bump awssdk to 2.46.17 (apache#10661) HDDS-10307. Speed up TestOzoneManagerHAWithStoppedNodes (apache#10658) HDDS-15651. Test case for DiskBalancer when markContainerForDelete fails (apache#10593) HDDS-15742. Bump nimbus-jose-jwt to 10.9.1 (apache#10662) HDDS-15719. Add check for allowed action usage in workflows (apache#10641) HDDS-15744. Bump javassist to 3.32.0-GA (apache#10665) HDDS-15737. Fix intermittent failure in balancerShouldOnlySelectConfiguredIncludeContainers (apache#10660) HDDS-15743. Bump gson to 2.14.0 (apache#10664) HDDS-11093. Fix intermittent failure in TestContainerBalancerDatanodeNodeLimit#testMetrics (apache#10659) ... Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScanHelper.java

jojochuang requested review from Copilot and errose28 June 15, 2026 20:36

Copilot started reviewing on behalf of jojochuang June 15, 2026 20:37 View session

Copilot AI reviewed Jun 15, 2026

View reviewed changes

errose28 reviewed Jun 16, 2026

View reviewed changes

Comment thread ...ver-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/metrics/SCMMetrics.java

HDDS-15552. Ratis events should not be published as metrics

4b67db7

Change-Id: I06634562bad09e1cf308b3d2f9ec93d4b6c078fe

jojochuang force-pushed the HDDS-15552 branch from 0b9b66e to 4b67db7 Compare June 22, 2026 21:42

jojochuang marked this pull request as ready for review June 22, 2026 21:42

jojochuang requested a review from errose28 June 23, 2026 17:34

Merge remote-tracking branch 'origin/master' into HDDS-15552

8fba2b6

HDDS-15660. Filter prometheus containers when creating stack dumps

d109239

Change-Id: I3845a01fbd97f6e51106d64037d00cc2d4fc9073

errose28 mentioned this pull request Jun 24, 2026

HDDS-14825. Add Grafana Dashboard and Metrics for ZDU #10602

Merged

jojochuang added 2 commits June 24, 2026 16:19

HDDS-15552. Redirect stderr of key list in ozone-shell-lib.robot

6ea9137

Change-Id: Ie72dd58955082f77262aa380c6e4e8e35c87c30d

HDDS-15552. Redirect stderr of getacl commands in ozone-shell-lib.robot

fc5d81b

Change-Id: Ib3a9a17b0459ea1bea679ebde8543d304342e485

adoroszlai marked this pull request as draft June 25, 2026 05:18

adoroszlai added 3 commits July 6, 2026 13:06

Merge remote-tracking branch 'origin/master' into HDDS-15552

3eab08d

Revert "HDDS-15552. Redirect stderr of getacl commands in ozone-shell…

e200d93

…-lib.robot" This reverts commit fc5d81b.

Revert "HDDS-15552. Redirect stderr of key list in ozone-shell-lib.ro…

9fd803d

…bot" This reverts commit 6ea9137.

adoroszlai marked this pull request as ready for review July 6, 2026 12:35

adoroszlai approved these changes Jul 7, 2026

View reviewed changes

adoroszlai merged commit 6ef19ac into apache:master Jul 7, 2026
60 checks passed

		String ratisEvents = (String) mbs.getAttribute(bean, "RatisEvents");
		assertEquals(scm.getMetrics().getRatisEvents(), ratisEvents);

Uh oh!

Conversation

jojochuang commented Jun 15, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

amaliujia Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

adoroszlai Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adoroszlai commented Jun 24, 2026

Uh oh!

adoroszlai commented Jun 24, 2026

Uh oh!

jojochuang commented Jun 24, 2026

Uh oh!

adoroszlai commented Jun 25, 2026

Uh oh!

adoroszlai commented Jun 25, 2026

Uh oh!

sravani-revuri commented Jun 26, 2026

Issue:

Uh oh!

adoroszlai commented Jun 26, 2026

Uh oh!

errose28 commented Jun 26, 2026

Uh oh!

jojochuang commented Jul 6, 2026

Uh oh!

Uh oh!

adoroszlai commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants