HDDS-14990. Show failed volumes in `ozone admin datanode list` output and SCM metrics by xichen01 · Pull Request #10058 · apache/ozone

xichen01 · 2026-04-09T03:30:43Z

What changes were proposed in this pull request?

ozone admin datanode list can dispaly "Total volume count" and "Healthy volume count", Datanode JMX can display the NumFailedVolumes

Add:

The command display which disk was failure.
SCM's Prometheus metrics for failed disk.
Add ozone admin datanode list --failed-volumes to disply failed disk Datanode only

Example

$ ozone admin datanode list
Datanode: d81a7da8-86d4-42ca-83e9-7015401ed11c (/default/1.2.3.01/host0/0 pipelines)
Operational State: IN_SERVICE
Health State: STALE
Total volume count: 4
Healthy volume count: 3
Failed volumes:    <--- Newly added
  /data/disk2

//...

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14990

How was this patch tested?

new test

…trics

adoroszlai

Thanks @xichen01 for the patch.

adoroszlai · 2026-04-09T06:00:58Z

      Integer healthyVolumeCount = node.hasHealthyVolumeCount() ? node.getHealthyVolumeCount() : null;
      BasicDatanodeInfo singleNodeInfo = new BasicDatanodeInfo.Builder(
          DatanodeDetails.getFromProtoBuf(node.getNodeID()), node.getNodeOperationalStates(0),
-          node.getNodeStates(0)).withVolumeCounts(totalVolumeCount, healthyVolumeCount).build();
+          node.getNodeStates(0)).withVolumeCounts(totalVolumeCount, healthyVolumeCount)
+          .withFailedVolumes(getFailedVolumes(node)).build();


I'd like to suggest a small refactoring to reduce duplication of code related to BasicDatanodeInfo creation.

HDDS-14990.patch

adoroszlai · 2026-04-09T06:02:15Z

      }
    } else {
      pipelineListInfo.append("No pipelines in cluster.");
+      pipelineListInfo.append(System.getProperty("line.separator"));


Why is this needed? pipelineListInfo is output with println below.

This is to maintain consistency with the pipelineListInfo in the previous branches, where pipelineListInfos always have line breaks.

Thanks for the info. Then please change to:

pipelineListInfo.append("No pipelines in cluster.") .append(System.getProperty("line.separator"));

due to recently added PMD rule (HDDS-14934).

(BTW, I think using \n would be fine for now, it is already used in few places instead of System.getProperty("line.separator"), both in this file and in 122 others.)

adoroszlai · 2026-04-09T06:03:12Z

+    assertTrue(output.contains("Failed volume"));
+    assertTrue(output.contains("/data/disk2"));
+    assertTrue(output.contains("/data/disk1"));
+    assertTrue(output.contains("/data/disk5"));


nit: please use assertThat(output).contains(...) for better failure message (see HDDS-9951)

adoroszlai · 2026-04-09T06:05:37Z

+  @CommandLine.Option(names = {"--failed-volumes"},
+      description = "Only show datanodes that have at least one failed volume.",


Would --with-failed-volume be better?

how about --show-failed-volumes ? --with-failed-volume seems mean that "display along with bad disk information".

--show-failed-volumes implies failed volumes are normally hidden, and included in the output only if this flag is used.

I think the command

ozone admin datanode list --with-failed-volume

reads better.

If we want to be extra clear, it can be --nodes-with-failed-volume.

Update to --nodes-with-failed-volume

adoroszlai

Thanks @xichen01 for updating the patch. Let's wait to see if others have comments.

adoroszlai · 2026-04-09T10:28:31Z

TestReconAndAdminContainerCLI is flaky (HDDS-11128), no need to re-run.

sreejasahithi

Thanks @xichen01 for working on this.

sreejasahithi · 2026-04-10T04:09:16Z

+  @CommandLine.Option(names = {"--nodes-with-failed-volumes"},
+      description = "Only show datanodes that have at least one failed volume.",
+      defaultValue = "false")
+  private boolean nodeWithFailedVolumes;
+


The --nodes-with-failed-volumes filter is silently ignored when --node-id is used.
We should make these options mutually exclusive to avoid confusion when a user provides both in the command but receives the result for the node whose ID was specified, regardless of whether it has failed volumes.

sreejasahithi

Thanks for updating the patch.

sreejasahithi · 2026-04-14T05:41:39Z

+    if (nodeWithFailedVolumes && exclusiveNodeOptions != null
+        && !Strings.isNullOrEmpty(exclusiveNodeOptions.getNodeId())) {
+      throw new IOException(
+          "--nodes-with-failed-volumes cannot be used with --id/--node-id. "
+          + "Use them separately.");
+    }


Picocli reports argument conflicts as CommandLine.MutuallyExclusiveArgsException (a subclass of CommandLine.ParameterException). Throwing IOException from execute() bypasses picocli's error handling entirely, producing a different error format than what the user would see from other mutual-exclusion violations. so change this to throw new CommandLine.ParameterException.

(you can refer to DecommissionStatusSubCommand to see how it can be used.)

Update the test as well with the correct exception.

sreejasahithi

Thanks @xichen01 for updating.
LGTM

adoroszlai · 2026-04-17T15:48:46Z

Thanks @xichen01 for the patch, @sarvekshayr, @sreejasahithi for the review.

HDDS-14990. Dispaly the failed Disk detail info in the command and Me…

f6c1b57

…trics

adoroszlai reviewed Apr 9, 2026

View reviewed changes

adoroszlai changed the title ~~HDDS-14990. Dispaly the failed Disk detail info in the command and Metrics~~ HDDS-14990. Let ozone admin datanode list show failed volumes and add metrics Apr 9, 2026

adoroszlai changed the title ~~HDDS-14990. Let ozone admin datanode list show failed volumes and add metrics~~ HDDS-14990. Show failed volumes in ozone admin datanode list output and SCM metrics Apr 9, 2026

xichen01 added 3 commits April 9, 2026 15:10

Fxi PMD Failure

4ab01e6

Address some comments

c4d22d3

fix PMD warn

b48fb63

adoroszlai approved these changes Apr 9, 2026

View reviewed changes

adoroszlai requested a review from sarvekshayr April 9, 2026 18:29

sreejasahithi reviewed Apr 10, 2026

View reviewed changes

make --nodes-with-failed-volumes and --node-id mutually exclusive.

008d8ee

sreejasahithi reviewed Apr 14, 2026

View reviewed changes

Use CommandLine.ParameterException; Add test

782048a

adoroszlai requested a review from sreejasahithi April 17, 2026 13:41

sreejasahithi approved these changes Apr 17, 2026

View reviewed changes

adoroszlai merged commit e744b81 into apache:master Apr 17, 2026
44 of 45 checks passed

		@CommandLine.Option(names = {"--failed-volumes"},
		description = "Only show datanodes that have at least one failed volume.",

Uh oh!

Conversation

xichen01 commented Apr 9, 2026

What changes were proposed in this pull request?

Add:

Example

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Apr 9, 2026

Uh oh!

sreejasahithi left a comment

Choose a reason for hiding this comment

Uh oh!

sreejasahithi Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sreejasahithi left a comment

Choose a reason for hiding this comment

Uh oh!

sreejasahithi Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sreejasahithi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adoroszlai commented Apr 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sreejasahithi Apr 10, 2026 •

edited

Loading

sreejasahithi Apr 14, 2026 •

edited

Loading