HDDS-15138. Add EC DN safemode rule and control RATIS/THREE background pipelines for EC-default clusters.

[aryangupta1998]

What changes were proposed in this pull request?

This patch switches SCM safemode handling to the new EC approach.

When default replication is EC, SCM now uses a dedicated ECMinDataNodeSafeModeRule to check if enough healthy DataNodes are available (data + parity) before safemode exit.

A new config flag controls whether BackgroundPipelineCreator should create RATIS/THREE pipelines in EC-default clusters:

ozone.scm.pipeline.creation.ec.ratis.three.enabled=true -> create RATIS/THREE pipelines in background
false -> do not create background pipelines for EC-default
For RATIS-default clusters, behavior stays the same as before.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15138

How was this patch tested?

Tests were updated to match this behavior:

added tests for the new EC DN safemode rule
added tests for EC-default + flag true/false behavior

Tested on a dockerized Ozone cluster with:

• Default replication: EC
• EC config: rs-3-2-1024k
• Datanodes: 5

Scenario A: ozone.scm.pipeline.creation.ratis.three=true
Observed:

• SCM exited safemode successfully.
• OPEN RATIS/THREE pipeline(s) were created in background.

Command:
docker compose exec scm ozone admin pipeline list --state OPEN --filter-by-factor THREE --json

Result: non-empty JSON list (OPEN RATIS/THREE pipeline present).

Scenario B: ozone.scm.pipeline.creation.ratis.three=false
Observed:

• SCM exited safemode successfully.
• RATIS/THREE pipeline safemode rules were disabled as expected.
• No OPEN background RATIS/THREE pipelines were created.

Commands:

docker compose logs scm | rg "RATIS/THREE pipeline safemode rules are disabled"
docker compose exec scm ozone admin pipeline list --state OPEN --filter-by-factor THREE --json

Results:

• Log contains:

RATIS/THREE pipeline safemode rules are disabled because ozone.scm.pipeline.creation.ratis.three is false for an EC-default cluster or the default replication config is invalid.

• Pipeline list output:

[]
Conclusion
Behavior matches the intended design:

• EC-default + flag true => background RATIS/THREE pipeline creation enabled.
• EC-default + flag false => RATIS/THREE safemode rules disabled and no background RATIS/THREE pipelines created.
• Safemode exits correctly in both scenarios.

[sodonnel]

Its never really been clear to me what the purpose of the pipeline safemode rule is and how it differs from EC to Ratis. Could you explain why its important?

Also, while a cluster can be configured to have EC as the default, that does not stop Ratis pipelines getting created and some data on the cluster being written in the Ratis format. The same goes the opposite way around too - a ratis default cluster can have EC data on it.

One difference between EC and Ratis pipelines is that EC pipelines are short lived and Ratis pipelines are long lived. Ratis pipelines also survive cluster restarts, EC pipelines do not.

Depending on the EC scheme, there will be a set of pipelines for each EC scheme 3-2, 6-3, 10-4 etc.

I guess my main question is - if the cluster is default EC and only has EC data, what problem could occur if there is no safemode pipeline check?

[aryangupta1998]

@sodonnel, thanks, this is a good point.

The pipeline safemode rules are there to ensure SCM has basic pipeline readiness before leaving safemode:

HealthyPipelineSafeModeRule checks that enough selected open pipelines are healthy.
OneReplicaPipelineSafeModeRule checks that selected open pipelines are reported by at least one DN.
The issue was that selection was effectively hardcoded to RATIS/THREE. In an EC-default (or EC-only) setup, that can be wrong.

A concrete case: with old behavior, the default threshold floor is 1 pipeline (min.datanode=3 / 3). If the cluster is EC-only and has no RATIS/THREE pipelines, this rule may never satisfy, and safemode can stay blocked unless config/workaround is used.

This patch fixes that by making pipeline safemode checks follow ReplicationConfig.getDefault(conf):

If the default is EC, rules check EC pipelines;
If the default is Ratis, behavior stays the same.
I also updated BackgroundPipelineCreator so EC pipeline creation is included when the default is EC, so creation and safemode checks stay aligned.

[rakeshadr]

Can you verify tests:-

1. Create RATIS file creation with replication factor 3. Both EC & RATIS should co-exists.
2. Restart SCM with EC files and make sure safemode check is out.

[Copilot]

Pull request overview

This PR updates SCM safemode behavior for EC-default clusters by introducing an EC-specific DataNode safemode rule and gating RATIS/THREE background pipeline creation (and corresponding safemode pipeline rules) behind a new configuration flag.

Changes:

• Add ECMinDataNodeSafeModeRule and load it via SafeModeRuleFactory for EC-default clusters.
• Add ozone.scm.pipeline.creation.ec.ratis.three.enabled to control RATIS/THREE background pipeline creation in EC-default clusters, and disable RATIS/THREE pipeline safemode rules accordingly.
• Update and add tests for the new EC safemode rule and the EC-default flag behavior.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeWithPipelineRules.java	Adds an integration test covering safemode exit after SCM restart with mixed EC and RATIS/THREE keys.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeManager.java	Updates assertions / adds tests to verify rule enablement with EC-default + flag true/false.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestSafeModeRuleFactory.java	Updates expected safemode rule count due to the new EC rule.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestOneReplicaPipelineSafeModeRule.java	Adjusts mocking for pipeline manager matcher usage.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestECMinDataNodeSafeModeRule.java	Adds unit tests for the new EC min DN safemode rule.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestBackgroundPipelineCreator.java	Adds tests validating replication-config selection for background pipeline creation in EC-default and RATIS-default clusters.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/SafeModeRuleFactory.java	Loads the new EC DN safemode rule and conditionally enables RATIS/THREE pipeline safemode rules based on EC-default + flag.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/OneReplicaPipelineSafeModeRule.java	Refactors to target a specific replication config (RATIS/THREE) more explicitly.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/HealthyPipelineSafeModeRule.java	Refactors to target a specific replication config (RATIS/THREE) and improves status/log text.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/ECMinDataNodeSafeModeRule.java	Introduces a new safemode rule to enforce a minimum DN count for EC-default clusters.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/BackgroundPipelineCreator.java	Changes background pipeline creation to derive configs from the default replication config, with EC-default gating of RATIS/THREE creation.
hadoop-hdds/common/src/main/resources/ozone-default.xml	Documents the new EC-default RATIS/THREE background pipeline creation flag.
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java	Adds the new config key + default value for EC-default RATIS/THREE pipeline creation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ashishkumar50]

@aryangupta1998 Thanks for the patch, please find the comments inline.

[rakeshadr]

Thanks @aryangupta1998!

+1 LGTM

[ashishkumar50]

LGTM

[ashishkumar50]

@aryangupta1998 Thanks for the patch, @rakeshadr for the review.