HDDS-14580. Remove OM Prepare for Upgrade Code for ZDU by errose28 · Pull Request #9723 · apache/ozone

errose28 · 2026-02-06T22:59:08Z

What changes were proposed in this pull request?

The prepare for upgrade step will not work in the context of ZDU. The design doc outlines a plan to upgrade the OMs without this command, so we can remove it from the ZDU feature branch. Since this is going in a branch, it is ok that we have not yet implemented the OM versioning required for guaranteed consistency without prepare for upgrade.

In this change, prepare for upgrade handling is removed from the server in a backwards compatible way. If the server receives a prepare request, it will return success but indicate that it is not supported in this server version. The CLI has been hidden on the client side, but left intact in case the new client is used to upgrade an old server.

After v100 for ZDU is added, we can add an upgrade action which clears the DB key and prepare marker file that may be left behind in the old version.

In the next major Ozone release following this change (3.0?), we can remove support for the CLI and OM API entirely.

What is the link to the Apache JIRA

HDDS-14580

How was this patch tested?

Passing run on my fork
New tests for the expected backwards compatible prepare functionality added in TestOzoneShellHA and TestOzoneManagerRequestHandler.java‎
- Some hsync/open file tests had existing issues causing them to fail when the new prepare test was added. These have been marked unhealthy and the resolution is tracked in HDDS-14686.
Existing tests should pass indicating no dependency on prepare
There was an existing bug in PrepareSubCommand where the client object used to check each OM for status had no port explicitly configured, and was falling back to the default. This usually worked, but had never been tested with MiniOzoneCluster, where the ports are randomly assigned. The command had to be updated to account for the configured OM port due to the new MiniOzoneCluster test added.
TestOMUpgradeFinalization was removed because it depended on prepare to force a follower to install a snapshot. We can add a new test that covers this functionality in HDDS-14687.

Generated-By: Cursor

errose28 · 2026-02-17T23:31:56Z

    CMD_AUDIT_ACTION_MAP.put(Type.SetVolumeProperty, OMAction.SET_OWNER);
    CMD_AUDIT_ACTION_MAP.put(Type.SetBucketProperty, OMAction.UPDATE_BUCKET);
-    CMD_AUDIT_ACTION_MAP.put(Type.Prepare, OMAction.UPGRADE_PREPARE);
-    CMD_AUDIT_ACTION_MAP.put(Type.CancelPrepare, OMAction.UPGRADE_CANCEL);


@octachoron I did actually remove the audit logging for prepare here since it became a no-op write request. Although as you suggested, it would probably still be good to leave the admin check on the server side for coompleteness.

This reverts commit 02749ef.

sodonnel

LGTM - minimal new code is added and mostly code is removed to remove the prepare feature.

errose28 · 2026-02-27T22:22:22Z

Even though the operations are no-ops, I restored the admin checks for consistency with the older version. I did this in a manner similar to what is used in OMAdminProtocolServerSideImpl, where we do not use acl.enabled as a pre-requisite for the admin checks since ACLs are for the namespace and independent of admin checks.

@sodonnel can you check the latest commit?

sodonnel

Latest commit looks good. If we get green CI we can commit it.

adoroszlai · 2026-03-09T05:46:24Z

Failure in TestOzoneManagerRequestHandler looks related, happened on both runs, as well as in fork.

Error:  Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.581 s <<< FAILURE! -- in org.apache.hadoop.ozone.protocolPB.TestOzoneManagerRequestHandler
Error:  org.apache.hadoop.ozone.protocolPB.TestOzoneManagerRequestHandler.testPrepareRequestsAreNoOps -- Time elapsed: 0.169 s <<< FAILURE!
org.opentest4j.AssertionFailedError: Prepare should return success ==> expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
	at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:214)
	at org.apache.hadoop.ozone.protocolPB.TestOzoneManagerRequestHandler.testPrepareRequestsAreNoOps(TestOzoneManagerRequestHandler.java:267)

Acceptance test failure is unrelated. However, "Re-run failed jobs" of acceptance possible only until ozone binaries expire, in 24 hours. Not a problem here, since the integration test needs a new commit anyway.

Testing OM DB Size Reduction After Compaction                         | FAIL |
OM DB size should be reduced after compaction. Before: 4623010, After: 5135582

errose28 · 2026-03-09T15:01:43Z

Failure in TestOzoneManagerRequestHandler looks related, happened on both runs, as well as in fork.

Yes, the way I'm reading the UGI returns null in tests and fails validation. I need to check how other tests are handling this and push an update.

github-actions · 2026-03-31T00:10:57Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

github-actions · 2026-04-07T00:11:36Z

Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.

* HDDS-14496-zdu: (363 commits) HDDS-15639. Adjust Upgrade status command to call OM intead of SCM (apache#10579) HDDS-15622. New finalize command should check OM server version (apache#10548) HDDS-15609. Legacy SCM Finalize command should become a no-op (apache#10543) HDDS-15528. Adjust upgrade finalize command to call OM instread of SCM (apache#10493) HDDS-15488. Recon upgrade actions should be idempotent (apache#10442) HDDS-15482. Add fencing based on datanode versions to SCM and Recon (apache#10504) HDDS-15374. Switch Recon to the new versioning framework (apache#10443) HDDS-15484. Decouple ComponentVersionManager from Storage (apache#10437) HDDS-15386. Remove LayoutFeature interface (apache#10376) HDDS-15379. Remove unused classes from the old upgrade framework (apache#10370) HDDS-15377. Switch SCM to use new ScmVersionManager (apache#10368) HDDS-15483. More renames missed in HDDS-15375 (apache#10436) HDDS-15376. Implement new ScmVersionManager. (apache#10367) HDDS-15208. OM should learn to finalize from SCM after finalization command (apache#10236) HDDS-15375. Renames to prepare to switch SCM to the new versioning framework. (apache#10366) HDDS-15195. Switch Datanode to the new versioning framework (apache#10215) HDDS-15263. Downgrade should happen in reverse order (apache#10269) HDDS-14821. Version specific data validation in rolling-upgrade suite (apache#10033) HDDS-15196. Split HDDS upgrade actions into specific SCM and Datanode actions. (apache#10206) HDDS-14670. SCM queryFinalizeStatus shouldFinalize should only be true when out of safemode (apache#10193) ... Conflicts: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMUpgradeFinalization.java hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMCancelPrepareRequest.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerStateMachine.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/protocolPB/TestOzoneManagerRequestHandler.java

* HDDS-14496-zdu: (416 commits) HDDS-15808. Replace json-simple with Jackson in httpfsgateway (apache#10710) HDDS-15753. Rename TestClock to MockClock (apache#10712) HDDS-14759. Local Empty file is written and retained even when key write fails (apache#10707) HDDS-15647. Bump Node.js to 24.17.0 (apache#10590) HDDS-15783. Handle empty lists in ScmListCodec deserialization (apache#10690) HDDS-15768. Bracket IPv6 literals in Ratis peer addresses (apache#10686) HDDS-15679. [Recon] Fix DN dropdown mismatch with pending deletion API (apache#10615). HDDS-15692. Enable GitHub Copilot code review when a pull request is opened (apache#10705) HDDS-15739. Fix ListObjectsV2 continuation-token: wrong XML element name and empty-token handling (apache#10668) HDDS-15648. Improve TestSnapshotDiffManager#testThreadPoolIsFull drain scenario (apache#10632) HDDS-15803. Fix parameter number warning in OMKeyRequest.allocateBlock. (apache#10701) HDDS-15480. Potential for NPE / infinite loop in StreamBlockReader (apache#10431) HDDS-15523. Speed up TestOMDbCheckpointServletInodeBasedXfer (apache#10680) HDDS-15788. Fix SST filtering BOOTSTRAP_LOCK contention introduced by RocksDB 10 upgrade (apache#10692) HDDS-15752. Rename base tests and helpers for naming convention in OM (apache#10672) HDDS-15754. Rename TestableCluster to MockCluster (apache#10691) HDDS-15755. Bump develocity-maven-extension to 2.5.0 (apache#10674) HDDS-14956. Add S3 HAProxy to ozonesecure-ha/test-haproxy-s3g.sh (apache#10685) HDDS-15762. Remove mini-chaos-tests (apache#10682) HDDS-14577. Handle missing metadata dir when updating container state (apache#10565). ... Conflicts: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/upgrade/TestOMCancelPrepareRequest.java

errose28 added 2 commits February 6, 2026 16:22

Initial removal of prepare for upgrade

741d84b

Generated-By: Cursor

Add initial draft of tests

f61268e

Generated-By: Cursor

errose28 added the zdu Pull requests for Zero Downtime Upgrade (ZDU) https://issues.apache.org/jira/browse/HDDS-14496 label Feb 6, 2026

yandrey321 reviewed Feb 7, 2026

View reviewed changes

Comment thread ...one-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java Outdated

errose28 added 3 commits February 9, 2026 14:26

Make prepare_completed the default state

23b6521

Update CLI and request comments/descriptions

530e096

Fix acceptance test files

b9f97ea

errose28 commented Feb 17, 2026

View reviewed changes

Update comments and messages

260a76f

errose28 marked this pull request as ready for review February 19, 2026 16:42

errose28 marked this pull request as draft February 19, 2026 16:44

errose28 added 7 commits February 19, 2026 11:54

Fix prepare request send to follower classification

bfee07b

Fix existing client side prepare bug exposed by new test

c5a9fd9

Restore state machine polling in upgrade finalization test

5a9ef56

Mark open key tests as unhealthy

631f842

Install snapshot test passes intermittently

02749ef

Revert "Install snapshot test passes intermittently"

d201f3f

This reverts commit 02749ef.

Remove finalize from snapshot test for now

22b5ef5

errose28 marked this pull request as ready for review February 23, 2026 02:37

sodonnel approved these changes Feb 23, 2026

View reviewed changes

Restore admin checks

414d2ed

sodonnel approved these changes Mar 4, 2026

View reviewed changes

github-actions Bot added the stale label Mar 31, 2026

github-actions Bot closed this Apr 7, 2026

Mock isAdmin for tests

d5891a8

errose28 reopened this Jul 13, 2026

errose28 force-pushed the HDDS-14580-remove-prepare branch from 67bf087 to 2a9942c Compare July 13, 2026 23:05

errose28 marked this pull request as draft July 13, 2026 23:05

github-actions Bot removed the stale label Jul 14, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-14580. Remove OM Prepare for Upgrade Code for ZDU#9723

HDDS-14580. Remove OM Prepare for Upgrade Code for ZDU#9723
errose28 wants to merge 17 commits into
apache:HDDS-14496-zdufrom
errose28:HDDS-14580-remove-prepare

errose28 commented Feb 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

errose28 Feb 17, 2026

Uh oh!

sodonnel left a comment

Uh oh!

errose28 commented Feb 27, 2026

Uh oh!

sodonnel left a comment

Uh oh!

adoroszlai commented Mar 9, 2026

Uh oh!

errose28 commented Mar 9, 2026

Uh oh!

github-actions Bot commented Mar 31, 2026

Uh oh!

github-actions Bot commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

errose28 commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Uh oh!

errose28 Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

sodonnel left a comment

Choose a reason for hiding this comment

Uh oh!

errose28 commented Feb 27, 2026

Uh oh!

sodonnel left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Mar 9, 2026

Uh oh!

errose28 commented Mar 9, 2026

Uh oh!

github-actions Bot commented Mar 31, 2026

Uh oh!

github-actions Bot commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

errose28 commented Feb 6, 2026 •

edited

Loading