HDDS-15715. Add helpful CLI flags to finalization CLIs#10678
Conversation
github-actions
Bot
added
the
zdu
errose28
left a comment
errose28
left a comment
There was a problem hiding this comment.
Thanks for adding this @dombizita.
| optional int32 numDatanodesFinalized = 2; | ||
| optional int32 numDatanodesTotal = 3; | ||
| optional bool shouldFinalize = 4; | ||
| optional uint32 scmSoftwareVersion = 5; |
There was a problem hiding this comment.
This will only be the leader SCM's software version. Unlike apparent version, these may not always match between SCMs. Maybe we should only return apparent versions of all components with the verbose option, not software version?
Note that software version can also be misleading if there was no component version change across an upgrade. In that case all versions can match even if the cluster is in mixed versions.
| return numFinalizedDatanodes == totalHealthyDatanodes; | ||
| } | ||
|
|
||
| public Integer getMinApparentVersion() { |
There was a problem hiding this comment.
Why Integer object instead of int primitive for these getters?
| if (dnApparentVersionInt < minApparentVersion) { | ||
| minApparentVersion = dnApparentVersionInt; | ||
| } | ||
| if (dnApparentVersionInt > maxApparentVersion) { | ||
| maxApparentVersion = dnApparentVersionInt; | ||
| } |
There was a problem hiding this comment.
nit.
| if (dnApparentVersionInt < minApparentVersion) { | |
| minApparentVersion = dnApparentVersionInt; | |
| } | |
| if (dnApparentVersionInt > maxApparentVersion) { | |
| maxApparentVersion = dnApparentVersionInt; | |
| } | |
| minApparentVersion = Math.min(minApparentVersion, dnApparentVersionInt); | |
| maxApparentVersion = Math.max(maxApparentVersion, dnApparentVersionInt); |
| return new DatanodeFinalizationCounts(finalizedNodes, totalHealthyNodes, | ||
| minApparentVersion, maxApparentVersion); |
There was a problem hiding this comment.
Optional, but might be better to make this a builder so all the int parameters don't accidentally get swapped.
| out.println(" SCM Apparent Version: " + formatHddsVersion(hdds.hasScmApparentVersion(), | ||
| hdds.getScmApparentVersion())); | ||
| out.println(" Datanodes finalized: " + hdds.getNumDatanodesFinalized() + "/" + hdds.getNumDatanodesTotal()); | ||
| out.println(" Min DN Apparent Version: " + formatHddsVersion(hdds.hasMinDatanodeApparentVersion(), |
There was a problem hiding this comment.
| out.println(" Min DN Apparent Version: " + formatHddsVersion(hdds.hasMinDatanodeApparentVersion(), | |
| out.println(" Min Datanode Apparent Version: " + formatHddsVersion(hdds.hasMinDatanodeApparentVersion(), |
| } | ||
|
|
||
| @Override | ||
| public HddsProtos.UpgradeStatus queryUpgradeStatus() throws IOException { |
There was a problem hiding this comment.
We should block this query when SCM is in safemode just like the finalize command. This way we don't have to handle the zero datanode case.
| } | ||
|
|
||
| @Test | ||
| public void testWaitFlagCompletesImmediatelyWhenAlreadyFinalized() throws Exception { |
There was a problem hiding this comment.
The test does not actually check that the poll interval is not used when the cluster is already finalized. In the current version the wait flag actually doesn't complete immediately when the cluster is already finalized: it still waits 5 seconds.
| public void testWaitFlagCompletesImmediatelyWhenAlreadyFinalized() throws Exception { | |
| public void testStatusCalledOnceWhenAlreadyFinalized() throws Exception { |
|
|
||
| String output = outContent.toString(DEFAULT_ENCODING); | ||
| assertTrue(output.contains("Waiting interrupted")); | ||
| verify(omClient, never()).queryUpgradeStatus(); |
There was a problem hiding this comment.
Once we remove the initial delay in the polling this should call the query command once.
| try { | ||
| cmd.call(); | ||
| } catch (Exception ignored) { | ||
| // assertion in main thread |
There was a problem hiding this comment.
We can still make sure this failed with the correct result, either by checking the exception type or the thread's interrupt flag.
| assertEquals(0, cmd.call()); | ||
|
|
||
| String output = outContent.toString(DEFAULT_ENCODING); | ||
| // Verbose output uses the StatusSubCommand.printVerbose layout. |
There was a problem hiding this comment.
I don't see any checks on verbose specific output here.
2 participants
What changes were proposed in this pull request?
Added new CLI flags to the upgrade finalization CLI:
finalize --wait: the client will periodically poll the cluster for status from OM until the whole cluster is finalized. The finalize command is idempotent so waiting can be resumed at any timestatus --json: json version of the status output for scriptingstatus --verbose: includes internal information like OM and SCM software and apparent versions, the min and max DN software and apparent versions in the clusterUsed Claude Opus 4.8 for planning and coding, reviewed by me.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15715
How was this patch tested?
Added unit tests, green run on my fork: https://github.com/dombizita/ozone/actions/runs/28660447709