HDDS-15823. Fix flaky TestFailureHandlingByClient#testContainerExclusionWithClosedContainerException#10721
HDDS-15823. Fix flaky TestFailureHandlingByClient#testContainerExclusionWithClosedContainerException#10721smengcl wants to merge 1 commit into
Conversation
…ionWithClosedContainerException Remove the getPipelineIds().isEmpty() assertion, which is not an invariant under the default ALL_COMMITTED watch level: a WATCH RaftRetryFailureException takes the else branch in KeyOutputStream.handleException and adds the pipeline to the exclude list. The container-id exclusion assertion, the actual property under test, is kept. Follow-up to HDDS-15605, which removed the analogous datanodes assertion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR stabilizes an intermittent integration test failure in the Ozone RPC client tests by removing an assertion that is not a valid invariant under the default Ratis ALL_COMMITTED watch level, while keeping the core container-exclusion behavior under test.
Changes:
- Removed the assertion that the client exclude list must have an empty pipeline-ID set in
testContainerExclusionWithClosedContainerException. - Added an explanatory comment documenting why pipeline IDs may legitimately appear in the exclude list under
ALL_COMMITTED.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@chihsuan could you please review? |
chihsuan
left a comment
There was a problem hiding this comment.
Thanks for the patch! @smengcl
I just looked into this a bit more and agree, under ALL_COMMITTED the pipeline can legitimately get excluded, so an empty set isn't guaranteed. LGTM.
One other option would be to pin this test to MAJORITY_COMMITTED so the assertion still holds, but that changes the config being tested, and watch-level exclusion is already covered by testDatanodeExclusionWithMajorityCommit, so dropping it seems better.
Left one small optional note on the doc comment.
| // ClosedContainerException path runs; that takes the else-branch in | ||
| // KeyOutputStream.handleException and adds the pipeline to the exclude | ||
| // list, so an empty pipeline set is not an invariant for this config. | ||
| // The container-id assertion above is the actual property under test. |
There was a problem hiding this comment.
nit: maybe fold this into the datanode comment above so it's one comment, not two? Something like:
// Only the container-id exclusion is asserted here. Under the default
// ALL_COMMITTED watch level neither the datanode nor the pipeline set is an
// invariant: a slow-but-healthy follower can be recorded as a failed
// datanode, and a WATCH RaftRetryFailureException takes the else-branch in
// KeyOutputStream.handleException and excludes the pipeline. Watch-level
// exclusion is covered by testDatanodeExclusionWithMajorityCommit.
Optional, feel free to skip.
Generated-by: Claude Code (Opus 4.8)
What changes were proposed in this pull request?
TestFailureHandlingByClient#testContainerExclusionWithClosedContainerExceptionfails intermittently withExpecting empty but was: [Pipeline-...]on the assertion that the client exclude list holds no pipeline IDs (seen in upstream CI on 2026-07-06, after HDDS-15605 removed the analogous datanodes assertion).The test forces a container to close and writes again, expecting a
ClosedContainerExceptionthat adds only the container to the exclude list. Under the defaultALL_COMMITTEDwatch level, a slow or unavailable follower can instead surface a WATCHRaftRetryFailureException. InKeyOutputStream.handleException,checkForRetryFailurereturns true for that exception, so the container exclusion branch is skipped and the else branch adds the pipeline to the exclude list. That is intended client behavior, so an empty pipeline set is not an invariant for this configuration.This PR removes the
getPipelineIds().isEmpty()assertion, consistent with how HDDS-15605 handled the datanodes case, and adds a comment explaining why the pipeline set is not an invariant here. The container ID exclusion assertion, which is the actual property under test, is retained. The change is limited to the test.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15823
How was this patch tested?
Ran the test 100 times locally with 0 failures, each run confirmed to execute the method (
Tests run: 1, Failures: 0, Errors: 0). Ran the fullTestFailureHandlingByClientclass (6 tests, 0 failures). Checkstyle clean.