HDDS-16097. Add unit tests for OzoneManagerRatisServer.checkRetryCache - #10985
HDDS-16097. Add unit tests for OzoneManagerRatisServer.checkRetryCache#10985jojochuang wants to merge 6 commits into
Conversation
Cover HDDS-13621: failed RetryCache entries return null instead of NPE, and successful cache hits still return the cached OMResponse. Add HA integration test for retry after a failed Ratis reply on a follower that later becomes leader. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: Ic5e27338b95473574bd4476196bb1124d5559e86
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I4e4a3c851678e4aa19301bbe518b062bd50a1a8e
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I4ec19aaa396a89e3ba109947e459315a90c6c7c6
Stub getLeaderId on the spied Ratis server so the successful cache-hit unit test does not NPE on mocked Division.getInfo(). Remove the HA integration test; Ratis does not populate RetryCache on follower NotLeader submits the way the test assumed. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I640404af5e43c874277b56d67d92522e93024a65
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I1090a0992f92dc51e931ec1030dab9b8f97eda76
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: Id30647ab9fb321e0027290a499895c647c9c0725
There was a problem hiding this comment.
Pull request overview
This PR adds JUnit coverage in TestOzoneManagerRatisServer to lock in the HDDS-13621 behavior of OzoneManagerRatisServer.checkRetryCache() returning null when a cached RaftClientReply is unsuccessful, while still returning an OMResponse on successful cached replies.
Changes:
- Adds a unit test asserting
checkRetryCache()returnsnullwhen the cachedRaftClientReplyindicates failure. - Adds a unit test asserting
checkRetryCache()returns a cachedOMResponsewhen the cachedRaftClientReplyindicates success. - Introduces a helper to inject a mocked RetryCache entry into a spied
OzoneManagerRatisServer.
Suppressed comments (3)
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerRatisServer.java:209
- To make this test exercise the same clientId mapping used by OzoneManagerRatisServer.getClientId() (UUID.nameUUIDFromBytes(Server.getClientId())), build ratisClientId from the clientIdBytes you place into Server.Call, rather than using ClientId.randomId() and then serializing it into Server.Call.
ClientId ratisClientId = ClientId.randomId();
int callId = 43;
OMResponse expected = OMResponse.newBuilder()
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerRatisServer.java:229
- The RetryCache in injectRetryCacheEntry() is currently stubbed with a broad matcher, so these tests don’t verify that checkRetryCache() actually queries the cache using the expected ClientInvocationId (clientId + callId). Consider capturing the RetryCache mock so the test can verify the exact key used.
OzoneManagerRatisServer spyServer = spy(omRatisServer);
injectRetryCacheEntry(spyServer, ratisClientId, callId, successReply);
doReturn(omRatisServer.getRaftPeerId()).when(spyServer).getLeaderId();
Server.Call previousCall = Server.getCurCall().get();
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerRatisServer.java:246
- injectRetryCacheEntry() currently hides the RetryCache mock inside the helper, which makes it hard for callers to verify that the correct key was used. Returning the RetryCache allows each test to assert the cache lookup was performed with the expected ClientInvocationId without loosening the stubbing.
private static void injectRetryCacheEntry(OzoneManagerRatisServer spyServer,
ClientId clientId, int callId, RaftClientReply reply) throws Exception {
RetryCache.Entry cacheEntry = mock(RetryCache.Entry.class);
when(cacheEntry.getReplyFuture())
.thenReturn(CompletableFuture.completedFuture(reply));
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| ClientId ratisClientId = ClientId.randomId(); | ||
| int callId = 42; | ||
| RaftGroupMemberId memberId = RaftGroupMemberId.valueOf( | ||
| omRatisServer.getRaftPeerId(), omRatisServer.getRaftGroup().getGroupId()); | ||
| RaftClientReply failedReply = RaftClientReply.newBuilder() | ||
| .setClientId(ratisClientId) | ||
| .setServerId(memberId) | ||
| .setGroupId(omRatisServer.getRaftGroup().getGroupId()) | ||
| .setCallId(callId) | ||
| .setSuccess(false) | ||
| .setMessage(Message.EMPTY) | ||
| .setException(new NotLeaderException(memberId, null, Collections.emptyList())) | ||
| .build(); | ||
|
|
||
| OzoneManagerRatisServer spyServer = spy(omRatisServer); | ||
| injectRetryCacheEntry(spyServer, ratisClientId, callId, failedReply); | ||
|
|
||
| Server.Call previousCall = Server.getCurCall().get(); | ||
| try { | ||
| Server.getCurCall().set(new Server.Call(callId, 0, null, null, | ||
| RPC.RpcKind.RPC_BUILTIN, ratisClientId.toByteString().toByteArray())); | ||
| assertNull(spyServer.checkRetryCache()); | ||
| } finally { | ||
| Server.getCurCall().set(previousCall); | ||
| } |
What changes were proposed in this pull request?
This PR adds unit test coverage for
OzoneManagerRatisServer.checkRetryCache().HDDS-13621 (#9711) fixed an NPE when a cached Ratis reply was unsuccessful: the server now returns
nullinstead of callinggetOMResponse()on a failed reply. This change adds targeted unit tests to lock in that behavior and to verify the success path still returns anOMResponse.Changes:
checkRetryCacheReturnsNullWhenCachedReplyFailed— mocks a failed cachedRaftClientReplyand assertscheckRetryCache()returnsnull.checkRetryCacheReturnsOmResponseWhenCachedReplySucceeded— mocks a successful cached reply and asserts anOMResponsewithStatus.OKis returned.Related: HDDS-13621, #9711.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16097
How was this patch tested?
TestOzoneManagerRatisServer(two new test methods).flaky-test-checkon commitb3c339a): https://github.com/jojochuang/ozone/actions/runs/31129294251 —TestOzoneManagerRatisServer#ALLpassed.Generated-by: Cursor (Composer)
Made with Cursor