HDDS-15918. Optimize TestSCMNodeManager execution time and stability by reducing heartbeat configs - #10837
HDDS-15918. Optimize TestSCMNodeManager execution time and stability by reducing heartbeat configs#10837rich7420 wants to merge 4 commits into
Conversation
…by reducing heartbeat configs Lower the second-level heartbeat interval and stale/dead node timeouts to 100ms/1s/2s and replace fixed Thread.sleep waits with GenericTestUtils.waitFor polling so the node-state transition tests converge quickly and tolerate slow CI. Reduces TestSCMNodeManager wall time ~104s -> ~82s with all 48 tests passing. Config-validation and stats tests that do not wait on stale/dead are left unchanged.
|
cc @chungen0126 |
|
Thanks @rich7420 for working on this. Could you also share the run time of the test? I hope it will speed up the test. |
|
Thanks @chungen0126! Full TestSCMNodeManager goes from ~104s to ~82s on my machine (48 tests, about 20% faster). |
| conf.setTimeDuration(OZONE_SCM_STALENODE_INTERVAL, 1, SECONDS); | ||
| conf.setTimeDuration(OZONE_SCM_DEADNODE_INTERVAL, 2, SECONDS); |
There was a problem hiding this comment.
Could you try setting OZONE_SCM_STALENODE_INTERVAL to 300 milliseconds and OZONE_SCM_DEADNODE_INTERVAL to 600 milliseconds, and check the stability of the test?
There was a problem hiding this comment.
ok, here's the results
https://github.com/rich7420/ozone/actions/runs/31389348030%5D
…DetectStaleAndDeadNode
| nodeManager.processHeartbeat(dn); | ||
| } | ||
| return nodeManager.getNodeCount(NodeStatus.inServiceStale()) == 1; | ||
| }, interval, 10 * 1000); |
There was a problem hiding this comment.
nit: max wait time should be named constant.
| nodeManager.processHeartbeat(dn); | ||
| } | ||
| return nodeManager.getNodeCount(NodeStatus.inServiceDead()) == 1; | ||
| }, interval, 10 * 1000); |
There was a problem hiding this comment.
nit: use the same named constant as above.
| Thread.sleep(500); | ||
| nodeManager.processHeartbeat(healthyNode); | ||
| Thread.sleep(2 * 1000); | ||
| Thread.sleep(667); |
yandrey321
left a comment
yandrey321
left a comment
There was a problem hiding this comment.
Please replace magic numbers with named constants, some of these constants might be reusable between tests.
What changes were proposed in this pull request?
Since TestSCMNodeManager should be lighter as it is an unit test. The heartbeat intervals and node state timeouts are configured in seconds, which unnecessarily prolongs the test duration. It will be better to set heartbeat interval and timeouts to a lower value. For example 100ms for heartbeats.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15918
How was this patch tested?
https://github.com/rich7420/ozone/actions/runs/29815586760
try 100times
https://github.com/rich7420/ozone/actions/runs/29815603643