HDDS-15849. Force channel shutdown in XceiverClientGrpc.close

[yandrey321]

What changes were proposed in this pull request?

XceiverClientGrpc.close() performs a blocking graceful channel shutdown, and because clients are torn down through XceiverClientManager's cache eviction, that blocking wait runs while the clientCache monitor is held — serializing every concurrent acquireClient() / releaseClient() call.
HDDS-14571 changed close() from a forced shutdownNow() (which cancels in-flight RPCs and terminates in ~1 ms) to a graceful shutdown() followed by a polling loop that always sleeps at least SHUTDOWN_WAIT_INTERVAL_MILLIS (100 ms) before it can observe termination:

for (ChannelInfo channelInfo : dnChannelInfoMap.values()) {
  channelInfo.getChannel().shutdown();
}
while (!nonTerminatedChannels.isEmpty() && System.nanoTime() < deadline) {
  nonTerminatedChannels.removeIf(ManagedChannel::isTerminated);
  Thread.sleep(SHUTDOWN_WAIT_INTERVAL_MILLIS); // >= 100ms floor per close()
}

Every close() now costs ≥100 ms instead of ~1 ms. The eviction path that invokes it holds the cache monitor throughout:

acquireClient()                      // synchronized (clientCache)
  -> getClient()
    -> clientCache.get(key, newClient)
       -> Guava evicts an entry on the calling thread
          -> RemovalListener.onRemoval()   // synchronized (clientCache)
             -> XceiverClientSpi.setEvicted() -> cleanup() -> close()  // blocks here

So concurrent readers that create/evict short-lived clients (e.g. EC read/checksum collection, which uses a distinct client per placement group) serialize behind a ≥100 ms lock hold per closed channel and degrade non-linearly with concurrency.
Fix: restore immediate termination in close() — use channel.shutdownNow() followed by channel.awaitTermination(SHUTDOWN_WAIT_MAX_SECONDS, ...), and drop the mandatory-sleep polling loop. This keeps HDDS-14571's goal (no synchronized methods on XceiverClientGrpc) while eliminating the blocking wait under the cache lock.

Co authored with Claude Code

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15849

How was this patch tested?

Added TestXceiverClientManagerCloseContention, a unit test that exercises the real XceiverClientManager and XceiverClientGrpc.close() (real lazily-connected plaintext gRPC channels to random local datanode addresses; no MiniOzoneCluster and no live datanode required, since close() only shuts channels down). maxCacheSize=1 plus a distinct pipeline per iteration forces an eviction-driven close on nearly every acquisition; 8 threads × 8 cycles produce 63 closes. It measures average wall-clock per close and asserts it stays well under the pre-fix ~100 ms floor.
The same test, unmodified, against the code before and after this change:

run	wall-clock	ms / close	result
Before (graceful shutdown() + 100 ms poll)	6874 ms	109 ms	❌ fails
After (shutdownNow() + awaitTermination)	201 ms	3 ms	✅ passes

~34× faster per close

[jojochuang]

cc @ptlrs

[Copilot]

Pull request overview

This PR addresses the HDDS-15849 performance regression by removing the fixed 100ms polling sleep from XceiverClientGrpc.close() and adding a regression test that detects close-time contention caused by eviction-driven shutdown under the XceiverClientManager cache monitor.

Changes:

• Switch XceiverClientGrpc.close() from graceful shutdown + sleep polling to shutdownNow() + awaitTermination() to avoid the mandatory 100ms floor per close.
• Add TestXceiverClientManagerCloseContention to assert eviction-driven closes do not serialize concurrent acquire/release activity.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java	Reworks gRPC channel shutdown behavior in close() to eliminate the fixed sleep-based termination polling.
hadoop-hdds/client/src/test/java/org/apache/hadoop/hdds/scm/TestXceiverClientManagerCloseContention.java	Adds a regression test that detects cache-eviction close contention and asserts acceptable per-close wall-clock cost.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yandrey321]

its an old behavior that was supposed to be addressed by HDDS-14571, I'll file a new JIRA to get the better fix that does not introduce this regression.