[fix][broker] Fix ownership-generation races in OwnershipCache removeOwnership and lock-expiry cleanup#26197
Open
SongOf wants to merge 1 commit into
Open
Conversation
…Ownership and lock-expiry cleanup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
OwnershipCache.removeOwnership(bundle)removes whicheverResourceLockhappens to be inlocallyAcquiredLocksat that moment, without checking that it belongs to the ownership"generation" the caller was working on. Three defects follow from this:
A stale unload chain can destroy a re-acquired ownership. When a bundle's resource lock
expires, the lock-expiry callback in the cache loader fires
unloadNamespaceBundle(...)asynchronously (dropping its future) and invalidates the local owner cache immediately. A
concurrent lookup can then re-acquire the bundle (new lock, new
OwnedBundle) while the staleunload chain is still running. When that chain reaches its final
removeOwnership(bundle), itremoves and releases the newly acquired lock, deleting the new owner's znode while the local
cache still claims active ownership. This opens a transient dual-serving window (fenced-ledger
errors on persistent topics, silent message loss on non-persistent topics) and causes ownership
flapping. The same stale chain also deactivates the newer
OwnedBundlethrough the cache lookupin
updateBundleState(bundle, false).removeOwnershipreports success while an acquisition is in flight, leaving zombieownership. The lock is only put into
locallyAcquiredLocksafteracquireLockcompletes, soa
removeOwnershipcall racing with an in-flight acquisition finds the map empty, returns acompleted future ("released"), and the lock installed afterwards is never cleaned up: the broker
silently keeps the lock and the cache entry for a bundle the caller believes released (e.g. a
deleted bundle/namespace), polluting load reports until restart.
The lock-expiry callback never removes the expired lock from
locallyAcquiredLocks(asymmetric add/remove bookkeeping), so a dead lock handle lingers in the map and drifts from
the owned-bundle cache. The callback also drops the future returned by
unloadNamespaceBundle(...), so failures of the triggered unload are silent.Modifications
OwnedBundleto theResourceLockacquired for it, so anOwnedBundleinstancerepresents a single ownership generation. Add
removeOwnership(OwnedBundle)which releases onlythat generation via
ConcurrentHashMap.remove(key, value); if the bundle has since beenre-acquired, the newer ownership is left untouched.
OwnedBundle.handleUnloadRequestnow uses itand no longer calls
updateBundleState(its ownisActiveflag is already flipped via CASearlier in the method; the cache lookup could deactivate a newer generation).
removeOwnership(NamespaceBundle)now waits for an in-flight acquisition to settle and thenreleases whatever it installed, instead of reporting success while the pending lock leaks. Its
"release the current ownership, whatever generation" semantics are intentionally kept for the
bundle-deletion and split paths.
locallyAcquiredLockswith ageneration-safe two-arg remove (mirroring
LockManagerImpl's own lock bookkeeping), invalidatesthe owned-bundle cache entry only if it still holds this generation, and logs failures of the
triggered unload instead of dropping the future.
Verifying this change
This change added tests and can be verified as follows:
Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes