Skip to content

[fix][broker] Fix ownership-generation races in OwnershipCache removeOwnership and lock-expiry cleanup#26197

Open
SongOf wants to merge 1 commit into
apache:masterfrom
SongOf:fix/broker-ownership-cache-generation-race
Open

[fix][broker] Fix ownership-generation races in OwnershipCache removeOwnership and lock-expiry cleanup#26197
SongOf wants to merge 1 commit into
apache:masterfrom
SongOf:fix/broker-ownership-cache-generation-race

Conversation

@SongOf

@SongOf SongOf commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Motivation

OwnershipCache.removeOwnership(bundle) removes whichever ResourceLock happens to be in
locallyAcquiredLocks at that moment, without checking that it belongs to the ownership
"generation" the caller was working on. Three defects follow from this:

  1. A stale unload chain can destroy a re-acquired ownership. When a bundle's resource lock
    expires, the lock-expiry callback in the cache loader fires unloadNamespaceBundle(...)
    asynchronously (dropping its future) and invalidates the local owner cache immediately. A
    concurrent lookup can then re-acquire the bundle (new lock, new OwnedBundle) while the stale
    unload chain is still running. When that chain reaches its final removeOwnership(bundle), it
    removes and releases the newly acquired lock, deleting the new owner's znode while the local
    cache still claims active ownership. This opens a transient dual-serving window (fenced-ledger
    errors on persistent topics, silent message loss on non-persistent topics) and causes ownership
    flapping. The same stale chain also deactivates the newer OwnedBundle through the cache lookup
    in updateBundleState(bundle, false).

  2. removeOwnership reports success while an acquisition is in flight, leaving zombie
    ownership.
    The lock is only put into locallyAcquiredLocks after acquireLock completes, so
    a removeOwnership call racing with an in-flight acquisition finds the map empty, returns a
    completed future ("released"), and the lock installed afterwards is never cleaned up: the broker
    silently keeps the lock and the cache entry for a bundle the caller believes released (e.g. a
    deleted bundle/namespace), polluting load reports until restart.

  3. The lock-expiry callback never removes the expired lock from locallyAcquiredLocks
    (asymmetric add/remove bookkeeping), so a dead lock handle lingers in the map and drifts from
    the owned-bundle cache. The callback also drops the future returned by
    unloadNamespaceBundle(...), so failures of the triggered unload are silent.

Modifications

  • Bind each OwnedBundle to the ResourceLock acquired for it, so an OwnedBundle instance
    represents a single ownership generation. Add removeOwnership(OwnedBundle) which releases only
    that generation via ConcurrentHashMap.remove(key, value); if the bundle has since been
    re-acquired, the newer ownership is left untouched. OwnedBundle.handleUnloadRequest now uses it
    and no longer calls updateBundleState (its own isActive flag is already flipped via CAS
    earlier in the method; the cache lookup could deactivate a newer generation).
  • removeOwnership(NamespaceBundle) now waits for an in-flight acquisition to settle and then
    releases whatever it installed, instead of reporting success while the pending lock leaks. Its
    "release the current ownership, whatever generation" semantics are intentionally kept for the
    bundle-deletion and split paths.
  • The lock-expiry callback removes the expired lock from locallyAcquiredLocks with a
    generation-safe two-arg remove (mirroring LockManagerImpl's own lock bookkeeping), invalidates
    the owned-bundle cache entry only if it still holds this generation, and logs failures of the
    triggered unload instead of dropping the future.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

- *Added `OwnershipCacheTest.testStaleUnloadDoesNotReleaseReacquiredOwnership`: simulates the
  expiry-callback interleaving (cache invalidated, bundle re-acquired, then the stale unload
  chain completes) and asserts the re-acquired lock, znode, and active state survive. Fails
  before the fix (the new lock is released and the znode deleted).*
- *Added `OwnershipCacheTest.testRemoveOwnershipWithAcquisitionInFlight`: gates `acquireLock`
  behind a controllable future to call `removeOwnership` deterministically inside the in-flight
  window, then asserts no lock/cache entry/znode remains once both settle. Fails before the fix
  (zombie ownership remains).*
- *Added `OwnershipCacheTest.testExpiredLockIsRemovedFromLocallyAcquiredLocks`: releases the lock
  without going through `removeOwnership` (same `expiredFuture` path as a session expiry) and
  asserts `locallyAcquiredLocks` is cleaned up together with the cache. Fails before the fix
  (dead lock handle lingers).*
- *All 8 pre-existing `OwnershipCacheTest` tests, `NamespaceServiceTest` (27),
  `NamespaceUnloadingTest`, `NamespaceOwnershipListenerTest` and
  `OwnerShipCacheForCurrentServerTest` pass locally.*

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant