Skip to content

[fix][test] Fix flaky ExtensibleLoadManagerImplTest.initializeState by recovering wedged channel ownership#25977

Merged
merlimat merged 2 commits into
apache:masterfrom
lhotari:lh-fix-flaky-elb-initializestate
Jun 8, 2026
Merged

[fix][test] Fix flaky ExtensibleLoadManagerImplTest.initializeState by recovering wedged channel ownership#25977
merlimat merged 2 commits into
apache:masterfrom
lhotari:lh-fix-flaky-elb-initializestate

Conversation

@lhotari

@lhotari lhotari commented Jun 8, 2026

Copy link
Copy Markdown
Member

Builds on #25976, which first hardened initializeState against channel disruption by force-serving
the channel topic (admin lookup + getStats) on each unload attempt. This PR adds a stronger
fallback for the case that force-serve alone cannot recover. (#25976 was merged while this PR was
open and is now merged into this branch.)

Motivation

ExtensibleLoadManagerImplTest.initializeState (a @BeforeMethod) intermittently fails with
org.awaitility.core.ConditionTimeoutException: Assertion condition null within 2 minutes.

Root cause (confirmed from the failing run's test-report XML and thread dump): the failing
initializeState runs immediately after a role-churning test such as testRoleChangeIdempotency,
whose direct playLeader()/playFollower() calls can leave the channel system topic
persistent://pulsar/system/loadbalancer-service-unit-state in an owns-but-doesn't-serve state.
Clients looking it up get "... not served by this instance:localhost:<port>. Please redo the lookup."
(logged ~120× over the 120s window) while the channel's ServiceUnitState reports a different
owner, and the channel's internal producer falls into escalating reconnect backoff (observed up to
61s) and never recovers. monitor() only reconciles broker roles (already consistent here), so the
unload's channel publish keeps failing for the full window and Awaitility times out. (The
Assertion condition null wording is just Awaitility reporting that only ignored / non-assertion
exceptions occurred; the thread dump confirmed no thread was actually stuck.)

The force-serve heal from #25976 handles the common case (an owner is recorded but the bundle is not
loaded). This PR adds a stronger fallback for the harder wedge where the lookup keeps resolving a
broker that refuses to serve.

Modifications

ExtensibleLoadManagerImplBaseTest.initializeState now:

  1. Tries a 30s fast path — awaitTestNamespaceUnloaded(30) — which drives monitor(), force-serves
    the channel topic (the lookup + getStats heal from [fix][test] Fix flaky ExtensibleLoadManagerImplTest by re-serving the channel topic in initializeState #25976), and unloads the test namespace.
  2. If that times out (the channel is still wedged), calls a new recoverChannelOwnership() and
    retries the unload for 60s.

recoverChannelOwnership() forces a clean channel owner via leader re-election: it closes the
current channel owner's LeaderElectionService so the other broker wins the election; that broker's
playLeader() re-creates and re-serves the channel topic, and the ownership change makes clients redo
their stale lookups. This reuses the exact LES-bounce pattern already used by makePrimaryAsLeader().
In the common (healthy) case the fast path succeeds immediately and the recovery never runs, so
behavior is unchanged.

A now-stale comment in ExtensibleLoadManagerImplTest (describing the initializeState backstop) is
updated to match.

Verifying this change

  • Make sure that the change passes the CI checks.

This change is already covered by existing tests (ExtensibleLoadManagerImplTest, run in the flaky
group with -PexcludedTestGroups=''). Verified locally:

  • testRoleChangeIdempotency + testHandleNoChannelOwner pass (both serviceUnitStateTableViewClassName parametrizations).
  • The recovery path was exercised by temporarily forcing it on every initializeState invocation (LES bounce before each test method); all selected tests still passed, confirming the recovery is benign in a healthy cluster.
  • spotlessJavaCheck and checkstyleTest are green.

The original wedge is timing-dependent and was not reproduced locally; confidence rests on the
artifact-confirmed root cause plus the above.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

This change is test-only (no production code changes).

…y recovering wedged channel ownership

The initializeState @BeforeMethod times out (ConditionTimeoutException after
120s) when it runs right after a role-churning test such as
testRoleChangeIdempotency, whose direct playLeader()/playFollower() calls can
leave the channel system topic (loadbalancer-service-unit-state) owned by a
broker that no longer serves it ("not served by this instance, redo the lookup"),
with the channel producer stuck in escalating reconnect backoff. The existing
monitor()+unload retry cannot heal this because monitor() only reconciles broker
roles, which are already consistent in this state.

When the fast unload retry does not recover, force a clean channel owner via
leader re-election: closing the current owner's LeaderElectionService makes the
other broker win the election, and its playLeader() re-creates and re-serves the
channel topic while the ownership change makes clients redo their stale lookups.

Assisted-by: Claude Code (Opus 4.8)
@lhotari

lhotari commented Jun 8, 2026

Copy link
Copy Markdown
Member Author

might not be needed after #25976

…itializestate

# Conflicts:
#	pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/ExtensibleLoadManagerImplBaseTest.java
@merlimat merlimat merged commit 65b4321 into apache:master Jun 8, 2026
43 checks passed
@lhotari lhotari added this to the 5.0.0-M1 milestone Jun 10, 2026
lhotari added a commit that referenced this pull request Jun 10, 2026
lhotari added a commit that referenced this pull request Jun 10, 2026
priyanshu-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 15, 2026
…y recovering wedged channel ownership (apache#25977)

(cherry picked from commit 476881f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants