[fix][fn] Fix orphan exclusive producer on creation timeout in WorkerUtils.createExclusiveProducerWithRetry#25942
Merged
lhotari merged 1 commit intoJun 5, 2026
Conversation
…ateExclusiveProducerWithRetry ### Motivation When the thread calling WorkerUtils.createExclusiveProducerWithRetry was interrupted while waiting for the producer creation to complete, the InterruptedException thrown by Future.get was swallowed by the generic `catch (Exception e)` retry handler. The retry loop kept running with the interrupt status cleared, and the in-flight createAsync() future was abandoned: when the exclusive producer was eventually created, nothing closed it, leaking the producer and holding on to the topic's exclusive producer access. The `.get(10, TimeUnit.SECONDS)` timeout had the same abandonment problem: a timed-out attempt left the pending producer creation behind while a new attempt was started. ### Modifications - Wait on the producer creation with FutureUtil.getAndCleanupOnInterrupt(producerFuture, Producer::closeAsync) so that an interrupt registers a cleanup that closes the producer if the pending creation completes later. - Handle InterruptedException explicitly: restore the interrupt status and propagate it, aborting the retry loop instead of retrying. - Add WorkerUtilsTest.testCreateExclusiveProducerWithRetryClosesProducerOnInterrupt covering that an interrupt aborts the retry loop, restores the interrupt status, and closes the producer once the pending creation completes.
3 tasks
eolivelli
approved these changes
Jun 5, 2026
nodece
approved these changes
Jun 5, 2026
priyanshu-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Jun 8, 2026
…Utils.createExclusiveProducerWithRetry (apache#25942) (cherry picked from commit 2177b0e) (cherry picked from commit d428118)
priyanshu-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Jun 9, 2026
…Utils.createExclusiveProducerWithRetry (apache#25942) (cherry picked from commit 2177b0e) (cherry picked from commit d428118)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #25936
Motivation
WorkerUtils.createExclusiveProducerWithRetrybounded the exclusive producer creation withcreateAsync().get(10, TimeUnit.SECONDS)inside a retry loop. Whenget()threwTimeoutException, the catch block logged and retried, abandoning the in-flightCompletableFuture<Producer>. The underlying producer state machine kept running on the client IO thread, and when the creation eventually succeeded, the producer registered with the broker as an orphan: no caller held a reference to it, so it was never closed.The orphan held the topic's exclusive producer access, so every subsequent retry was rejected — locking up leader election in the function worker, since the elected leader could not acquire the exclusive producer it needs on the internal topics and remained stuck retrying. The orphan was never reaped: the client never sends
CloseProducerfor it (no reference exists to invokeclose()), the TCP connection stays alive because it is shared with other producers/consumers on the samePulsarClient, and the functions internal topics are created with deduplication disabled, so the broker's inactivity-based producer reaper does not apply.In addition, the generic
catch (Exception e)swallowedInterruptedException, so the retry loop kept running with the interrupt status cleared and abandoned the in-flight creation the same way. This is theWorkerUtilscounterpart of the client-side interrupt fix in #24539.Modifications
.getcall:.get(10, TimeUnit.SECONDS)is effectively replaced with.get(), since the Pulsar client already bounds producer creation with the client'soperationTimeout(applied via the lookup timeout, which defaults tooperationTimeout) and completes the future exceptionally instead of abandoning it. This eliminates theTimeoutExceptionpath that orphaned in-flight producer creations.FutureUtil.getAndCleanupOnInterrupt(producerFuture, Producer::closeAsync)(the helper introduced in [fix][client] Close orphan producer or consumer when the creation is interrupted #24539) so that an interrupt registers a cleanup that closes the producer if the pending creation completes later.InterruptedExceptionexplicitly: restore the interrupt status and propagate it, aborting the retry loop instead of retrying.Verifying this change
This change added tests and can be verified as follows:
WorkerUtilsTest.testCreateExclusiveProducerWithRetryClosesProducerOnInterruptverifying that an interrupt aborts the retry loop instead of retrying, restores the interrupt status, and closes the producer once the pending creation completes after the interrupt.Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes