KAFKA-20854 A more obvious busy loop due to KIP-909(WIP) - #23014
Open
m1a2st wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FetchRequestManagercurrently wakes up theFetchBufferunconditionally whenever
prepareFetchRequests()returns no requeststo send. This behavior was intended for one specific case: all
currently fetchable partitions already have buffered data waiting to be
consumed. However, the wakeup also happens for any other case that
produces an empty result, including states where no progress can be
made until some external event occurs:
positions, paused, pending revocation/callback). - The
partition's leader is unknown (
metadata.requestUpdate()was justtriggered). - The target node is in its reconnect backoff window.
data.
Because this wakeup bypasses the caller's normal
retryBackoffMsdelay,the application thread and the background
ConsumerNetworkThreadcanenter a tight loop:
poll()issues a fetch-request event, thebackground thread finds nothing to send, wakes the buffer,
poll()returns immediately, and the cycle repeats without any meaningful
delay.
This is a pre-existing issue introduced with the non-blocking
AsyncPollEventredesign in KAFKA-18376. However, KIP-909's asyncbootstrap DNS resolution extends the "no node available" window from
effectively instantaneous to up to
bootstrapResolveTimeoutMs, makingthe busy loop much more visible.