Skip to content

KAFKA-20854 A more obvious busy loop due to KIP-909(WIP) - #23014

Open
m1a2st wants to merge 1 commit into
apache:trunkfrom
m1a2st:KAFKA-20854
Open

KAFKA-20854 A more obvious busy loop due to KIP-909(WIP)#23014
m1a2st wants to merge 1 commit into
apache:trunkfrom
m1a2st:KAFKA-20854

Conversation

@m1a2st

@m1a2st m1a2st commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

FetchRequestManager currently wakes up the FetchBuffer
unconditionally whenever prepareFetchRequests() returns no requests
to send. This behavior was intended for one specific case: all
currently fetchable partitions already have buffered data waiting to be
consumed. However, the wakeup also happens for any other case that
produces an empty result, including states where no progress can be
made until some external event occurs:

  • No partitions are fetchable at all (no assignment yet, invalid
    positions, paused, pending revocation/callback). - The
    partition's leader is unknown (metadata.requestUpdate() was just
    triggered). - The target node is in its reconnect backoff window.
  • The target node already has an in-flight fetch request outstanding.
  • The target node hosts other partitions that already have buffered
    data.

Because this wakeup bypasses the caller's normal retryBackoffMs delay,
the application thread and the background ConsumerNetworkThread can
enter a tight loop: poll() issues a fetch-request event, the
background thread finds nothing to send, wakes the buffer, poll()
returns immediately, and the cycle repeats without any meaningful
delay.

This is a pre-existing issue introduced with the non-blocking
AsyncPollEvent redesign in KAFKA-18376. However, KIP-909's async
bootstrap DNS resolution extends the "no node available" window from
effectively instantaneous to up to bootstrapResolveTimeoutMs, making
the busy loop much more visible.

@github-actions github-actions Bot added triage PRs from the community consumer clients small Small PRs labels Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clients consumer small Small PRs triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant