KAFKA-20876: Retry windowed restore after control record - #23029
Conversation
A control record at the end of a changelog partition is filtered from the consumer, so the existing end-offset probe can return no data and abandon the retention-based restore optimization. Retry from the preceding offset before falling back to the beginning, and cover both visible and filtered latest offsets. Generated-by: OpenAI Codex (GPT-5)
|
Verification update: the full |
|
Thanks for working on this. I tried the probe sequence with First, the retry still returned empty: Second, the Here, the latest visible record was eight offsets back because the aborted batch was filtered as a whole. Would it be better to start the probe from an earlier offset, clamped to I can share the reproduction test if needed. |
Summary
When a windowed changelog ends with a transaction control record, the
restore consumer filters that record from poll results. Probing only
endOffset - 1therefore returns no record and incorrectly abandons theretention-based restore optimization.
This change retries the probe from the preceding offset before falling
back to the beginning. The regression test covers both a visible latest
record and the filtered control-record case.
Testing
because restoration fell back to the beginning.
./gradlew :streams:test --tests org.apache.kafka.streams.processor.internals.StoreChangelogReaderTest.shouldSeekByTimestampForWindowedStoreWithoutCheckpoint --no-build-cache --console=plain./gradlew :streams:test --no-build-cache --console=plain(BUILD SUCCESSFUL, 6m 54s)SpotBugs tasks successfully.
git diff --checkJira: https://issues.apache.org/jira/browse/KAFKA-20876 Reviewers: Mingi
Cho (github:ChoMinGi)