Skip to content

Fix subscription WAL replay cursor boundaries - #18810

Merged
jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix/subscription-at-end-cursor
Oct 10, 2026
Merged

jt2594838 merged 2 commits into
apache:masterfrom
Caideyipi:fix/subscription-at-end-cursor

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Description

The subscription replay cursor stores the next local search index to read. seekToBeginning() used the empty-WAL sentinel 0, although valid local indexes start at 1. Replaying the first local record therefore reported an unavailable [0, 1) range and incremented walGapSkippedEntries despite delivering the record. At the other boundary, AT_END replay lookup and seekToEnd() used the last written index, allowing the tail record to be reread and exposing a one-entry raw WAL gap.

Use FIRST_CONSENSUS_SEARCH_INDEX for beginning seeks and the index after the last written record for end positions. Preserve per-writer recovery progress and replay of follower records without a local search index. Real missing local indexes remain counted.

Add public-API seek regression tests that run seek reset and replay through an isolated, real prefetch worker, with mocked WAL input. They verify delivered timestamps, the next read cursor, and the cumulative skipped-entry counter for the first local record, a preceding follower record, and a retained WAL beginning at local index 3.

Verification

  • Before the beginning fix, all three new tests delivered every expected row but failed on the skipped-entry counter: 0 versus 1 for the no-gap cases and 2 versus 3 for missing local indexes 1 and 2.
  • After the fix, the three new tests and four existing replay-boundary/gap-counter tests passed: 7 tests, 0 failures, 0 errors.
  • Standard Maven reactor execution in a separate worktree, including DataNode dependencies, with build cache disabled:
mvn -o test -pl iotdb-core/datanode -am \
  '-Dtest=ConsensusPrefetchingQueueSeekTest,ConsensusPrefetchingQueueTest#testReplayStartPreservesUncoveredFollowerEntries+testAtEndReplayLookupPreservesRequestedWriterFrontier+testAtEndInitializationHasNoTailGap+testWalReplayCountsOnlyUnavailableSearchIndexes' \
  -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false \
  -Dmaven.build.cache.enabled=false -Ddevelocity.off=true
  • Spotless formatting and git diff --check passed.

This PR has:

  • been self-reviewed.
  • added comments explaining the index boundaries.
  • added or updated unit tests for the changed paths.

Key changed/added classes in this PR
  • ConsensusPrefetchingQueue
  • ConsensusPrefetchingQueueTest
  • ConsensusPrefetchingQueueSeekTest

@Caideyipi Caideyipi changed the title Fix subscription AT_END replay cursor boundary Fix subscription WAL replay cursor boundaries Oct 9, 2026
@jt2594838
jt2594838 merged commit 83f6428 into apache:master Oct 10, 2026
36 of 40 checks passed
@jt2594838
jt2594838 deleted the fix/subscription-at-end-cursor branch October 10, 2026 02:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants