[fix][broker] Fix readCompacted skipping tombstones within compaction horizon

[grishaf]

Summary

readCompacted=true readers could miss delete tombstones when a second compaction ran before those tombstones were consumed. CompactedTopicUtils jumped the cursor to horizon + 1 whenever the compacted ledger returned no entry, skipping unread messages in the original topic between the current position and the compaction horizon.

The fix falls back to the original managed ledger at the current cursor instead of seeking past the horizon.

Issue

readCompacted uses a hybrid read path:

• positions ≤ compaction horizon → compacted BookKeeper ledger
• positions > horizon → original managed ledger

When findStartPoint returns NEWER_THAN_COMPACTED (e.g. a tombstone exists in the original topic but not in the compacted ledger), readCompactedEntries returns empty. The broker then seeked to horizon + 1, skipping tombstones still sitting in the original ledger between the reader cursor and the horizon.

Race

1. Reader opens with readCompacted=true and consumes the compacted snapshot.
2. Producer sends tombstones (key-only deletes).
3. Compaction runs again before the continuous reader consumes those tombstones.
4. Reader asks for the next message; compacted ledger has no entry at the current position.
5. Bug: broker seeks to horizon + 1 → tombstones are never delivered.

Impact on TableView

TableView builds its map from a continuous readCompacted reader. If tombstones are skipped:

• deleted keys stay in the in-memory map
• onUpdate/onRemove callbacks for those deletes never fire
• consumers see stale entries that should have been removed

Any client using a long-lived readCompacted reader or TableView (Java, Go, etc.) inherits this broker behavior; no client-side change is required once the broker stops skipping.

Fix

In CompactedTopicUtils, when the compacted read returns empty, read from the original managed ledger at the current cursor (readEntries / readEntriesWithSkipOrWait) instead of seeking to horizon + 1.

Made with Cursor

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.