feat: use independent region manifest for MemWAL #5689
+2,190
−2,986
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thanks for pushing progress on this, I will take a look tomorrow morning! |
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
jackye1995
reviewed
Jan 13, 2026
- Mark as experimental - Restructure sections (architecture first, then components) - Use bloom filter instead of btree for PK existence - Fix vector index: use same IVF type/centroids/codebook as base - Add region pruning, sync/async ops summary - Update query plans for multi-region (no cross-region dedup) - Rename _memwal to _mem_wal - Remove redundant sections Co-Authored-By: Claude Opus 4.5 <[email protected]>
touch-of-grey
force-pushed
the
MemTable-LSM
branch
from
535ed44 to
5a7d379
Compare
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
- Replace RegionId struct with Uuid type alias for consistency - Rename snapshot_timestamp to snapshot_ts_millis per spec - Add conflict resolution between UpdateMemWalState and CreateIndex of MemWalIndex - Implement generation comparison: committed >= to_commit is non-retryable, committed < to_commit is retryable - Add rebase logic to merge merged_generations when CreateIndex rebases against UpdateMemWalState - Add comprehensive unit and integration tests for conflict scenarios Co-Authored-By: Jack Ye <[email protected]> Co-Authored-By: Claude <[email protected]>
- Propagate load_mem_wal_index_details errors instead of silently returning Ok - Consume conflicting_mem_wal_merged_gens directly to avoid clone Co-Authored-By: Jack Ye <[email protected]> Co-Authored-By: Claude <[email protected]>
jackye1995
approved these changes
Jan 13, 2026
Contributor
jackye1995
left a comment
jackye1995
left a comment
There was a problem hiding this comment.
thanks for working with me for all the comments! Looks good to me now, pending CI
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on discussion with @jackye1995 past few days, update the current MemWAL structure to introduce region specific manifest for independent update so that it can more efficiently run updates in streaming engines like Flink. This mostly comes from draft from @jackye1995, but separate out the MemWAL index part first with some my custom edits.