[core] Introduce deletion vector meta cache at bucket level #6407
+812
−18
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JingsongLi
reviewed
Oct 16, 2025
paimon-api/src/main/java/org/apache/paimon/options/CatalogOptions.java
Outdated
Show resolved
Hide resolved
JingsongLi
reviewed
Oct 16, 2025
paimon-api/src/main/java/org/apache/paimon/options/CatalogOptions.java
Outdated
Show resolved
Hide resolved
JingsongLi
reviewed
Oct 16, 2025
paimon-api/src/main/java/org/apache/paimon/options/CatalogOptions.java
Outdated
Show resolved
Hide resolved
JingsongLi
reviewed
Oct 16, 2025
paimon-core/src/main/java/org/apache/paimon/index/IndexFileHandler.java
Outdated
Show resolved
Hide resolved
JingsongLi
reviewed
Oct 16, 2025
paimon-core/src/main/java/org/apache/paimon/index/IndexFileHandler.java
Outdated
Show resolved
Hide resolved
JingsongLi
reviewed
Oct 16, 2025
|
|
||
| @Nullable | ||
| // Construct DataFile -> DeletionFile based on IndexFileMeta | ||
| public Map<String, DeletionFile> extractDeletionFileByMeta( |
There was a problem hiding this comment.
Remove public and add @VisibleForTesting.
There was a problem hiding this comment.
remove public will lead to compilation problems
JingsongLi
reviewed
Oct 16, 2025
| DELETION_VECTORS_INDEX, | ||
| partitionBuckets.stream().map(Pair::getLeft).collect(Collectors.toSet())); | ||
| Map<Pair<BinaryRow, Integer>, Map<String, DeletionFile>> result = new HashMap<>(); | ||
| partitionBuckets.forEach( |
There was a problem hiding this comment.
Just use partitionFileMetas.forEach?
There was a problem hiding this comment.
I think unnecessary buckets need to be filtered out?
JingsongLi
reviewed
Oct 16, 2025
paimon-core/src/main/java/org/apache/paimon/index/IndexFileHandler.java
Outdated
Show resolved
Hide resolved
JingsongLi
reviewed
Oct 16, 2025
paimon-core/src/main/java/org/apache/paimon/index/IndexFileHandler.java
Outdated
Show resolved
Hide resolved
JingsongLi
reviewed
Oct 16, 2025
paimon-core/src/main/java/org/apache/paimon/index/IndexFileHandler.java
Outdated
Show resolved
Hide resolved
JingsongLi
reviewed
Oct 16, 2025
paimon-core/src/main/java/org/apache/paimon/index/IndexFileHandler.java
Outdated
Show resolved
Hide resolved
JingsongLi
reviewed
Oct 16, 2025
paimon-core/src/main/java/org/apache/paimon/utils/DVMetaCache.java
Outdated
Show resolved
Hide resolved
JingsongLi
reviewed
Oct 16, 2025
JingsongLi
reviewed
Oct 16, 2025
paimon-core/src/main/java/org/apache/paimon/utils/DVMetaCache.java
Outdated
Show resolved
Hide resolved
JingsongLi
reviewed
Oct 16, 2025
|
|
||
| public DVMetaCache(long maxElementSize) { | ||
| this.cache = | ||
| Caffeine.newBuilder().maximumSize(maxElementSize).executor(Runnable::run).build(); |
There was a problem hiding this comment.
Should it be set to max number of DVMetaCacheValue? And use softValues?
There was a problem hiding this comment.
I've added softValues. Do you mean to use List<DVMetaCacheValue> as the weight? I think the List might be empty, and those cache items won't be restricted because of the 0 weight
added 2 commits
October 22, 2025 21:04
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
In high-concurrency point query scenarios on the primary key table, we observed high CPU usage mainly caused by deserialization overhead of DV metadata. Currently, reading deletion vector metadata for a single bucket requires reading and deserializing a large number of entries from the index manifest, if the table has many partition and buckets.
This PR introduces a bucket-level dv meta cache which reduces CPU load and significantly improves QPS for single-bucket query scenarios on primary key tables.
Tests
API and Format
Documentation