perfect hash join #19411
Open
perfect hash join #19411
+1,936
−485
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
common
Dandandan
reviewed
Dec 19, 2025
| ) -> Result<Self> { | ||
| // Initialize with 0 (sentinel for not found) | ||
| let mut data: Vec<u32> = vec![0; range]; | ||
| let mut next: Option<Vec<u32>> = None; |
Contributor
There was a problem hiding this comment.
Suggested change
| let mut next: Option<Vec<u32>> = None; | |
| let mut next: Vec<u32> = vec![]; |
I think this should work as well
Contributor
Author
There was a problem hiding this comment.
Done, it's cleaner this way
Contributor
|
run benchmarks |
|
🤖 |
Contributor
|
run benchmark tpcds |
Dandandan
reviewed
Dec 19, 2025
| /// | ||
| /// TODO: Currently only supports cases where left_side.num_rows() < u32::MAX. | ||
| /// Support for left_side.num_rows() >= u32::MAX will be added in the future. | ||
| pub perfect_hash_join_min_key_density: f64, default = 0.99 |
Contributor
There was a problem hiding this comment.
This seems very high? For a hashmap I believe it's ~75% default (plus it has some more overhead per key), so I think a 75% probably could still be better overall?
Contributor
Author
There was a problem hiding this comment.
That's a great point.
I'll add some benchmarks to compare the performance at different densities, including 75%, to find the optimal value for our use case. I'll update this based on the results.
Thanks for the suggestion!
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Contributor
Author
There was a problem hiding this comment.
I think we can set it to 20%.
- In terms of memory usage (run with this code: ),
ArrayMapconsumes less memory thanJoinHashMapat a 20% density, even with duplicate keys. - Based on the hj.rs benchmark(results in the PR description),
ArrayMapis also faster thanJoinHashMapat 20% density, regardless of whether there are duplicate keys.
Memory Comparison Matrix (num_rows = 1000000)
| Density | Dup Rate | ArrayMap (MB) | JoinHashMap (MB) | Ratio (AM/JHM) |
|---------|----------|---------------|------------------|----------------|
| 100% | 0% | 3.81 | 37.81 | 0.10x |
| 100% | 25% | 7.63 | 37.81 | 0.20x |
| 100% | 50% | 7.63 | 37.81 | 0.20x |
| 100% | 75% | 7.63 | 37.81 | 0.20x |
| 75% | 0% | 5.09 | 37.81 | 0.13x |
| 75% | 25% | 8.90 | 37.81 | 0.24x |
| 75% | 50% | 8.90 | 37.81 | 0.24x |
| 75% | 75% | 8.90 | 37.81 | 0.24x |
| 50% | 0% | 7.63 | 37.81 | 0.20x |
| 50% | 25% | 11.44 | 37.81 | 0.30x |
| 50% | 50% | 11.44 | 37.81 | 0.30x |
| 50% | 75% | 11.44 | 37.81 | 0.30x |
| 20% | 0% | 19.07 | 37.81 | 0.50x |
| 20% | 25% | 22.89 | 37.81 | 0.61x |
| 20% | 50% | 22.89 | 37.81 | 0.61x |
| 20% | 75% | 22.89 | 37.81 | 0.61x |
| 10% | 0% | 38.15 | 37.81 | 1.01x |
| 10% | 25% | 41.96 | 37.81 | 1.11x |
| 10% | 50% | 41.96 | 37.81 | 1.11x |
| 10% | 75% | 41.96 | 37.81 | 1.11x |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
UBarney
commented
Dec 21, 2025
| comparison = BenchmarkRun.load_from_file(comparison_path) | ||
|
|
||
| console = Console() | ||
| console = Console(width=200) |
Contributor
Author
There was a problem hiding this comment.
I've increased the console width to 200. I added more information like 'density' to the queryName, which made it longer and caused it to be cut off in the output before
| baseline_header = baseline_path.parent.stem | ||
| comparison_header = comparison_path.parent.stem | ||
| baseline_header = baseline_path.parent.name | ||
| comparison_header = comparison_path.parent.name |
Contributor
Author
There was a problem hiding this comment.
Before, a path like .../density=0.1/... was incorrectly shortened to density=0. Now, by using .parent.name, we correctly get the full directory name, density=0.1
github-actions
bot
added
proto
UBarney
force-pushed
the
perfect_hj
branch
4 times, most recently
from
8044bce to
954f1df
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
This PR introduces a Perfect Hash Join optimization by using an array-based direct mapping(
ArrayMap) instead of a HashMap.The array-based approach outperforms the standard Hash Join when the build-side keys are dense (i.e., the ratio of
count / (max - min+1)is high) or when the key range(max - min)is sufficiently small.The following results from the hj.rs benchmark suite. The benchmark was executed with the optimization enabled by setting
DATAFUSION_EXECUTION_PERFECT_HASH_JOIN_MIN_KEY_DENSITY=0.1The following results from tpch-sf10
What changes are included in this PR?
collect_left_input(build) phase, we now conditionally use anArrayMapinstead of a standardJoinHashMapType. This optimization is triggered only when the following conditions are met:u32::MAXArrayMapworks by storing the minimum key as an offset and using a Vec to directly map a keykto its build-side index viadata[k- offset].HashMap performance across varying key densities and probe hit rates
Are these changes tested?
Yes
Are there any user-facing changes?
Yes, this PR introduces two new session configuration parameters to control the behavior of the Perfect Hash Join optimization:
perfect_hash_join_small_build_threshold: This parameter defines the maximum key range (max_key - min_key) for the build side to be considered "small." If the key range is below this threshold, the array-based join will be triggered regardless of key density.perfect_hash_join_min_key_density: This parameter sets the minimum density (row_count / key_range) required to enable the perfect hash join optimization for larger key ranges