chore(rust/sedona-spatial-join): use RTree partitioner for probe side when partition count > 48#598
Conversation
…on count > 48 The probe side partitioner was always using FlatPartitioner (linear scan, O(n) per query). Benchmarks show that RTreePartitioner becomes faster at ~36 partitions, with the advantage growing significantly beyond that (e.g. 2.87x at 256 partitions, 3.29x at 400). Use a threshold of 48 partitions to switch from Flat to RTree, providing a comfortable margin above the measured crossover point. Also adds a flat_vs_rtree benchmark for head-to-head comparison across partition counts (16-400).
There was a problem hiding this comment.
Pull request overview
This PR optimizes the spatial join probe-side partitioning by switching from a linear-scan (FlatPartitioner) to an RTree-based partitioner when the partition count exceeds 48, based on benchmark evidence showing RTree outperforms linear scan above ~36 partitions.
Changes:
- Introduce a partition count threshold (48) to select between flat and RTree partitioners for probe-side partitioning
- Add benchmark comparing FlatPartitioner vs RTreePartitioner across various partition counts (16–400)
- Remove unused
build_index.rsfile
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| rust/sedona-spatial-join/src/prepare.rs | Adds threshold constant and conditional logic to switch between FlatPartitioner and RTreePartitioner based on partition count |
| rust/sedona-spatial-join/src/build_index.rs | Removes entire file (appears to be dead code) |
| rust/sedona-spatial-join/bench/partitioning/flat_vs_rtree.rs | Adds new benchmark comparing FlatPartitioner vs RTreePartitioner performance across different partition counts |
| rust/sedona-spatial-join/Cargo.toml | Registers the new flat_vs_rtree benchmark |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Kontinuation
changed the title
| //! | ||
| //! This module is included (`mod common;`) by multiple independent bench binaries, | ||
| //! not all of which use every item. Suppress dead-code warnings accordingly. | ||
| #![allow(dead_code)] |
There was a problem hiding this comment.
I would love to avoid a module-level clippy allow (so we can know that this needs to be removed if there's a time when all the benchmarks stop using it). Maybe you can make this a pub export of a regular module in src called internal_benchmark_util?
There was a problem hiding this comment.
Moved to src/utils. Also switched from rand to fastrand to avoid introducing new dependency.
…rk_util Replace the bench/partitioning/common.rs module (which needed a module-level #![allow(dead_code)]) with a proper library module at src/utils/internal_benchmark_util.rs. Switch from rand to fastrand (already a regular dependency) so no feature gate is needed.
Summary
FlatPartitioner(linear scan) to usingRTreePartitionerwhen the number of partitions exceeds 48.flat_vs_rtreebenchmark for head-to-head comparison across partition counts (16–400).Benchmark Results
The crossover is at ~36 partitions. A threshold of 48 provides a comfortable margin.