fix(rust/sedona-spatial-join): prevent filter pushdown past KNN joins

[Kontinuation]

Summary

KNN joins have different semantics than regular spatial joins — pushing filters to the object (build) side changes which objects are the k nearest neighbors, producing incorrect results. DataFusion's builtin PushDownFilter optimizer rule doesn't know this and incorrectly pushes filters through KNN joins.

This PR adds a KnnJoinEarlyRewrite optimizer rule that converts KNN joins to SpatialJoinPlanNode extension nodes before DataFusion's PushDownFilter rule runs. Extension nodes naturally block filter pushdown via prevent_predicate_push_down_columns() returning all columns.

Changes

• New KnnJoinEarlyRewrite optimizer rule — handles two patterns:
  1. Join(filter=ST_KNN(...)) — when the ON clause has only the spatial predicate
  2. Filter(ST_KNN(...), Join(on=[...])) — when the ON clause also has equi-join conditions (DataFusion's SQL planner separates these)
• Positional rule insertion — MergeSpatialProjectionIntoJoin and KnnJoinEarlyRewrite are inserted before PushDownFilter, while SpatialJoinLogicalRewrite (for non-KNN joins) remains after so non-KNN joins still benefit from filter pushdown
• Updated SpatialJoinLogicalRewrite — skips KNN joins (already handled by the early rewrite)
• Integration tests verifying that object-side filters are NOT pushed down for KNN joins, but ARE pushed down for non-KNN spatial joins

Rule ordering

... → MergeSpatialProjectionIntoJoin → KnnJoinEarlyRewrite → PushDownFilter → ... → SpatialJoinLogicalRewrite

Follow-ups

1. We don't enforce ST_KNN to appear first in the chain of AND expressions. For instance, ST_KNN(L.geom, R.geom, 5) AND L.id = R.id has the same semantics as L.id = R.id AND ST_KNN(L.geom, R.geom, 5). This seems to be unnatural. Optimization rule does not seem to be a good place to enforce this, so we leave it to future patches that work on SQL parser and ASTs.
2. We don't allow any filter pushdown for ST_KNN for now. Actually filters applied to the query side could be pushed down without any problem, we need to implement such rules ourselves in future patches.

TODO

prevent_predicate_push_down_columns method seems to do the trick. I'll experiment with it. Hopefully we can implement query side filter pushdown easily.
UPDATE: No. It is a terrible idea. There's no shortcut. We have to implement the optimization rule ourselves.

Closes #605

[Kontinuation]

This is for working around a bug in DataFusion. I'll submit a patch later.

[Kontinuation]

Submitted apache/datafusion#20393

[Copilot]

Pull request overview

Adds an early logical optimizer rewrite to ensure KNN joins are converted into SpatialJoinPlanNode extension nodes before DataFusion’s PushDownFilter runs, preventing incorrect filter pushdown onto the KNN build side.

Changes:

• Insert MergeSpatialFilterIntoJoin + new KnnJoinEarlyRewrite before PushDownFilter, and run SpatialJoinLogicalRewrite after it for non-KNN joins.
• Remove is_spatial_predicate and update predicate-name collection tests accordingly.
• Add integration tests asserting object-side filter pushdown is blocked for KNN joins but allowed for non-KNN spatial joins; add necessary_children_exprs to the extension node.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
rust/sedona-spatial-join/src/planner/optimizer.rs	Adds KnnJoinEarlyRewrite, reorders optimizer rules around PushDownFilter, refactors join→extension rewrite helper.
rust/sedona-spatial-join/src/planner/logical_plan_node.rs	Adds necessary_children_exprs implementation for SpatialJoinPlanNode.
rust/sedona-spatial-join/src/planner/spatial_expr_utils.rs	Removes is_spatial_predicate and updates tests to validate collect_spatial_predicate_names.
rust/sedona-spatial-join/tests/spatial_join_integration.rs	Expands KNN filter correctness coverage and adds physical-plan assertions for filter pushdown behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[paleolimbot]

Thank you!