ORCA: fall back to planner for ORDER BY with ordering operators (amcanorderbyop)

[yjhjstz]

Summary

ORCA does not support amcanorderbyop (KNN ordered index scans). Queries like ORDER BY col <-> 'value' LIMIT N on GiST indexes cannot produce ordered index scans in ORCA, resulting in inefficient Seq Scan + Sort plans instead of KNN-GiST Index Scan.

Previously, these queries accidentally got correct plans because column-level COLLATE "C" caused a blanket fallback to the PostgreSQL planner, which does support amcanorderbyop. After commit 3f4ce85 added COLLATE "C" support to ORCA, these queries lost their fallback path and regressed to slow plans.

Changes

1. Add has_orderby_ordering_op() in walkers.c: Detects when a query's ORDER BY clause contains an operator registered as AMOP_ORDER in pg_amop (e.g., <-> for distance). When detected, ORCA falls back to the PostgreSQL planner which can generate KNN ordered index scans.

2. Refine fallback to skip lossy distance functions: Only fall back when ALL ordering-operator expressions have at least one direct Var (column reference) argument. Expressions like circle(p,1) <-> point(0,0) wrap the column in a function call, which causes "lossy distance functions are not supported in index-only scans" errors in the planner. Such queries are left for ORCA to handle via Seq Scan + Sort.

3. Update expected test outputs: btree_gist, pg_trgm, create_index, and gist test expected files updated to reflect the improved plans (Index Only Scan instead of Seq Scan + Sort).

Key design decisions

• The check is precise: only ORDER BY with ordering operators triggers fallback. Other queries on the same tables (e.g., WHERE with LIKE/%, equality filters) continue to use ORCA normally.
• The Var-argument check avoids a known planner limitation with lossy distance functions wrapped in expressions.

Test plan

• btree_gist regression tests pass with updated expected outputs
• pg_trgm regression tests pass with updated expected outputs
• create_index and gist regression tests pass with updated expected outputs
• make installcheck-world passes