[DavidMStraub]

Follow-up to #1107: list endpoints sort and page in SQL when the sort keys are secondary columns, instead of reading the raw data of the whole table.

Implementation

• Sort keys in SQL: change, private and gramps_id, in any combination and direction. Everything else still uses the existing Python paths.
  • Ties are broken by handle, with COLLATE "C" on PostgreSQL, so the order equals that of sort_handles_by_raw_data.
  • gramps_id uses the locale's collation, as created by check_collation() (moved from object_query.py to db_backend.resolve_collation).
• Paging in SQL: LIMIT/OFFSET and a COUNT(*) for X-Total-Count, both scoped to the tree on SharedPostgreSQL.
• Guests: RestrictedDb.get_sorted_handles adds its policy's condition (private = 0) to the query. Citations and notes, whose visibility also depends on other records, are sorted in SQL but paged after filtering in Python.
• Default order: lists without sort are sorted by Gramps ID since Replace the private proxy with RestrictedDb #1107, so they now take the SQL path too.
• Fallbacks: if a backend has no secondary columns, or a sort key isn't a column, the existing path is used unchanged.

Tests

tests/test_sort_sql.py checks for every class, sort key combination and direction that the SQL order equals the raw-data order, that pages match the slices of the full order, and that the SQL path is actually used. The equivalence was also checked by hand on PostgreSQL.

Performance

One page of a list on a synthetic tree with 100k people (~485k objects), before → after:

Setup	People, sort=-change (frontend default)
SQLite, owner	770 → 190 ms
SQLite, guest	1.2 s → 250 ms
PostgreSQL (shared), owner	950 → 200 ms
PostgreSQL (shared), guest	1.3 s → 310 ms

Across the same four setups, events with sort=-change went from about 0.9–1.4 s to 180–200 ms, and lists sorted by Gramps ID from about 0.8–1.4 s to 190–380 ms.

[DavidMStraub]

CC @dsblank

[dsblank]

I tested this against a SharedPostgreSQL tree with about 100k people (PostgreSQL 16; 1.1k–331k rows per class). For 7 classes and the sorts -change, gramps_id, -gramps_id and -private,change, as both owner and guest, the SQL order was identical to the sort_handles_by_raw_data order. Pages at offsets 0, 20, near the end and past the end matched too, with the same totals. Guest citations and notes, which are filtered before paging, matched as well. The Gramps ID sorts were only compared with the en locale (en_US libc collation). Sorting a whole class took 0.07–0.35 s in SQL, against 0.9–3.9 s for the raw-data path. The new tests plus tests/test_access.py pass locally (154 passed).

One problem is confirmed (inline below), plus two smaller points. On tests: nothing covers a collation that can't be created or the list endpoint itself (pages and X-Total-Count through base.py), and test_uses_sql checks only Person with -change.