[dsblank]

The evaluator path (run_query, evaluate_where) can now read records as raw data instead of building a Gramps object for each one, and run_query_with_total returns the number of matches from the same pass. For a guest in gramps-web-api with RestrictedDb (gramps-project/gramps-web-api#1107), this makes /query 1.3–13× faster on a 100k-person tree. The default behavior, and so the Gramps addons, is unchanged in speed.

Changes

• Records in either form. evaluate_where and resolve_column_ref work on a Gramps object or its raw data, and never convert wholesale. Related records are read in the same form through a RecordReader:

  • ObjectReader (the default) reads through db's object getters, as before, so Gramps proxies and CacheProxyDb behave the same.
  • RawDataReader (opt-in) reads through get_raw_<table>_data and the table cursor, and never builds an object per record.

  Both use only db's public methods, so a proxy's __getattr__ never forwards a read to the unfiltered database. Related records are cached per query in an LRU cache of 10,000 entries.

• run_query no longer goes through GenericFilter. It takes a reader= argument, and turns only the final page into objects when no select is given.

• run_query_with_total returns (rows, total). The total ignores after and limit, like compile_count_query. run_query keeps its return type.

• Behavior change, matching the SQL path: a related record that doesn't read back (hidden or missing) is never counted in len(...) or any(...). Before, any(...) and len(...) without a condition still counted a related record the database hides (its getter returns None).

What gramps-web-api can change

Once this is released and #1107 is merged, _post_proxied in resources/object_query.py can use both:

from gramps_object_query_language.evaluator import RawDataReader
from gramps_object_query_language.proxied_query import run_query_with_total

reader = RawDataReader(db)
rows, total = run_query_with_total(
    db, self.spec, where,
    order_by=order_by, limit=args["limit"] + 1, after=after,
    select=fetch_refs, reader=reader,
)

• The total from the same pass. X-Total-Count then comes from total, and the second, unlimited run_query goes away. Today that second call reads and cleans every record again whenever the matches span more than one page.
• Raw data instead of objects. RawDataReader suits RestrictedDb, which cleans raw data directly. On a Gramps proxy it would be slower, because the proxy builds the object anyway.
• Version bound. The gramps-object-query-language bound needs to move to the release that has this.

Performance

Guests on a 100k-person tree (shared PostgreSQL; about 91,000 of its people are visible to a guest). Before is GOQL 0.5.6's run_query through RestrictedDb, which is what gramps-web-api does with #1107. After is this branch with RawDataReader. Median of 3 runs, before and after alternating, gc.collect() before each. Every run checked that both return the same records.

Query	Before	After	Speedup	Matches
gender == Person.FEMALE	28.1 s	6.7 s	4.2×	42,818
primary_name.first_name == 'Mary'	27.8 s	4.8 s	5.8×	19
birth.date.sortval >= Date('Jan 1, 1950')	33.1 s	12.2 s	2.7×	13,979
like(birth.place.title, '%Texas%')	36.1 s	9.9 s	3.6×	0
Family any(children, given_name == 'Steve')	22.4 s	17.3 s	1.3×	0
Event like(place.title, '%Texas%')	53.9 s	4.1 s	13.2×	0

These times are for one full pass. Today a counted request spanning more than one page pays for a second full pass; with run_query_with_total it pays for one.

• Most of what's left is reading and cleaning every visible record.
• children gains the least. My guess, not profiled: RestrictedDb re-reads each parent family when it cleans a person fetched by handle (noted on #1107).

Gramps' example tree, default reader, median ms of 7 runs. The RawDataReader column is new and has no "before". This run overlapped the 100k run; old and new alternated, so both sides saw the same load.

Query	plain old → new	PrivateProxyDb old → new	CacheProxyDb(PrivateProxyDb) old → new	RestrictedDb old → new	RestrictedDb + RawDataReader
flat	65 → 58	383 → 373	392 → 390	130 → 121	54
json	113 → 107	442 → 448	462 → 400	155 → 151	43
hop	133 → 131	460 → 460	643 → 639	234 → 233	69
hop2	143 → 135	479 → 490	565 → 563	214 → 202	76
children	61 → 67	363 → 376	376 → 379	110 → 116	65
backlinks	30 → 27	106 → 104	112 → 109	103 → 100	83
events	83 → 61	198 → 166	183 → 176	113 → 84	40

The addons' path is a Gramps filter rule calling evaluate_where per record, as GOQLFilter's whereexprrule.py does. It's within about 2% on every query, under both PrivateProxyDb and CacheProxyDb(PrivateProxyDb), with the same matches.

The largest slowdown is children on a plain database, 61 → 67 ms. Holding the cached related records makes Python's garbage collection a little more expensive on queries where those records rarely repeat.

About the cache:

• A cache that empties itself when full collapsed on the 100k tree once a query needed more places than fit: Event like(place.title, …) took 13.6 s with 1,000 entries against 4.0 s with 10,000.
• LRU at 10,000 was as fast or faster than that cache on every 100k query I ran it on (events 4.2 → 4.1 s, hop2 12.1 → 10.4 s, children 19.8 → 16.9 s).

Tests

• GOQL's suite: 790 pass. The new tests cover:
  • the total ignoring limit and after
  • private parts of visible records matched as cleaned, with both readers: a private name, a private birth event, and a negated condition
  • sorting by the cleaned value: a private name sorts as "Private"
  • unreadable related records not counted, checked against the SQL path
  • evaluate_where giving the same answer on raw data as on objects
  • LRU eviction order
• Old-vs-new parity: every example query in README-query-language.md and docs/where_expr.md (82 queries), sorted and unsorted, on a plain database, PrivateProxyDb and RestrictedDb, with about 10% of records and their parts made private. Both readers were checked. 492 comparisons, 0 differences.

The benchmark and parity scripts aren't in this PR, because they depend on local trees and paths. I can add cleaned-up versions if they're wanted.

🤖 Generated with Claude Code

[dsblank]

CC: @DavidMStraub

[DavidMStraub]

Nice!