Skip to content

Query endpoint: evaluate guest queries on RestrictedDb raw data, and fix select value types on SQL backends - #1113

Merged
DavidMStraub merged 3 commits into
gramps-project:masterfrom
dsblank:select-schema-types
Oct 27, 2026
Merged

DavidMStraub merged 3 commits into
gramps-project:masterfrom
dsblank:select-schema-types

Conversation

@dsblank

@dsblank dsblank commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

This needs gramps-object-query-language 0.5.7, now on PyPI (release notes).

Summary

Guests: GOQL reads RestrictedDb as raw data, and counts in the same pass (proxied path)

These are the suggestions from dsblank/gramps-object-query-language#7:

  • Raw data. RestrictedDb is read with RawDataReader, so no Gramps object is built per record. Gramps proxies (ProxyDbBase) keep the default object reader, since they build the object anyway.
  • One pass for the count. X-Total-Count comes from run_query_with_total, which counts matches during the same pass that sorts them. Before, a counted request whose matches spanned more than one page read and cleaned every record a second time.

On a 100k-person PostgreSQL tree, #7 measured guest queries 1.3–13× faster with RawDataReader, before counting the second pass removed here.

Bug fix: selected values have their schema types (SQL path)

On the SQL path, /query/ returned some selected values in the database's own form rather than their Gramps type:

Selected value Before (SQLite) Before (PostgreSQL) After (both)
flat boolean (private, birth.private) 0 0 false
JSON boolean (primary_name.private) 0 "true" true
JSON number (birth.date.sortval) 2389036 "2389036" 2389036
object or list (birth.date) parsed parsed parsed
string (primary_name.first_name) "123" "123" "123"

The proxied path (guests) already returned these as true/false and numbers. Owners and guests now get the same values. "false" as a string counts as true in JS and Python, so a client testing if (item.private) on PostgreSQL got the wrong answer.

_post_sql now runs each row through GOQL's select_normalizer. That replaces _needs_json_decoding / _normalize_json_value, which only decoded objects and lists. Their docstring said PostgreSQL values "arrive parsed", but scalars don't.

Client-visible change: on SQL backends private is now true/false instead of 1/0, and on PostgreSQL JSON numbers and booleans are no longer strings. gramps-connect's list views already convert place_type with Number() and select no boolean columns, so they're unaffected.

Changes

  • resources/object_query.py: select_normalizer in _post_sql. RawDataReader and run_query_with_total in _post_proxied. The two decoding helpers are removed.
  • pyproject.toml: gramps-object-query-language>=0.5.7,<0.6.
  • Tests:
    • New test_select_values_have_schema_types: checks value types for owner (SQL) and guest (proxied), and that a public person reads the same either way.
    • The proxied count tests now expect one pass whether or not the result is paginated.
    • New test_guest_cannot_see_private_related_records: a guest can't match, select or count a private place or event through a person (birth.place.title, birth.description, any(...), len(...), count(...)), while the owner can. It fails if related records are read past RestrictedDb.
    • New test_restricted_db_is_read_as_raw_data: a guest's query gets RawDataReader, a Gramps proxy the default reader.
    • The unit tests for the removed helpers are gone; the same cases are covered in GOQL by Endpoints: citation, citations #8.

Test plan

  • test_object_query_parsing.py, test_access.py, test_privacy_canary.py, and in test_endpoints/: test_people_query.py, test_object_query.py, test_object_query_where_tree.py, test_filters.py: 276 passed against gramps-object-query-language 0.5.7
  • The new type test fails without the change (private came back as 0)
  • The related-record privacy test fails when related reads bypass RestrictedDb, and the reader test fails when guests get no reader
  • CI with gramps-object-query-language 0.5.7 from PyPI

isort and mypy report errors on unchanged lines that are already on master.

🤖 Generated with Claude Code

dsblank and others added 2 commits October 26, 2026 09:31
…atches in one pass

SQL path: rows go through GOQL's `select_normalizer`, so each selected
value has the type Gramps' schema gives it, as on the proxied path.
Booleans were 0/1 (and "true"/"false" text for JSON booleans on
PostgreSQL), and JSON numbers were strings on PostgreSQL. This replaces
`_needs_json_decoding` / `_normalize_json_value`, which only decoded
objects and lists.

Proxied path: `RestrictedDb` is read as raw data (`RawDataReader`), and
`X-Total-Count` comes from `run_query_with_total`'s single pass instead
of a second, unlimited `run_query` whenever the matches span more than
one page.

Needs gramps-object-query-language 0.5.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dsblank dsblank changed the title Query endpoints: typed select values, and one pass for guests' counts Query endpoint: evaluate guest queries on RestrictedDb raw data, and fix select value types on SQL backends Oct 26, 2026
A guest can't match, select or count a private place or event through
a person. Reading related records past RestrictedDb makes the test fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dsblank
dsblank marked this pull request as ready for review October 26, 2026 18:51
@DavidMStraub

Copy link
Copy Markdown
Member

Looks good!

@DavidMStraub
DavidMStraub merged commit a5eef18 into gramps-project:master Oct 27, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants