Skip to content

feat!: relation chains through hasMany, semi-join filters, strict errors - #37

Merged
fratzinger merged 13 commits into
mainfrom
feat/relation-chains
Sep 2, 2026
Merged

fratzinger merged 13 commits into
mainfrom
feat/relation-chains

Conversation

@fratzinger

@fratzinger fratzinger commented Sep 1, 2026 •

Copy link
Copy Markdown
Owner

Relation paths may mix belongsTo and hasMany hops in any order and to any depth, every relation filter compiles to a correlated EXISTS (a semi-join), and sorting goes through an aggregate rather than a join that can duplicate rows.

await app.service('assignment-events').find({
  query: {
    assignment: {
      assignmentCategories: { $some: { typeId: { $in: ids } } },
    },
  },
})
select "assignment_events".* from "assignment_events"
where exists (
  select 1 from "assignments" as "assignment"
  where "assignment"."id" = "assignment_events"."assignmentId"
    and exists (
      select 1 from "assignment_categories" as "assignment__assignmentCategories"
      where "assignment__assignmentCategories"."assignmentId" = "assignment"."id"
        and "assignment__assignmentCategories"."typeId" in (?)
    )
)

A belongsTo inside $some / $none / $every resolves in the related service's own scope, so 'assignment.assignmentCategories.type.name' works too.

Breaking changes

Unresolvable filters and sorts are rejected

A filter the adapter could not resolve used to be dropped from the WHERE clause. That widens the result set, which is how an authorization filter built by a hook turns into a data leak — the same reasoning already applied to $or: []. It was also inconsistent: an unknown column raises a database error and an unknown operator a BadRequest, while an unknown relation path returned every row. Now rejected:

  • a collection operator on anything but a hasMany relation
  • a path that starts at a declared relation but breaks further along (typo, missing target service, or a chain that cannot be walked because app.setup() never ran)
  • a dot path inside a hasMany sub-filter that is neither a relation of the related service nor one of its own columns
  • the same for $sort, plus a hasMany reached through another relation (unsupported)

Still quiet, because neither can be proven wrong: a dot path whose first segment is not a declared relation (indistinguishable from a qualified column ref or JSON access), and an empty condition object, which is a no-op like $not: {}.

Relation filters no longer join

A belongsTo hop used to be a LEFT JOIN plus a null-protect predicate. That broke in three places, all from the same choice:

  • $not with a relation path emitted a ref to a table that was never joined — the JOIN pass descended into $and/$or but not $not. Even with the JOIN registered, the null-protect would have ended up inside the negation.
  • patch and remove failed the same way on postgres and sqlite: UPDATE/DELETE take no portable JOIN, so that branch never ran the JOIN pass. (The mysql branch resolves ids with a find first and was unaffected.)
  • A relation declared asArray: false on a non-unique column multiplied parent rows — two rows came back as four, with total reporting four.

A semi-join cannot duplicate rows, needs no null-protect, and stays correct under negation and in UPDATE/DELETE. $none semantics behind a belongsTo are unchanged (a missing parent excludes the row), now because the outer EXISTS is false.

Sorting goes through an aggregate

Filters stopped duplicating; sorting still did, because sorting needs the related value and not just its existence. asArray: false is intent, not a guarantee, so a to-one hop is now joined only when the adapter can prove it matches at most one row — keyThere is the target service's id, which is how belongsTo is declared in practice and needs no promise from the caller. Anything else, and every to-many hop, takes the ordering value from a GROUP BY derived table keyed on the join column: exactly one row per key by construction.

That makes it the mechanism for both sort paths, replacing the correlated aggregate the hasMany sort evaluated once per candidate row.

Performance

Measured with the benchmark below, same harness, only src/adapter.ts swapped. Noise floor established by running the same code twice: median 3.8%, max 8.4%.

Filters (JOIN → semi-join): 16 of 19 comparable cases inside the noise floor. Two moved:

case JOIN EXISTS
$or over two relation legs 1.774 ms 1.035 ms −41.6%
filter and sort on the same relation 0.319 ms 0.359 ms +12.6%

$or gets faster because the EXISTS form lets postgres decorrelate both legs into hashed subplans built once (540 → 56 buffer hits); JOIN + OR forces a nested loop re-evaluating the correlated subplan per row. The slower case reads assignments twice — once for the sort's join, once for the filter's subquery — where one join used to serve both.

Sorting (correlated aggregate → derived table):

case before after
$sort by hasMany column 221.8 ms 1.3 ms −99.4%

Buffer hits on postgres: 49,513 → 46. Every other case stayed inside the noise floor.

Regression guards

test/sql-shape.test.ts snapshots the compiled SQL and parameters for 20 relation cases. Kysely compiles without a database, so it runs in the normal suite with no schema and no data, pinned to sqlite regardless of DB — the subject is our own output, not per-dialect syntax.

bench/relations.bench.ts executes the same cases against a seeded database (~11k rows, fixed PRNG seed, secondary indexes off by default because that is where the SQL shape decides the plan):

pnpm bench --outputJson=bench/baseline.json   # reference commit
pnpm bench --compare=bench/baseline.json      # after the change

test/query-costs.test.ts reports blocks touched, rows returned and the plan spine per case from EXPLAIN (ANALYZE, BUFFERS). Deterministic where timings are not, and the row count doubles as a correctness signal. Postgres only, opt-in via BENCH_COSTS.

Details and traps in bench/README.md.

Deliberate choices

  • Dot notation implies $some at every hasMany hop. $none / $every stay nested-notation only — not expressible in a path.
  • JSON traversal does not apply inside $some / $none / $every: column types come from the queried service's own properties, so a dot path in a sub-filter is a relation path, and now raises a BadRequest instead of emitting a ref to the outer table.
  • A hasMany reached through another relation cannot be sorted by. Rejected explicitly rather than emitting broken SQL.

Testing

795 tests pass against postgres, 672 against sqlite, typecheck clean, lint warnings at baseline. New tests cover mixed chains in both notations, $none with an orphan parent, belongsTo inside $some, nested hasMany alias namespacing, three chained hasMany hops, $not over relation paths, patch/remove by relation filters including a self-referencing one, non-duplication for both filters and sorts on a non-unique to-one, the aggregate direction for a non-unique sort, and every rejection case.

🤖 Generated with Claude Code

Frederik Schmatz and others added 2 commits September 2, 2026 00:19
Four paths turned an unresolvable relation reference into a column ref,
producing SQL that errors at runtime instead of being skipped:

- `resolveRelationPath` accepted a terminal segment that names a relation,
  so `{ 'user.reports': ... }` became `"user"."reports"`.
- `buildPropertyExpression` fell through to an equality check when an
  operator-only object produced no condition, comparing the column against
  the raw object (`"user"."reports" = '{"$some":...}'`).
- A `$`-prefixed key reaching the column path was qualified as a column,
  so `{ user: { $some: ... } }` became `"user"."$some"`.
- Inside a hasMany EXISTS subquery the relation handlers resolved paths
  against the *outer* service's relations and FROM clause, so
  `{ reports: { $some: { 'manager.name': 'x' } } }` emitted a ref to a
  table the subquery never joined.

All four now drop the condition, matching the documented behaviour for
unresolvable relation paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A relation path may now mix belongsTo and hasMany hops in any order and to
any depth. Each belongsTo hop is a LEFT JOIN, each hasMany hop opens a
correlated EXISTS subquery, and the remainder of the path is resolved
inside that subquery against the related service's own relations:

  { assignment: { categories: { $some: { typeId: { $in: ids } } } } }
  { 'assignment.categories.type.name': 'urgent' }

Both previously produced no condition (or, before the preceding fix,
invalid SQL). Dot notation implies $some at every hasMany hop; $none and
$every stay nested-notation only.

Path resolution is now driven by an explicit scope (alias + relations of
the service owning it) instead of always reading this.options, so a
sub-filter resolves against the right service. Aliases are namespaced by
the scope they are built in, so a nested or self-referencing hasMany no
longer shadows the row source it correlates to, and the EXISTS correlation
is always qualified rather than relying on the column being declared in
`properties`.

resolveRelationPath/applyJoinsForWhere/flattenRelationQuery take a scope;
handleHasMany and handleBelongsTo are replaced by a single scope-aware
handleRelation. JSON traversal stays limited to the service's own row
source, since column types are read from its `properties`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pkg-pr-new

pkg-pr-new Bot commented Sep 1, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@fratzinger/feathers-kysely@37

commit: 2d45b5e

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 1, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
feathers-kysely 2d45b5e Commit Preview URL

Branch Preview URL
Sep 02 2026, 08:52 AM

A relation filter the adapter cannot resolve was dropped from the WHERE
clause. That widens the result set, which is how an authorization filter
built by a hook turns into a data leak — the same reasoning already
applied to `$or: []`. It was also inconsistent: an unknown column raises a
database error and an unknown operator a BadRequest, while an unknown
relation path returned every row.

Now rejected:

- a collection operator ($some/$none/$every) on anything but a hasMany
  relation
- a path that starts at a declared relation but breaks further along
  ('user.bogus.name', a missing or non-Kysely target service, or a chain
  that cannot be walked because `app.setup()` never ran)
- a dot path inside a hasMany sub-filter that is neither a relation of the
  related service nor one of its own columns

Still quiet, because neither can be proven wrong: a dot path whose first
segment is not a declared relation (indistinguishable from a qualified
column ref or JSON access — an unknown column there still surfaces as a
database error), and an empty condition object, which is a no-op like
`$not: {}`.

BREAKING CHANGE: queries that previously returned rows with a silently
dropped relation filter now fail with a BadRequest. Chains longer than one
hop require `app.setup()` to have run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fratzinger fratzinger changed the title feat: resolve relation chains through hasMany hops feat!: relation chains through hasMany hops, and strict errors for unresolvable filters Sep 1, 2026
Frederik Schmatz and others added 3 commits September 2, 2026 08:43
A belongsTo hop in a filter was a LEFT JOIN on the outer builder plus a
null-protect predicate. That broke in three places:

- `$not` with a relation path emitted a ref to a table that was never
  joined ("no such column: user.name"): the JOIN pass descended into
  `$and`/`$or` but not `$not`. Even with the JOIN registered, the
  null-protect would have ended up inside the negation.
- `patch` and `remove` failed the same way on postgres and sqlite:
  UPDATE/DELETE take no portable JOIN, so that branch never ran the JOIN
  pass at all. (The mysql branch resolves ids with a `find` first and was
  unaffected.)
- A relation declared `asArray: false` on a non-unique column multiplied
  parent rows: two users came back as four, with `total` reporting four.

All three come from the same choice, so all three are fixed by the same
change: a relation filter now compiles to a correlated EXISTS at every hop
and every depth. Further belongsTo hops in one chain become INNER JOINs
inside that subquery; a hasMany hop behind a belongsTo prefix nests inside
it. A semi-join cannot duplicate rows, needs no null-protect, and stays
correct under negation and in UPDATE/DELETE.

JOINs remain for `$sort`, which needs the related value rather than its
existence. `applyJoinsForWhere` is gone; `applyJoins` now only normalizes
the query shape and joins what `$sort` requires.

Behaviour is unchanged for every query that already worked — the semantics
of `$none` behind a belongsTo (a missing parent excludes the row) are the
same, now because the outer EXISTS is false rather than because of the
null-protect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A chain is walked through `app.service(name)`, which can fail three ways:
no app was ever set up, the target service is not registered, or it is not
a KyselyService. Each leaves the hop's relations unknown, so the chain is
unresolvable and must be rejected rather than silently dropped.

Also asserts that a single hop still works in all three cases — it only
needs the querying service's own relation definition.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two guards against regressions in query building, sharing one relation
graph and one table of query cases (`test/relation-graph.ts`): three
belongsTo hops, two hasMany levels and a to-many behind a to-one, covering
both notations, the collection operators, `$not`, `$or` and both sort
paths.

`test/sql-shape.test.ts` snapshots the compiled SQL and parameters for
every case. Kysely compiles without touching a database, so it runs in the
normal suite with no schema and no data, pinned to sqlite regardless of
`DB` — the subject is our own output, not per-dialect syntax. A change to
query building shows up as a readable diff instead of a timing wobble.

`bench/relations.bench.ts` executes the same cases against a seeded
database (~72k rows from a fixed PRNG seed, `BENCH_SCALE` to scale):

  pnpm bench --outputJson=bench/baseline.json   # reference commit
  pnpm bench --compare=bench/baseline.json      # after the change

Every column a case filters or sorts on is indexed — without that, a
chained filter degrades to a sequential scan inside a correlated subquery
and the timing measures table size instead of the SQL.

Two traps are documented in `bench/README.md` because both surface as
`NaN` rather than as an error: vitest skips suite hooks in benchmark mode
(so setup happens at module scope), and the default 500ms budget can
complete zero samples when every iteration is a round-trip.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fratzinger fratzinger changed the title feat!: relation chains through hasMany hops, and strict errors for unresolvable filters feat!: relation chains through hasMany, semi-join filters, strict errors Sep 2, 2026
Frederik Schmatz and others added 7 commits September 2, 2026 09:03
Including the quoting trap: an empty unquoted value makes loadEnvFile take
the next line as the value, so the following variable is silently lost and
the run connects to whatever the fallback in test/dialect.ts names.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Indexing every filtered column made the benchmark measure the best case and
assume a schema few users have. An un-indexed table is exactly where the
shape of the SQL decides the plan, so a regression a covering index would
hide is still a regression in practice.

Secondary indexes are now behind `BENCH_INDEXES=1`; primary keys remain, so
a FK referencing one still gets an index lookup on the target side. The
default row counts drop to ~11k, sized so that every case stays measurable
un-indexed: 19 of the 20 land between 0.15ms and 1.4ms with rme under 1%,
which reads more stable than the indexed run did, and the ordering now
tracks hop depth instead of filter selectivity.

`$sort by hasMany column` is the exception at ~220ms — a correlated
aggregate subquery per row is quadratic without an index. It stays in the
table because it is worth watching; `bench/README.md` notes that it
completes only a dozen iterations and that scaling up grows it
quadratically while the rest grow linearly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sqlite cannot bind a JS boolean and postgres will not take 0/1 for a
boolean column, so the benchmark seed failed outright on `DB=sqlite`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`asArray: false` is the caller's intent, not a guarantee — nothing stops a
relation from pointing at a non-unique column. Sorting by such a hop joined
it plainly, so three users came back as five rows with `total` reporting
five. Filters stopped duplicating when they became semi-joins; sorting
still did, because sorting needs the related value and not just its
existence.

A to-one hop is now joined only when the adapter can prove it matches at
most one row: `keyThere` is the target service's `id`, which is how
belongsTo is declared in practice and needs no promise from the caller.
Anything else — and every to-many hop — takes the ordering value from a
`GROUP BY` derived table keyed on the join column, which yields exactly one
row per key by construction. The aggregate follows the sort direction, MIN
ascending and MAX descending, the rule hasMany sorting already used.

That makes it the mechanism for both sort paths, replacing the correlated
aggregate the hasMany sort evaluated once per candidate row. On the
benchmark's un-indexed data, `$sort` by a hasMany column goes from 221.8ms
to 1.3ms (~165x); on postgres its buffer hits drop from 49,513 to 46. Every
other case stays inside the measurement noise.

`$sort` paths that start at a declared relation and do not resolve now
raise a BadRequest instead of emitting SQL that errors: a broken chain, a
path not ending on a column, or a hasMany reached through another relation
(still unsupported).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wall-clock timings in the benchmark drift a few percent between runs — the
same code twice varies by up to 8% — which drowns out most of what is worth
catching. Blocks touched and rows returned do not: they are a property of
the plan and the data, so a 2% change in them is a real change.

`test/query-costs.test.ts` compiles every case, runs it through
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` and reports blocks, rows and the
plan spine, optionally diffed against a recorded baseline. Postgres only
(`BUFFERS` is postgres syntax) and skipped unless `BENCH_COSTS` is set, so
`pnpm test` is unaffected.

The `rows` column earned itself immediately: `nested hasMany two levels`
was matching zero rows and therefore measuring nothing, because it filtered
on one specific title in randomly seeded data. It now filters on a range.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two sort keys aggregating the same relation
(`{ 'todos.text': 1, 'todos.assigneeId': -1 }`) joined two derived tables
under one alias and the query failed as ambiguous. A regression from moving
sorting off correlated subqueries: those carried no outer alias, so two
keys on one relation used to work.

The alias is now keyed on the sort key's position among the keys that need
a derived table, not on the relation.

Also covers three capabilities the rewrite added but nothing exercised: a
chain whose first or middle hop is not provably unique (aggregated as one
derived table with the remaining hops joined inside it), sorting by a
belongsTo column of a hasMany relation, and a hasMany sort `filter` with
operators or relation paths — it goes through the normal query builder now,
where it used to be equality-only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…skill

- `test/dialect.ts` reads `POSTGRES_HOST` instead of hardcoding localhost,
  so the postgres run can point somewhere else.
- The README keeps the `.env` example but drops the parser explanation; the
  quoted empty password in the example carries the point.
- `.claude/skills/other-orms` records the prior-art candidates to consult
  before naming an operator or settling a filter semantic, and the
  constraints that outrank them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fratzinger
fratzinger merged commit ab66dc8 into main Sep 2, 2026
36 checks passed
@fratzinger
fratzinger deleted the feat/relation-chains branch September 2, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant