Skip to content

feat(graph): materialize projected graphs as arrow CSR at PROJECT_GRAPH time - #800

Merged
adsharma merged 2 commits into
LadybugDB:mainfrom
zachwinter:feat/project-graph-arrow-csr
Aug 11, 2026
Merged

feat(graph): materialize projected graphs as arrow CSR at PROJECT_GRAPH time#800
adsharma merged 2 commits into
LadybugDB:mainfrom
zachwinter:feat/project-graph-arrow-csr

Conversation

@zachwinter

Copy link
Copy Markdown
Contributor

Item 1 from the GDS zero-copy discussion: PROJECT_GRAPH now materializes each projected rel table as arrow CSR and pins it on the graph entry, so GDS functions can wrap indptr/indices zero-copy instead of re-scanning storage per algorithm.

Mechanism. After entry validation, an internal connection runs MATCH (a:N)-[r:R]->(b:N) RETURN a.rowid, b.rowid through queryAsArrow — the #626 collector tracks CSR for that shape with no row materialization — and the ArrowQueryResult is stored on the ParsedNativeGraphEntry (session lifetime, freed on DROP).

Fallback-first. Materialization is skipped, and consumers fall back to scanning storage, whenever the CSR can't faithfully represent the projection yet:

  • multiple node tables (rowids would be ambiguous across dense spaces)
  • per-table predicates (filtered projection)
  • manual transactions (the internal read connection can't see the caller's uncommitted writes)
  • a rel whose result shape isn't CSR-tracked pins nullptr

Tests. New ProjectGraphCsrTest api suite: exact CSR content on a small graph (indptr [0,2,3,3], indices [1,2,2]), pinned-result survival across later statements, empty rel table, and all three skip guards. Full api suite 263 passed / 21 skipped (same as main, all pre-existing feature-gated skips); core gds~/graph~ e2e suites green.

Design question. This materializes eagerly on every qualifying PROJECT_GRAPH. The alternative is lazy-on-first-GDS-use. Eager matches your "PROJECT_GRAPH should materialize" framing and keeps the entry immutable after creation; the cost is paid by projections that never run a GDS function. Happy to switch if you prefer lazy.

Follow-ups. (1) Property columns (weights, timestamps) on the indices table per your point 3 — the query shape extends naturally. (2) The extensions-side shared helper consuming this + symmetrize() (#799) for undirected algorithms, replacing the InMemGraph copy in all three GDS functions.

🤖 Generated with Claude Code

…PH time

PROJECT_GRAPH now runs each projected rel table's scan through the arrow
CSR collector (queryAsArrow on an internal connection; the MATCH..RETURN
rowid shape tracks CSR with no row materialization, see LadybugDB#626) and pins the
resulting ArrowQueryResult on the graph entry. GDS consumers can wrap the
CSR indptr/indices zero-copy instead of re-scanning storage per algorithm.

Fallback-first: materialization is skipped — and consumers fall back to
scanning storage — whenever the CSR can't faithfully represent the
projection yet (multiple node tables, per-table predicates) or when the
internal read connection would not see the caller's writes (manual
transactions). A rel whose result shape isn't CSR-tracked pins nullptr.

Lifetime: the session's GraphEntrySet owns the pinned results; dropped
with the graph.

Tests: api suite ProjectGraphCsrTest — CSR content on a small graph,
survival across later statements, empty rel table, and all three skip
guards.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@adsharma

Copy link
Copy Markdown
Contributor

Let's get eager working first. We can revisit if there are cases where lazy is preferable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@adsharma
adsharma merged commit bb03772 into LadybugDB:main Aug 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants