feat(graph): materialize projected graphs as arrow CSR at PROJECT_GRAPH time - #800
Merged
adsharma merged 2 commits intoAug 11, 2026
Merged
Conversation
…PH time PROJECT_GRAPH now runs each projected rel table's scan through the arrow CSR collector (queryAsArrow on an internal connection; the MATCH..RETURN rowid shape tracks CSR with no row materialization, see LadybugDB#626) and pins the resulting ArrowQueryResult on the graph entry. GDS consumers can wrap the CSR indptr/indices zero-copy instead of re-scanning storage per algorithm. Fallback-first: materialization is skipped — and consumers fall back to scanning storage — whenever the CSR can't faithfully represent the projection yet (multiple node tables, per-table predicates) or when the internal read connection would not see the caller's writes (manual transactions). A rel whose result shape isn't CSR-tracked pins nullptr. Lifetime: the session's GraphEntrySet owns the pinned results; dropped with the graph. Tests: api suite ProjectGraphCsrTest — CSR content on a small graph, survival across later statements, empty rel table, and all three skip guards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Let's get eager working first. We can revisit if there are cases where lazy is preferable. |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item 1 from the GDS zero-copy discussion:
PROJECT_GRAPHnow materializes each projected rel table as arrow CSR and pins it on the graph entry, so GDS functions can wrapindptr/indiceszero-copy instead of re-scanning storage per algorithm.Mechanism. After entry validation, an internal connection runs
MATCH (a:N)-[r:R]->(b:N) RETURN a.rowid, b.rowidthroughqueryAsArrow— the #626 collector tracks CSR for that shape with no row materialization — and theArrowQueryResultis stored on theParsedNativeGraphEntry(session lifetime, freed onDROP).Fallback-first. Materialization is skipped, and consumers fall back to scanning storage, whenever the CSR can't faithfully represent the projection yet:
nullptrTests. New
ProjectGraphCsrTestapi suite: exact CSR content on a small graph (indptr [0,2,3,3],indices [1,2,2]), pinned-result survival across later statements, empty rel table, and all three skip guards. Full api suite 263 passed / 21 skipped (same as main, all pre-existing feature-gated skips); coregds~/graph~e2e suites green.Design question. This materializes eagerly on every qualifying
PROJECT_GRAPH. The alternative is lazy-on-first-GDS-use. Eager matches your "PROJECT_GRAPH should materialize" framing and keeps the entry immutable after creation; the cost is paid by projections that never run a GDS function. Happy to switch if you prefer lazy.Follow-ups. (1) Property columns (weights, timestamps) on the indices table per your point 3 — the query shape extends naturally. (2) The extensions-side shared helper consuming this +
symmetrize()(#799) for undirected algorithms, replacing theInMemGraphcopy in all three GDS functions.🤖 Generated with Claude Code