Skip to content

Optimize CTE-heavy lineage by sharing scopes via Rc - #366

Merged
tobilg merged 2 commits into
tobilg:mainfrom
ashambalev:perf-lineage-rc
Jul 17, 2026
Merged

Optimize CTE-heavy lineage by sharing scopes via Rc#366
tobilg merged 2 commits into
tobilg:mainfrom
ashambalev:perf-lineage-rc

Conversation

@ashambalev

@ashambalev ashambalev commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

I've run into performance issue in my project - parsing lineage in a complex dbt project with many CTEs was taking too much time (relatively, still faster than dbt itself). So checked lineage.rs and found the culprit:

to_node_inner deep-cloned every ancestor CTE scope (full expression trees) at each recursion level, and resolve_qualified_column re-cloned the entire set again for every column reference resolved through a CTE.

This PR switches the lineage recursion plumbing to Rc<Scope>.
Also self-exclusion filter now has a cheap pre-check by name, while still falling back to expression equality.

I've run a benchmark (on M5 Pro):

CTEs main PR speedup
4 281 µs 135 µs 2.1×
8 1.78 ms 410 µs 4.4×
16 17.8 ms 1.63 ms 10.9×
24 67.1 ms 4.74 ms 14.1×

@tobilg
tobilg merged commit 4bb4e32 into tobilg:main Jul 17, 2026
@tobilg

tobilg commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Thanks a lot for the PR, merged! THis will be included in the upcoming version 0.6.2.

@tobilg

tobilg commented Jul 17, 2026

Copy link
Copy Markdown
Owner

I added some smallish improvements on top of it as GPT-5.6 xhigh indicated… Performance should be improved by 20-70% on a per-case basis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants