feat: column lineage API, SQL tags, and PyO3 bindings - #1
Open
suryaiyer95 wants to merge 1 commit into
Open
Conversation
Column Lineage (`lineage.rs`): - Public `column_lineage(sql, dialect)` API that extracts all output columns, traces each through the AST, and returns structured edges with lens classification (Unchanged/Alias/Transformation) - Star expansion through CTEs and derived tables — `SELECT * FROM cte` resolves to individual column edges instead of opaque star terminals - Depth guard (`MAX_LINEAGE_DEPTH=50`) prevents stack overflow on deeply nested derived tables and recursive CTEs - Zero-clone scope passing (`&[&Scope]` instead of `&[Scope]`) eliminates expensive AST cloning on the recursive hot path — 2.5x speedup on CTE-heavy queries, 60-83x faster than Python sqlglot overall SQL Tag Analysis (`polyglot-sql-tags`): - Standalone crate for static SQL analysis producing 6 tags: `create_or_replace_table`, `select_star`, `filter_has_func`, `join_has_func`, `agg_before_join`, `select_without_limit` - Expression tree walker (`deep_dfs`) handles three-part column refs, `Expression::Exists`, `CountFunc` star detection - 17x faster than equivalent Python/sqlglot implementation PyO3 Bindings (`polyglot-sql-python`): - `polyglot.column_lineage(sql, dialect)` → dict with edges and errors - `polyglot.analyze_tags(sql, dialect)` → dict of boolean tags - GIL-released execution via `py.allow_threads()` Validated on 593K production Snowflake queries across 6 tenants: - 430K edges, zero crashes, 1000-4700 queries/sec - 78% edge-level parity with Python sqlglot on Block queries - 738/738 Rust unit tests pass Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
lineage.rs—column_lineage(sql, dialect)traces source→target column edges through CTEs, derived tables, UNIONs, and subqueries with lens classification (Unchanged/Alias/Transformation)WITH cte AS (SELECT a FROM t) SELECT * FROM cteresolves to individual column edgesMAX_LINEAGE_DEPTH=50) prevents stack overflow on deeply nested queries and recursive CTEs&[&Scope]) eliminates AST cloning on recursive hot path — 2.5x speedup on CTE-heavy queriespolyglot-sql-tags) with 6 tags:create_or_replace_table,select_star,filter_has_func,join_has_func,agg_before_join,select_without_limitpolyglot-sql-python) exposingpolyglot.column_lineage()andpolyglot.analyze_tags()Validation
Tested on 593K production Snowflake queries across 6 tenants:
Test plan
cargo test -p polyglot-sql --lib— 738 tests passmaturin develop --release— Python bindings build🤖 Generated with Claude Code