feat: SQL query-plan diagrams, and a reports/ folder for generated artifacts

.claude/commands/project-costs.md

@@ -4,10 +4,10 @@ analysis into the generated markdown report.
 ## Steps
 
 1. Run `make project-costs` via Bash and capture the full output. It prints to stdout **and**
-   writes `cost_report/YYYY-MM-DD.md` (gitignored) containing every table in markdown, ending with
+   writes `reports/cost/YYYY-MM-DD.md` (gitignored) containing every table in markdown, ending with
    an empty `## Analysis` section. The script writes only numbers, never prose.
 2. Analyze the output (see below).
-3. **Edit `cost_report/YYYY-MM-DD.md` and replace the `## Analysis` placeholder with your written
+3. **Edit `reports/cost/YYYY-MM-DD.md` and replace the `## Analysis` placeholder with your written
    analysis.** Do not retype the tables — they are already in the file, and re-transcribing numbers
    risks introducing errors. Reference them instead.
 4. Report the same analysis back in chat, and link the report path.

.claude/commands/sql-diagram.md

@@ -0,0 +1,90 @@
+Diagram a SQL query and explain what it shows — either its execution steps or its column lineage.
+
+## Steps
+
+1. Get the SQL. If the user named a file, use it. If the query is embedded in Python (most of this
+   repo's SQL lives in f-strings under `scripts/`), extract it to a scratch `.sql` file first and
+   **replace the interpolated placeholders with literals** — `sqlglot` parses SQL, not f-strings.
+2. Pick the mode. `mode=plan` (the default) answers "what does this query *do*, step by step";
+   `mode=lineage` answers "where does this output column come from". When the user asks about
+   joins, stages, filters or ordering, they want `plan`.
+3. Run `make sql-diagram sql=<path> name=<basename> comments=1` via Bash. It writes three files to
+   `reports/sql-diagram/`: `<basename>.sql` (the query as analysed), `.mmd` and `.svg` — all
+   gitignored, so `git add -f` them only if they are meant to be a committed example. Pass
+   `--stdout` to `scripts/sql_diagram.py` for a throwaway look with no files written.
+4. Read the `.mmd`, show it in a ```mermaid fence, and explain it (see below). The `.svg` is the
+   same graph for linking from prose where no Mermaid renderer is available.
+
+The emitted `.sql` is what makes the diagram auditable: it is the query *after* any f-string
+placeholders were filled in, so `make sql-diagram sql=reports/sql-diagram/<basename>.sql` reproduces
+the diagram exactly. When you commit a diagram as an example, commit its `.sql` with it.
+
+Both diagrams come from the parsed AST, so they are exactly what the query says — do not "improve"
+one by adding a node or edge you believe should be there. If it looks wrong, the query is the thing
+to question.
+
+## Reading `mode=plan`
+
+Nodes are the query's steps, bottom-up: `SCAN` per table, one `JOIN n` per individual join, then
+`WHERE`, `AGGREGATE`, `SORT`, `OUTPUT`. A CTE appears as its own sub-pipeline feeding the `SCAN`
+that reads it.
+
+- **Each join is numbered in the order the query writes it** and carries its side and keys.
+  `sqlglot` models a multi-table join as one n-ary step; the script splits it back apart. An
+  `INNER JOIN` silently drops rows where a `LEFT JOIN` keeps them — always say which, because it
+  changes what a blank in the output means.
+- **Extra `ON` predicates beyond the equality keys** are listed under the keys as `and …`. On a
+  slowly-changing dimension those range predicates are what stop the join fanning out; call them
+  out rather than treating them as noise.
+- **This is the logical plan, not the physical one.** Databricks reorders joins, chooses broadcast
+  versus shuffle, and prunes columns. Say "as written" — and if the real execution matters, point
+  at the query profile in the UI or `EXPLAIN FORMATTED`, which is the only authority on what ran.
+- `AGGREGATE` may show synthetic operand names (`_a_0`) for `DISTINCT`/expression arguments that
+  `sqlglot` lifted out. Read the intent off the original SQL rather than repeating the placeholder.
+
+## Reading `mode=lineage`
+
+- **Subgraphs are source tables**, one node per source column actually read. A column the query
+  never touches does not appear — that is the point.
+- **The `output` subgraph** is the projected column list, in select order.
+- **`(unqualified)`** collects columns referenced without a table prefix in a multi-table join.
+  `sqlglot` will not guess which side they came from without the table schemas, and neither should
+  you. Call it out: it is usually a readability defect in the query worth fixing at the source.
+- **Struct columns collapse to their root.** `u.usage_metadata.job_id` traces back to
+  `usage_metadata`, not to the leaf field. Say so rather than implying field-level precision.
+- Columns in `WHERE`/`GROUP BY` but not in the output do **not** appear. Use `mode=plan` when
+  filtering is the point.
+
+## In either mode
+
+**The grey line under a table name** is its Unity Catalog comment, present only when the run passed
+`comments=1` and the profile could read the table. It is fetched, never written by you — if a table
+has no comment the space is blank, and that absence is itself worth reporting.
+
+## What to say about it
+
+- The **shape** of the query first: how many tables, how many joins, what it groups by. A reader
+  who cannot restate the query after your first paragraph has learned nothing.
+- Any source column or table feeding **many** outputs — the query's hub, where a schema change has
+  the widest blast radius.
+- Any table contributing **only one or two** columns, especially through a `LEFT JOIN`. That is
+  often a lookup that could be a smaller subquery, and a join whose only job is one column is a
+  cheap thing to get wrong.
+- Join predicates that look under-constrained. A join on a slowly-changing dimension without a
+  time-range predicate fans rows out and silently multiplies aggregates — this repo has been bitten
+  by exactly that (see the `#47` entry in `specs/CHANGELOG.md`).
+
+## Limits worth stating rather than hiding
+
+- `SELECT *` errors out in lineage mode by design — tracing it needs the table schemas, which the
+  script does not have. Plan mode draws it fine.
+- Lineage mode also refuses a query whose output projects the same column name twice
+  (`SELECT a.id, b.id`): `sqlglot` resolves lineage by name and would trace both to the first
+  match, drawing a confident wrong graph. Alias them, or use plan mode.
+- `CREATE TABLE … AS SELECT` and `INSERT … SELECT` are unwrapped to their SELECT and diagrammed.
+  Anything with no SELECT at all (a `DELETE`, a DDL statement) exits with a one-line message.
+- Dialect defaults to `databricks`; pass `--dialect` to `scripts/sql_diagram.py` directly for others.
+- CTEs resolve through to their base tables, but a query reading a **view** stops at the view name;
+  the view's own definition is not expanded.
+- `--comments` is the only part that touches the network, and it uses the `dev` profile: the MCP
+  service principal lacks `USE SCHEMA` on `system.billing`.

.github/workflows/onpush.yml

@@ -57,7 +57,7 @@ jobs:
         uses: actions/upload-artifact@v4
         with:
           name: coverage-report
-          path: coverage_reports/
+          path: reports/coverage/
           retention-days: 14
 
       # Pin the CLI to a tagged release so an upstream change can't silently break CI.

.gitignore

@@ -8,7 +8,7 @@ notes/
 .pytest_cache/
 dist/
 build/
-coverage_reports/
+reports/coverage/
 src/template.egg-info/
 *.pyc
 *.lock
@@ -23,4 +23,7 @@ resources/orders_dashboard_deploy.lvdash.json
 # Generated spend reports — local artifacts containing account cost figures.
 # A report that has been committed stays tracked (.gitignore only governs untracked files), so
 # new dated reports are ignored by default; `git add -f` a specific one to keep it.
-cost_report/
+reports/cost/
+# Generated query diagrams — same rule: ignored by default, `git add -f` one to keep it as an
+# example. They are derived artifacts, regenerable from the query at any time.
+reports/sql-diagram/

CLAUDE.md

@@ -8,13 +8,14 @@ A production-ready PySpark/Databricks ETL pipeline template using medallion arch
 
 ## Tooling: MCP servers, CLI, skills → see [`specs/tooling.md`](specs/tooling.md)
 
-Developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutions/ai-dev-kit) (user-level, `~/.ai-dev-kit/`). MCP config (`.mcp.json`) and `.claude/` are gitignored — user-level tooling, not committed; the exception is `.claude/commands/`, which is un-ignored and committed. Quick decision list (full reference in `specs/tooling.md`):
+Developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutions/ai-dev-kit) — **user-level tooling** (`~/.ai-dev-kit/`), never installed into or committed to this repo ([why that matters](specs/tooling.md#install-layout)). Quick decision list (full reference in [`specs/tooling.md`](specs/tooling.md)):
 
-- **Workspace / UC / Jobs / Pipelines / Apps / Serving / SQL** → prefer `mcp__databricks__*` tools over `databricks` CLI shell-outs or hand-rolled SDK scripts.
-- **Bundle / job changes** → `databricks-bundles` / `databricks-jobs` skills, and route job edits through `scripts/sdk_generate_template_job.py` + `make deploy`.
+- **Workspace / UC / Jobs / Pipelines / Apps / Serving / SQL** → prefer `mcp__databricks__*` tools over `databricks` CLI shell-outs or hand-rolled SDK scripts ([servers](specs/tooling.md#mcp-servers)).
+- **Bundle / job changes** → `databricks-bundles` / `databricks-jobs` skills, and route job edits through `scripts/sdk_generate_template_job.py` + `make deploy` ([skills](specs/tooling.md#skills)).
 - **Library/SDK docs** (PySpark, Databricks SDK, uv, ruff) → `context7` MCP, not memory or web search.
-- **Cloud spend / cost analysis** → `aws-billing-cost` MCP (`AWS_PROFILE=costs`) + `/project-costs` skill. **AWS docs** → `aws-documentation` MCP.
+- **Cloud spend / cost analysis** → `aws-billing-cost` MCP (`AWS_PROFILE=costs`) + `/project-costs`. **AWS docs** → `aws-documentation` MCP.
 - Use the `dev` profile unless told otherwise (`prod` for prod ops). If MCP tools are unavailable, fall back to CLI/SDK and flag it.
+- **MCP calls run as the prod SP, not as you** — `dev` is your user account, but the `databricks` MCP server is pinned to `DEFAULT`, which resolves to the same `template-sp` that `prod` uses. It can read/write `prod` tables; the catalog is the guardrail ([why](specs/tooling.md#mcp-runs-as-the-production-service-principal)).
 
 ## Commands
 
@@ -29,6 +30,7 @@ make run env=dev       # Run integration test job on a target env (dev or stagin
 make drop env=dev      # Drop all medallion tables in a target env (schema migrations; staging/prod need yes=--yes)
 make whoami            # Print the identity the env's profile authenticates as (runs implicitly before deploy/run/drop)
 make project-costs     # AWS + Databricks spend report (--aws-profile costs); backs the /project-costs skill
+make sql-diagram sql=q.sql  # Query plan (or mode=lineage) .mmd + .svg into reports/sql-diagram/; backs /sql-diagram
 make star-history      # Regenerate the README star-history SVGs (assets/star_history*.svg) from the GitHub API
 ```
 
@@ -71,8 +73,8 @@ The detailed specs live in [`specs/`](specs/) — read the relevant one **before
 
 - **Ask "should I open a new branch?" before executing a plan**, and **never commit directly to `main`** — cut a feature branch and land via PR (a hook blocks direct commits and pushes to `main`).
 - **Hold commits until asked.** Before merging, update the PR description (a hook uses it as the merge commit message body) following the What / Why / How / Validation / **Impact in prod** template in [`.github/PULL_REQUEST_TEMPLATE.md`](.github/PULL_REQUEST_TEMPLATE.md); any table schema/data change needs the production-table impact check.
-- **Keep docs in sync in the same commit.** Don't ship changes to the CLI surface (`main.py:arg_parser`), runtime env vars, catalog/schema model, or production guardrails without updating `README.md`, the relevant doc under `specs/`, and this file (`CLAUDE.md`) together.
-- **Add a `specs/CHANGELOG.md` entry immediately before merging a PR** (not while the work is in progress — scope grows, and an early entry just gets rewritten). Append-only (never edit old ones). Each entry is **exactly 3 sentences** and **at most ~5 rendered lines** (~475 chars); keep it one unwrapped paragraph — the line cap is a length budget, not a wrap width.
+- **Keep docs in sync in the same commit** — the CLI surface (`main.py:arg_parser`), runtime env vars, catalog/schema model, and production guardrails each need `README.md` + the relevant `specs/` doc + this file updated together ([full rule](specs/workflow.md#keep-docs-in-sync)).
+- **Add a `specs/CHANGELOG.md` entry immediately before merging** — never earlier; append-only; one unwrapped paragraph, ~1000 characters ([full rule](specs/workflow.md#changelog-discipline)).
 
 ## Keep It Simple
 

Makefile

@@ -37,6 +37,13 @@ project-costs: aws-profile ?= costs
 project-costs:
 	uv run python ./scripts/project_costs.py $(if $(aws-profile),--aws-profile $(aws-profile),)
 
+# Diagram a SQL query with sqlglot, writing .mmd + .svg into reports/sql-diagram/. Pass the query
+# with sql=path/to/query.sql (stdin if omitted); optionally name=basename, mode=plan|lineage
+# (default plan: the query's steps) and comments=1 (looks up each table's Unity Catalog comment).
+sql-diagram:
+	uv run python ./scripts/sql_diagram.py $(if $(sql),--file $(sql),) $(if $(name),--name $(name),) \
+		$(if $(mode),--mode $(mode),) $(if $(comments),--comments,)
+
 # Regenerate the README star-history chart from the GitHub API. The SVGs are committed,
 # so the README renders from this repo rather than a third-party chart service.
 star-history:

README.md

@@ -58,7 +58,8 @@ This project template demonstrates how to:
 - utilize the [Databricks SDK for Python](https://docs.databricks.com/en/dev-tools/sdk-python.html) to manage catalogs, schemas, workspaces, and accounts. Refer to the `scripts` folder for examples.
 - utilize [Databricks Unity Catalog](https://www.databricks.com/product/unity-catalog) to manage permissions and get data lineage.
 - enforce production guardrails out of the box — identity-locked CI deploys, a health-check task, wheel version pinning, per-task timeouts, schema-drift guards, queued runs, and on-call alerting.
-- track project cloud spend in USD across AWS (Cost Explorer) and Databricks ([`system.billing`](https://docs.databricks.com/aws/en/admin/system-tables/pricing)) with `make project-costs` — see an [example report](cost_report/2026-07-16.md).
+- track project cloud spend in USD across AWS (Cost Explorer) and Databricks ([`system.billing`](https://docs.databricks.com/aws/en/admin/system-tables/pricing)) with `make project-costs` — see an [example report](reports/cost/2026-07-22.md).
+- diagram any SQL query with `make sql-diagram sql=<file>` — [`sqlglot`](https://github.com/tobymao/sqlglot) parses the AST and writes a Mermaid flowchart plus a standalone SVG to `reports/sql-diagram/`, either as the query's execution steps (each scan, each join with its keys, filter, aggregate, sort — [example](reports/sql-diagram/job_spend_plan.svg)) or as column-level lineage (`mode=lineage`), so what is drawn is what the query says rather than what a model guessed.
 - utilize serverless job clusters on [Databricks Free Edition](https://docs.databricks.com/aws/en/getting-started/free-edition) to deploy your pipelines.
 
 

pyproject.toml

@@ -24,6 +24,7 @@ dev = [
     "pytest-cov==5.0.0",
     "pyspark==4.1.0",
     "databricks-bundles>=0.298.0",
+    "sqlglot==30.13.0",
 ]
 
 [project.scripts]
@@ -46,7 +47,7 @@ testpaths = ["tests"]
 python_files = ["test_*.py", "*_test.py", "unit_test_*.py"]
 python_classes = ["Test*"]
 python_functions = ["test_*"]
-addopts = "--cov=. --cov-report=term --cov-report=xml:coverage_reports/coverage.xml --cov-report=html:coverage_reports/html"
+addopts = "--cov=. --cov-report=term --cov-report=xml:reports/coverage/coverage.xml --cov-report=html:reports/coverage/html"
 
 [tool.ruff]
 line-length = 120