diff --git a/.claude/commands/project-costs.md b/.claude/commands/project-costs.md index fd46630..94568da 100644 --- a/.claude/commands/project-costs.md +++ b/.claude/commands/project-costs.md @@ -4,10 +4,10 @@ analysis into the generated markdown report. ## Steps 1. Run `make project-costs` via Bash and capture the full output. It prints to stdout **and** - writes `cost_report/YYYY-MM-DD.md` (gitignored) containing every table in markdown, ending with + writes `reports/cost/YYYY-MM-DD.md` (gitignored) containing every table in markdown, ending with an empty `## Analysis` section. The script writes only numbers, never prose. 2. Analyze the output (see below). -3. **Edit `cost_report/YYYY-MM-DD.md` and replace the `## Analysis` placeholder with your written +3. **Edit `reports/cost/YYYY-MM-DD.md` and replace the `## Analysis` placeholder with your written analysis.** Do not retype the tables — they are already in the file, and re-transcribing numbers risks introducing errors. Reference them instead. 4. Report the same analysis back in chat, and link the report path. diff --git a/.claude/commands/sql-diagram.md b/.claude/commands/sql-diagram.md new file mode 100644 index 0000000..5c8ed70 --- /dev/null +++ b/.claude/commands/sql-diagram.md @@ -0,0 +1,90 @@ +Diagram a SQL query and explain what it shows — either its execution steps or its column lineage. + +## Steps + +1. Get the SQL. If the user named a file, use it. If the query is embedded in Python (most of this + repo's SQL lives in f-strings under `scripts/`), extract it to a scratch `.sql` file first and + **replace the interpolated placeholders with literals** — `sqlglot` parses SQL, not f-strings. +2. Pick the mode. `mode=plan` (the default) answers "what does this query *do*, step by step"; + `mode=lineage` answers "where does this output column come from". When the user asks about + joins, stages, filters or ordering, they want `plan`. +3. Run `make sql-diagram sql= name= comments=1` via Bash. It writes three files to + `reports/sql-diagram/`: `.sql` (the query as analysed), `.mmd` and `.svg` — all + gitignored, so `git add -f` them only if they are meant to be a committed example. Pass + `--stdout` to `scripts/sql_diagram.py` for a throwaway look with no files written. +4. Read the `.mmd`, show it in a ```mermaid fence, and explain it (see below). The `.svg` is the + same graph for linking from prose where no Mermaid renderer is available. + +The emitted `.sql` is what makes the diagram auditable: it is the query *after* any f-string +placeholders were filled in, so `make sql-diagram sql=reports/sql-diagram/.sql` reproduces +the diagram exactly. When you commit a diagram as an example, commit its `.sql` with it. + +Both diagrams come from the parsed AST, so they are exactly what the query says — do not "improve" +one by adding a node or edge you believe should be there. If it looks wrong, the query is the thing +to question. + +## Reading `mode=plan` + +Nodes are the query's steps, bottom-up: `SCAN` per table, one `JOIN n` per individual join, then +`WHERE`, `AGGREGATE`, `SORT`, `OUTPUT`. A CTE appears as its own sub-pipeline feeding the `SCAN` +that reads it. + +- **Each join is numbered in the order the query writes it** and carries its side and keys. + `sqlglot` models a multi-table join as one n-ary step; the script splits it back apart. An + `INNER JOIN` silently drops rows where a `LEFT JOIN` keeps them — always say which, because it + changes what a blank in the output means. +- **Extra `ON` predicates beyond the equality keys** are listed under the keys as `and …`. On a + slowly-changing dimension those range predicates are what stop the join fanning out; call them + out rather than treating them as noise. +- **This is the logical plan, not the physical one.** Databricks reorders joins, chooses broadcast + versus shuffle, and prunes columns. Say "as written" — and if the real execution matters, point + at the query profile in the UI or `EXPLAIN FORMATTED`, which is the only authority on what ran. +- `AGGREGATE` may show synthetic operand names (`_a_0`) for `DISTINCT`/expression arguments that + `sqlglot` lifted out. Read the intent off the original SQL rather than repeating the placeholder. + +## Reading `mode=lineage` + +- **Subgraphs are source tables**, one node per source column actually read. A column the query + never touches does not appear — that is the point. +- **The `output` subgraph** is the projected column list, in select order. +- **`(unqualified)`** collects columns referenced without a table prefix in a multi-table join. + `sqlglot` will not guess which side they came from without the table schemas, and neither should + you. Call it out: it is usually a readability defect in the query worth fixing at the source. +- **Struct columns collapse to their root.** `u.usage_metadata.job_id` traces back to + `usage_metadata`, not to the leaf field. Say so rather than implying field-level precision. +- Columns in `WHERE`/`GROUP BY` but not in the output do **not** appear. Use `mode=plan` when + filtering is the point. + +## In either mode + +**The grey line under a table name** is its Unity Catalog comment, present only when the run passed +`comments=1` and the profile could read the table. It is fetched, never written by you — if a table +has no comment the space is blank, and that absence is itself worth reporting. + +## What to say about it + +- The **shape** of the query first: how many tables, how many joins, what it groups by. A reader + who cannot restate the query after your first paragraph has learned nothing. +- Any source column or table feeding **many** outputs — the query's hub, where a schema change has + the widest blast radius. +- Any table contributing **only one or two** columns, especially through a `LEFT JOIN`. That is + often a lookup that could be a smaller subquery, and a join whose only job is one column is a + cheap thing to get wrong. +- Join predicates that look under-constrained. A join on a slowly-changing dimension without a + time-range predicate fans rows out and silently multiplies aggregates — this repo has been bitten + by exactly that (see the `#47` entry in `specs/CHANGELOG.md`). + +## Limits worth stating rather than hiding + +- `SELECT *` errors out in lineage mode by design — tracing it needs the table schemas, which the + script does not have. Plan mode draws it fine. +- Lineage mode also refuses a query whose output projects the same column name twice + (`SELECT a.id, b.id`): `sqlglot` resolves lineage by name and would trace both to the first + match, drawing a confident wrong graph. Alias them, or use plan mode. +- `CREATE TABLE … AS SELECT` and `INSERT … SELECT` are unwrapped to their SELECT and diagrammed. + Anything with no SELECT at all (a `DELETE`, a DDL statement) exits with a one-line message. +- Dialect defaults to `databricks`; pass `--dialect` to `scripts/sql_diagram.py` directly for others. +- CTEs resolve through to their base tables, but a query reading a **view** stops at the view name; + the view's own definition is not expanded. +- `--comments` is the only part that touches the network, and it uses the `dev` profile: the MCP + service principal lacks `USE SCHEMA` on `system.billing`. diff --git a/.github/workflows/onpush.yml b/.github/workflows/onpush.yml index 1b88ba2..9e7a0c5 100644 --- a/.github/workflows/onpush.yml +++ b/.github/workflows/onpush.yml @@ -57,7 +57,7 @@ jobs: uses: actions/upload-artifact@v4 with: name: coverage-report - path: coverage_reports/ + path: reports/coverage/ retention-days: 14 # Pin the CLI to a tagged release so an upstream change can't silently break CI. diff --git a/.gitignore b/.gitignore index ddbf9cf..99b2d3d 100644 --- a/.gitignore +++ b/.gitignore @@ -8,7 +8,7 @@ notes/ .pytest_cache/ dist/ build/ -coverage_reports/ +reports/coverage/ src/template.egg-info/ *.pyc *.lock @@ -23,4 +23,7 @@ resources/orders_dashboard_deploy.lvdash.json # Generated spend reports — local artifacts containing account cost figures. # A report that has been committed stays tracked (.gitignore only governs untracked files), so # new dated reports are ignored by default; `git add -f` a specific one to keep it. -cost_report/ +reports/cost/ +# Generated query diagrams — same rule: ignored by default, `git add -f` one to keep it as an +# example. They are derived artifacts, regenerable from the query at any time. +reports/sql-diagram/ diff --git a/CLAUDE.md b/CLAUDE.md index c999fe9..19a0f9f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,13 +8,14 @@ A production-ready PySpark/Databricks ETL pipeline template using medallion arch ## Tooling: MCP servers, CLI, skills → see [`specs/tooling.md`](specs/tooling.md) -Developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutions/ai-dev-kit) (user-level, `~/.ai-dev-kit/`). MCP config (`.mcp.json`) and `.claude/` are gitignored — user-level tooling, not committed; the exception is `.claude/commands/`, which is un-ignored and committed. Quick decision list (full reference in `specs/tooling.md`): +Developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutions/ai-dev-kit) — **user-level tooling** (`~/.ai-dev-kit/`), never installed into or committed to this repo ([why that matters](specs/tooling.md#install-layout)). Quick decision list (full reference in [`specs/tooling.md`](specs/tooling.md)): -- **Workspace / UC / Jobs / Pipelines / Apps / Serving / SQL** → prefer `mcp__databricks__*` tools over `databricks` CLI shell-outs or hand-rolled SDK scripts. -- **Bundle / job changes** → `databricks-bundles` / `databricks-jobs` skills, and route job edits through `scripts/sdk_generate_template_job.py` + `make deploy`. +- **Workspace / UC / Jobs / Pipelines / Apps / Serving / SQL** → prefer `mcp__databricks__*` tools over `databricks` CLI shell-outs or hand-rolled SDK scripts ([servers](specs/tooling.md#mcp-servers)). +- **Bundle / job changes** → `databricks-bundles` / `databricks-jobs` skills, and route job edits through `scripts/sdk_generate_template_job.py` + `make deploy` ([skills](specs/tooling.md#skills)). - **Library/SDK docs** (PySpark, Databricks SDK, uv, ruff) → `context7` MCP, not memory or web search. -- **Cloud spend / cost analysis** → `aws-billing-cost` MCP (`AWS_PROFILE=costs`) + `/project-costs` skill. **AWS docs** → `aws-documentation` MCP. +- **Cloud spend / cost analysis** → `aws-billing-cost` MCP (`AWS_PROFILE=costs`) + `/project-costs`. **AWS docs** → `aws-documentation` MCP. - Use the `dev` profile unless told otherwise (`prod` for prod ops). If MCP tools are unavailable, fall back to CLI/SDK and flag it. +- **MCP calls run as the prod SP, not as you** — `dev` is your user account, but the `databricks` MCP server is pinned to `DEFAULT`, which resolves to the same `template-sp` that `prod` uses. It can read/write `prod` tables; the catalog is the guardrail ([why](specs/tooling.md#mcp-runs-as-the-production-service-principal)). ## Commands @@ -29,6 +30,7 @@ make run env=dev # Run integration test job on a target env (dev or stagin make drop env=dev # Drop all medallion tables in a target env (schema migrations; staging/prod need yes=--yes) make whoami # Print the identity the env's profile authenticates as (runs implicitly before deploy/run/drop) make project-costs # AWS + Databricks spend report (--aws-profile costs); backs the /project-costs skill +make sql-diagram sql=q.sql # Query plan (or mode=lineage) .mmd + .svg into reports/sql-diagram/; backs /sql-diagram make star-history # Regenerate the README star-history SVGs (assets/star_history*.svg) from the GitHub API ``` @@ -71,8 +73,8 @@ The detailed specs live in [`specs/`](specs/) — read the relevant one **before - **Ask "should I open a new branch?" before executing a plan**, and **never commit directly to `main`** — cut a feature branch and land via PR (a hook blocks direct commits and pushes to `main`). - **Hold commits until asked.** Before merging, update the PR description (a hook uses it as the merge commit message body) following the What / Why / How / Validation / **Impact in prod** template in [`.github/PULL_REQUEST_TEMPLATE.md`](.github/PULL_REQUEST_TEMPLATE.md); any table schema/data change needs the production-table impact check. -- **Keep docs in sync in the same commit.** Don't ship changes to the CLI surface (`main.py:arg_parser`), runtime env vars, catalog/schema model, or production guardrails without updating `README.md`, the relevant doc under `specs/`, and this file (`CLAUDE.md`) together. -- **Add a `specs/CHANGELOG.md` entry immediately before merging a PR** (not while the work is in progress — scope grows, and an early entry just gets rewritten). Append-only (never edit old ones). Each entry is **exactly 3 sentences** and **at most ~5 rendered lines** (~475 chars); keep it one unwrapped paragraph — the line cap is a length budget, not a wrap width. +- **Keep docs in sync in the same commit** — the CLI surface (`main.py:arg_parser`), runtime env vars, catalog/schema model, and production guardrails each need `README.md` + the relevant `specs/` doc + this file updated together ([full rule](specs/workflow.md#keep-docs-in-sync)). +- **Add a `specs/CHANGELOG.md` entry immediately before merging** — never earlier; append-only; one unwrapped paragraph, ~1000 characters ([full rule](specs/workflow.md#changelog-discipline)). ## Keep It Simple diff --git a/Makefile b/Makefile index 33bac1b..1e2a57e 100644 --- a/Makefile +++ b/Makefile @@ -37,6 +37,13 @@ project-costs: aws-profile ?= costs project-costs: uv run python ./scripts/project_costs.py $(if $(aws-profile),--aws-profile $(aws-profile),) +# Diagram a SQL query with sqlglot, writing .mmd + .svg into reports/sql-diagram/. Pass the query +# with sql=path/to/query.sql (stdin if omitted); optionally name=basename, mode=plan|lineage +# (default plan: the query's steps) and comments=1 (looks up each table's Unity Catalog comment). +sql-diagram: + uv run python ./scripts/sql_diagram.py $(if $(sql),--file $(sql),) $(if $(name),--name $(name),) \ + $(if $(mode),--mode $(mode),) $(if $(comments),--comments,) + # Regenerate the README star-history chart from the GitHub API. The SVGs are committed, # so the README renders from this repo rather than a third-party chart service. star-history: diff --git a/README.md b/README.md index 7a21e8e..8c30dbd 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,8 @@ This project template demonstrates how to: - utilize the [Databricks SDK for Python](https://docs.databricks.com/en/dev-tools/sdk-python.html) to manage catalogs, schemas, workspaces, and accounts. Refer to the `scripts` folder for examples. - utilize [Databricks Unity Catalog](https://www.databricks.com/product/unity-catalog) to manage permissions and get data lineage. - enforce production guardrails out of the box — identity-locked CI deploys, a health-check task, wheel version pinning, per-task timeouts, schema-drift guards, queued runs, and on-call alerting. -- track project cloud spend in USD across AWS (Cost Explorer) and Databricks ([`system.billing`](https://docs.databricks.com/aws/en/admin/system-tables/pricing)) with `make project-costs` — see an [example report](cost_report/2026-07-16.md). +- track project cloud spend in USD across AWS (Cost Explorer) and Databricks ([`system.billing`](https://docs.databricks.com/aws/en/admin/system-tables/pricing)) with `make project-costs` — see an [example report](reports/cost/2026-07-22.md). +- diagram any SQL query with `make sql-diagram sql=` — [`sqlglot`](https://github.com/tobymao/sqlglot) parses the AST and writes a Mermaid flowchart plus a standalone SVG to `reports/sql-diagram/`, either as the query's execution steps (each scan, each join with its keys, filter, aggregate, sort — [example](reports/sql-diagram/job_spend_plan.svg)) or as column-level lineage (`mode=lineage`), so what is drawn is what the query says rather than what a model guessed. - utilize serverless job clusters on [Databricks Free Edition](https://docs.databricks.com/aws/en/getting-started/free-edition) to deploy your pipelines. diff --git a/pyproject.toml b/pyproject.toml index 40b37fa..0d2f634 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -24,6 +24,7 @@ dev = [ "pytest-cov==5.0.0", "pyspark==4.1.0", "databricks-bundles>=0.298.0", + "sqlglot==30.13.0", ] [project.scripts] @@ -46,7 +47,7 @@ testpaths = ["tests"] python_files = ["test_*.py", "*_test.py", "unit_test_*.py"] python_classes = ["Test*"] python_functions = ["test_*"] -addopts = "--cov=. --cov-report=term --cov-report=xml:coverage_reports/coverage.xml --cov-report=html:coverage_reports/html" +addopts = "--cov=. --cov-report=term --cov-report=xml:reports/coverage/coverage.xml --cov-report=html:reports/coverage/html" [tool.ruff] line-length = 120 diff --git a/cost_report/2026-07-16.md b/reports/cost/2026-07-22.md similarity index 59% rename from cost_report/2026-07-16.md rename to reports/cost/2026-07-22.md index 65affce..83d679c 100644 --- a/cost_report/2026-07-16.md +++ b/reports/cost/2026-07-22.md @@ -1,6 +1,6 @@ -# Cost Report — 2026-07-16 +# Cost Report — 2026-07-22 -Window: **2026-06-16 → 2026-07-16** (30 days). Generated by `scripts/project_costs.py`. +Window: **2026-06-22 → 2026-07-22** (30 days). Generated by `scripts/project_costs.py`. > Quantity spans DBU/DSU/GB and is never totalled; only USD is comparable across SKUs and clouds. > Databricks USD is list price (system.billing.list_prices, effective_list) — it excludes account discounts and commit contracts, so treat it as an upper bound. @@ -8,51 +8,55 @@ Window: **2026-06-16 → 2026-07-16** (30 days). Generated by `scripts/project_c ## Combined Totals by Service -Total: **$34.06** (AWS $0.16, Databricks $33.91) +Total: **$39.81** (AWS $0.38, Databricks $39.43) | Cloud | Service | Quantity | Unit | USD | |---|---|---|---|---| -| AWS | Amazon Simple Storage Service | — | — | 0.0994 | -| AWS | AWS Cost Explorer | — | — | 0.0400 | +| AWS | AWS Cost Explorer | — | — | 0.1900 | +| AWS | Amazon Simple Storage Service | — | — | 0.1400 | +| AWS | Tax | — | — | 0.0300 | | AWS | Amazon EC2 Container Registry (ECR) | — | — | 0.0177 | -| AWS | AWS Secrets Manager | — | — | 0.0000 | -| Databricks | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 68.7187 | DBU | 24.0515 | -| Databricks | PREMIUM_SERVERLESS_SQL_COMPUTE_US_EAST_OHIO | 13.2725 | DBU | 9.2908 | -| Databricks | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 23.6661 | DSU | 0.5443 | -| Databricks | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.3960 | GB | 0.0178 | -| Databricks | INTER_REGION_EGRESS_FROM_US_EAST_OHIO | 0.0279 | GB | 0.0006 | -| Databricks | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0026 | GB | 0.0002 | +| Databricks | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 68.0867 | DBU | 23.8304 | +| Databricks | PREMIUM_SERVERLESS_SQL_COMPUTE_US_EAST_OHIO | 21.5698 | DBU | 15.0988 | +| Databricks | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 20.6628 | DSU | 0.4752 | +| Databricks | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.4952 | GB | 0.0223 | +| Databricks | INTER_REGION_EGRESS_FROM_US_EAST_OHIO | 0.0470 | GB | 0.0009 | +| Databricks | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0027 | GB | 0.0002 | + +## Databricks — by Job / Pipeline (USD) + +Attributed total: **$21.49** over 30 days at list price. + +| Entity | Kind | Quantity | Unit | USD | Days | +|---|---|---|---|---|---| +| job1_prod | job | 18.8054 | DBU | 6.5819 | 31 | +| job1_sdp_prod | pipeline | 11.5150 | DBU | 4.0302 | 31 | +| job1_prod_integration | job | 10.7937 | DBU | 3.7778 | 31 | +| job1_staging_integration_test | job | 7.6346 | DBU | 2.6721 | 5 | +| job1_staging | job | 7.0733 | DBU | 2.4757 | 5 | +| job1_sdp_staging | pipeline | 4.6397 | DBU | 1.6239 | 5 | +| [dev andre_f_salvati] job1_dev_integration_test | job | 0.4090 | DBU | 0.1432 | 1 | +| [dev andre_f_salvati] job1_dev | job | 0.3392 | DBU | 0.1187 | 1 | +| [dev andre_f_salvati] job1_sdp_dev | pipeline | 0.1850 | DBU | 0.0647 | 1 | + +> Only usage tagged with a job_id or dlt_pipeline_id is attributable (55% of Databricks spend here); SQL warehouse and other interactive compute carry neither, so this table is a breakdown of scheduled work, not of the whole bill. ## AWS — by Week × Service (USD) -Total: **$0.1572** over 30 days. Weeks marked `*` contain estimated (not-yet-finalized) days. +Total: **$0.3777** over 30 days. Weeks marked `*` contain estimated (not-yet-finalized) days. -| Week | Cost Explorer | EC2 Container Registry (ECR) | Secrets Manager | Simple Storage Service | Total | +| Week | Cost Explorer | EC2 Container Registry (ECR) | Simple Storage Service | Tax | Total | |---|---|---|---|---|---| -| 2026-06-15 | 0.0200 | 0.0036 | 0.0000 | 0.0125 | 0.0361 | -| 2026-06-22 | 0.0200 | 0.0042 | 0.0000 | 0.0355 | 0.0597 | -| 2026-06-29* | 0.0000 | 0.0041 | 0.0000 | 0.0461 | 0.0503 | -| 2026-07-06* | 0.0000 | 0.0041 | 0.0000 | 0.0045 | 0.0086 | -| 2026-07-13* | 0.0000 | 0.0017 | 0.0000 | 0.0008 | 0.0025 | +| 2026-06-22 | 0.0200 | 0.0042 | 0.0355 | 0.0000 | 0.0597 | +| 2026-06-29* | 0.0000 | 0.0041 | 0.0461 | 0.0300 | 0.0803 | +| 2026-07-06* | 0.0000 | 0.0041 | 0.0045 | 0.0000 | 0.0086 | +| 2026-07-13* | 0.1500 | 0.0041 | 0.0384 | 0.0000 | 0.1925 | +| 2026-07-20* | 0.0200 | 0.0012 | 0.0154 | 0.0000 | 0.0366 |
Daily detail (non-zero rows) | Date | Service | USD | Estimated | |---|---|---|---| -| 2026-06-16 | Amazon EC2 Container Registry (ECR) | 0.0006 | False | -| 2026-06-16 | Amazon Simple Storage Service | 0.0003 | False | -| 2026-06-17 | AWS Cost Explorer | 0.0200 | False | -| 2026-06-17 | AWS Secrets Manager | 0.0000 | False | -| 2026-06-17 | Amazon EC2 Container Registry (ECR) | 0.0006 | False | -| 2026-06-17 | Amazon Simple Storage Service | 0.0017 | False | -| 2026-06-18 | Amazon EC2 Container Registry (ECR) | 0.0006 | False | -| 2026-06-18 | Amazon Simple Storage Service | 0.0015 | False | -| 2026-06-19 | Amazon EC2 Container Registry (ECR) | 0.0006 | False | -| 2026-06-19 | Amazon Simple Storage Service | 0.0067 | False | -| 2026-06-20 | Amazon EC2 Container Registry (ECR) | 0.0006 | False | -| 2026-06-20 | Amazon Simple Storage Service | 0.0021 | False | -| 2026-06-21 | Amazon EC2 Container Registry (ECR) | 0.0006 | False | -| 2026-06-21 | Amazon Simple Storage Service | 0.0002 | False | | 2026-06-22 | Amazon EC2 Container Registry (ECR) | 0.0006 | False | | 2026-06-22 | Amazon Simple Storage Service | 0.0007 | False | | 2026-06-23 | AWS Cost Explorer | 0.0200 | False | @@ -74,6 +78,7 @@ Total: **$0.1572** over 30 days. Weeks marked `*` contain estimated (not-yet-fin | 2026-06-30 | Amazon Simple Storage Service | 0.0003 | False | | 2026-07-01 | Amazon EC2 Container Registry (ECR) | 0.0006 | True | | 2026-07-01 | Amazon Simple Storage Service | 0.0011 | True | +| 2026-07-01 | Tax | 0.0300 | True | | 2026-07-02 | Amazon EC2 Container Registry (ECR) | 0.0006 | True | | 2026-07-02 | Amazon Simple Storage Service | 0.0014 | True | | 2026-07-03 | Amazon EC2 Container Registry (ECR) | 0.0006 | True | @@ -100,31 +105,73 @@ Total: **$0.1572** over 30 days. Weeks marked `*` contain estimated (not-yet-fin | 2026-07-13 | Amazon Simple Storage Service | 0.0003 | True | | 2026-07-14 | Amazon EC2 Container Registry (ECR) | 0.0006 | True | | 2026-07-14 | Amazon Simple Storage Service | 0.0003 | True | -| 2026-07-15 | Amazon EC2 Container Registry (ECR) | 0.0005 | True | +| 2026-07-15 | Amazon EC2 Container Registry (ECR) | 0.0006 | True | | 2026-07-15 | Amazon Simple Storage Service | 0.0003 | True | +| 2026-07-16 | AWS Cost Explorer | 0.1500 | True | +| 2026-07-16 | Amazon EC2 Container Registry (ECR) | 0.0006 | True | +| 2026-07-16 | Amazon Simple Storage Service | 0.0368 | True | +| 2026-07-17 | Amazon EC2 Container Registry (ECR) | 0.0006 | True | +| 2026-07-17 | Amazon Simple Storage Service | 0.0003 | True | +| 2026-07-18 | Amazon EC2 Container Registry (ECR) | 0.0006 | True | +| 2026-07-18 | Amazon Simple Storage Service | 0.0003 | True | +| 2026-07-19 | Amazon EC2 Container Registry (ECR) | 0.0006 | True | +| 2026-07-19 | Amazon Simple Storage Service | 0.0003 | True | +| 2026-07-20 | Amazon EC2 Container Registry (ECR) | 0.0006 | True | +| 2026-07-20 | Amazon Simple Storage Service | 0.0002 | True | +| 2026-07-21 | AWS Cost Explorer | 0.0200 | True | +| 2026-07-21 | Amazon EC2 Container Registry (ECR) | 0.0006 | True | +| 2026-07-21 | Amazon Simple Storage Service | 0.0152 | True |
## Databricks — by Week × SKU (USD) -Total: **$33.91** over 30 days at list price. Native DBU/DSU quantities are in the Combined Totals table above. +Total: **$39.43** over 30 days at list price. Native DBU/DSU quantities are in the Combined Totals table above. | Week | DATABRICKS_STORAGE | INTERNET_EGRESS_FROM | INTER_REGION_EGRESS_FROM | JOBS_SERVERLESS_COMPUTE | PUBLIC_CONNECTIVITY_DATA_PROCESSED | SERVERLESS_SQL_COMPUTE | Total | |---|---|---|---|---|---|---|---| -| 2026-06-15 | 0.1408 | 0.0000 | 0.0000 | 5.6916 | 0.0024 | 0.6730 | 6.5080 | | 2026-06-22 | 0.1905 | 0.0001 | 0.0003 | 8.5157 | 0.0060 | 8.6177 | 17.3303 | | 2026-06-29 | 0.0994 | 0.0001 | 0.0003 | 5.1243 | 0.0054 | 0.0000 | 5.2294 | | 2026-07-06 | 0.0823 | 0.0000 | 0.0000 | 2.9122 | 0.0027 | 0.0000 | 2.9972 | -| 2026-07-13 | 0.0313 | 0.0000 | 0.0000 | 1.8077 | 0.0013 | 0.0000 | 1.8404 | +| 2026-07-13 | 0.0788 | 0.0000 | 0.0003 | 5.2072 | 0.0054 | 2.9036 | 8.1953 | +| 2026-07-20 | 0.0242 | 0.0000 | 0.0001 | 2.0710 | 0.0028 | 3.5775 | 5.6758 |
Daily detail | Date | SKU | Quantity | Unit | USD | |---|---|---|---|---| -| 2026-07-16 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 1.2668 | DBU | 0.4434 | -| 2026-07-16 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0074 | GB | 0.0003 | -| 2026-07-16 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 0.0069 | DSU | 0.0002 | -| 2026-07-16 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0000 | GB | 0.0000 | +| 2026-07-22 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 1.0076 | DBU | 0.3526 | +| 2026-07-22 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0074 | GB | 0.0003 | +| 2026-07-22 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 0.0044 | DSU | 0.0001 | +| 2026-07-22 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0000 | GB | 0.0000 | +| 2026-07-21 | PREMIUM_SERVERLESS_SQL_COMPUTE_US_EAST_OHIO | 5.1108 | DBU | 3.5775 | +| 2026-07-21 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 3.6907 | DBU | 1.2918 | +| 2026-07-21 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 0.6479 | DSU | 0.0149 | +| 2026-07-21 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0472 | GB | 0.0021 | +| 2026-07-21 | INTER_REGION_EGRESS_FROM_US_EAST_OHIO | 0.0059 | GB | 0.0001 | +| 2026-07-21 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0002 | GB | 0.0000 | +| 2026-07-20 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 1.2189 | DBU | 0.4266 | +| 2026-07-20 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 0.4020 | DSU | 0.0092 | +| 2026-07-20 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0074 | GB | 0.0003 | +| 2026-07-20 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0000 | GB | 0.0000 | +| 2026-07-19 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 1.3301 | DBU | 0.4656 | +| 2026-07-19 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 0.4254 | DSU | 0.0098 | +| 2026-07-19 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0075 | GB | 0.0003 | +| 2026-07-19 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0001 | GB | 0.0000 | +| 2026-07-18 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 1.1092 | DBU | 0.3882 | +| 2026-07-18 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 0.4077 | DSU | 0.0094 | +| 2026-07-18 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0074 | GB | 0.0003 | +| 2026-07-18 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0000 | GB | 0.0000 | +| 2026-07-17 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 1.0592 | DBU | 0.3707 | +| 2026-07-17 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 0.4014 | DSU | 0.0092 | +| 2026-07-17 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0074 | GB | 0.0003 | +| 2026-07-17 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0000 | GB | 0.0000 | +| 2026-07-16 | PREMIUM_SERVERLESS_SQL_COMPUTE_US_EAST_OHIO | 4.1480 | DBU | 2.9036 | +| 2026-07-16 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 7.4810 | DBU | 2.6183 | +| 2026-07-16 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 0.8389 | DSU | 0.0193 | +| 2026-07-16 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0755 | GB | 0.0034 | +| 2026-07-16 | INTER_REGION_EGRESS_FROM_US_EAST_OHIO | 0.0137 | GB | 0.0003 | +| 2026-07-16 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0003 | GB | 0.0000 | | 2026-07-15 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 1.4313 | DBU | 0.5010 | | 2026-07-15 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 0.4137 | DSU | 0.0095 | | 2026-07-15 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0075 | GB | 0.0003 | @@ -230,92 +277,54 @@ Total: **$33.91** over 30 days at list price. Native DBU/DSU quantities are in t | 2026-06-22 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0083 | GB | 0.0004 | | 2026-06-22 | INTER_REGION_EGRESS_FROM_US_EAST_OHIO | 0.0004 | GB | 0.0000 | | 2026-06-22 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0001 | GB | 0.0000 | -| 2026-06-21 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 2.5523 | DBU | 0.8933 | -| 2026-06-21 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 1.0001 | DSU | 0.0230 | -| 2026-06-21 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0075 | GB | 0.0003 | -| 2026-06-21 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0001 | GB | 0.0000 | -| 2026-06-20 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 2.7903 | DBU | 0.9766 | -| 2026-06-20 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 1.0249 | DSU | 0.0236 | -| 2026-06-20 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0075 | GB | 0.0003 | -| 2026-06-20 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0001 | GB | 0.0000 | -| 2026-06-19 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 2.8831 | DBU | 1.0091 | -| 2026-06-19 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 1.0947 | DSU | 0.0252 | -| 2026-06-19 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0157 | GB | 0.0007 | -| 2026-06-19 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0001 | GB | 0.0000 | -| 2026-06-19 | INTER_REGION_EGRESS_FROM_US_EAST_OHIO | 0.0005 | GB | 0.0000 | -| 2026-06-18 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 2.5505 | DBU | 0.8927 | -| 2026-06-18 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 1.0165 | DSU | 0.0234 | -| 2026-06-18 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0075 | GB | 0.0003 | -| 2026-06-18 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0001 | GB | 0.0000 | -| 2026-06-17 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 2.8623 | DBU | 1.0018 | -| 2026-06-17 | PREMIUM_SERVERLESS_SQL_COMPUTE_US_EAST_OHIO | 0.9615 | DBU | 0.6730 | -| 2026-06-17 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 1.0044 | DSU | 0.0231 | -| 2026-06-17 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0075 | GB | 0.0003 | -| 2026-06-17 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0001 | GB | 0.0000 | -| 2026-06-16 | PREMIUM_JOBS_SERVERLESS_COMPUTE_US_EAST_OHIO | 2.6234 | DBU | 0.9182 | -| 2026-06-16 | PREMIUM_DATABRICKS_STORAGE_US_EAST_OHIO | 0.9836 | DSU | 0.0226 | -| 2026-06-16 | PUBLIC_CONNECTIVITY_DATA_PROCESSED_US_EAST_OHIO | 0.0075 | GB | 0.0003 | -| 2026-06-16 | INTERNET_EGRESS_FROM_US_EAST_OHIO | 0.0001 | GB | 0.0000 |
## Analysis -**$34.06 for the 30 days, and Databricks is $33.91 of it — 99.5%.** AWS contributes $0.16, which is -rounding error. Databricks figures are list price and therefore an upper bound. - -### Databricks Costs ($33.91) - -**Jobs Serverless is the whole story at $24.05** (68.7187 DBU × $0.35), SQL Serverless second at -$9.29 (13.2725 DBU × $0.70). Storage is $0.54 (23.6661 DSU) and all networking together is under two -cents. SQL bills at 2× the Jobs rate, so it punches above its quantity: 16% of DBUs, 27% of cost. - -The heaviest week is **2026-06-22 at $17.33** — more than the other four combined — split $8.52 Jobs -/ $8.62 SQL. Within it, **Jun 24 alone was $9.13** (9.78 Jobs DBU = $3.42, 8.08 SQL DBU = $5.65), -followed by Jun 23 at $3.66. **Jul 3 is the other peak at $2.64** (7.46 Jobs DBU), pure batch with no -SQL at all. - -**SQL Serverless ran in only two weeks — $0.67 in wk 06-15 (Jun 17) and $8.62 in wk 06-22 (Jun 23, -24, 26) — then nothing for three weeks.** That $9.29 burst is 27% of the month's spend from four -days of interactive work, and the warehouse has been silent since. Worth confirming that's intended -rather than a dashboard with no audience. - -Trend: a **step-down, not a drift**. Jobs cost per day runs $0.95 → $1.22 → $0.73 → $0.42 → **$0.45** -across the five weeks — roughly a 55% cut landing around Jun 24–25 that has since **flattened, not -kept falling**. The raw weekly totals ($2.99 → $1.84) imply an ongoing collapse; that's an artifact -of the 4-day final week. Storage tracks the same halving, ~1.0 DSU/day mid-June to ~0.5/day in July. - -### AWS Costs ($0.16) - -**S3 at $0.0994 (63%)** leads, then Cost Explorer at $0.0400 (two $0.02 API-charge days, Jun 17 and -Jun 23) and ECR at $0.0177 of flat image storage. The week of **2026-06-29 carries $0.0461 of S3**, -and the daily detail pins it on **Jul 3 at $0.0418** — ~165× a typical $0.00025 S3 day. Normalized -per day, S3 falls from ~$0.0051/day (wk 06-22) to ~$0.0003/day and flattens — the same shape as -Databricks. - -The framing that matters: **Jul 3's "165× S3 spike" is 4 cents; that same day's Databricks compute -was $2.64** — 16× the entire month of AWS. AWS here is a signal, not a cost. - -### Cross-cloud observation - -The correlation holds where it should. **Jul 3**: 7.46 Jobs DBU ($2.61) against S3 $0.0418 and -public connectivity 0.0730 GB, 10× the flat 0.0075 GB/day, plus the window's largest inter-region -egress. **Jun 24** and **Jun 23** repeat it — $9.13 and $3.66 of compute against S3 at $0.0181 and -$0.0152. - -Two informative exceptions. **Jul 12's S3 bump ($0.0028) has no Databricks counterpart** — compute, -connectivity and egress were all flat, so that I/O came from something other than a pipeline run. -And **Jun 26's $1.92 (mostly $1.51 of SQL) moved AWS not at all** ($0.0008, a baseline day), which is -expected: warehouse queries read Databricks-managed storage, not project S3. So S3 tracks **Jobs -Serverless specifically, not total DBU**. - -That's structural. Serverless SKUs are all-in — compute is billed inside the DBU rate rather than -charged to the AWS account as EC2 — which is why the AWS bill is only S3, ECR and Cost Explorer, and -why AWS spend can never measure pipeline cost. - -### Bottom line - -At ~$34/month list this is a cheap project, but every lever is on the Databricks side. Two things -worth a decision: the **$9.29 of SQL Serverless** consumed in four days of late June and silent -since, and whether the **~55% Jobs step-down around Jun 24–25** was deliberate. If you're on a -discounted contract the real invoice is below $33.91 — nothing in the system tables says by how much. +**$39.81 total over the 30-day window, of which Databricks is $39.43 (99.0%) at list price and AWS +is $0.38 (1.0%).** Any conversation about this project's cost is a conversation about DBUs; the AWS +account is a rounding error and is only useful as a proxy for job activity. + +**AWS.** $0.38 for the window, and the single biggest line is not infrastructure — it is the Cost +Explorer API at $0.19, half of all AWS spend. S3 is $0.14, ECR a flat $0.0006/day. The one week +above baseline is 2026-07-13 at $0.1925, and the daily block attributes essentially all of it to +2026-07-16: $0.15 of Cost Explorer calls plus $0.0368 of S3, the day the weekly-pivot rework landed +and the script was run repeatedly. Cost Explorer bills per request, so that spike is this very +report measuring itself. Trend is flat noise; nothing here is worth optimizing. + +**Databricks.** $39.43 at list, from 68.09 DBU of Jobs Serverless ($23.83, 60%) and 21.57 DBU of +SQL Serverless ($15.10, 38%); storage adds 20.66 DSU ($0.48) and egress $0.02. The window is +front-loaded: the week of 2026-06-22 alone is $17.33, 44% of the total, split almost evenly between +jobs ($8.52) and SQL ($8.62), and inside it 2026-06-24 is a single $9.09 day — 23% of the entire +30-day bill in one day. Normalizing away the partial edge weeks, Jobs Serverless runs about +$1.22/day in the week of 06-22, dips to $0.42/day in the quiet week of 07-06, and settles at +~$0.72/day since; that is a stable baseline with a late-June spike, not a downward trend. SQL +Serverless is the opposite of steady — it fires on 06-23/24/26, then goes **completely silent for +the weeks of 06-29 and 07-06 ($0.00)**, and returns only on 07-16 ($2.90) and 07-21 ($3.58). The +warehouse only wakes when someone is actively working on the dashboard or running queries by hand; +two and a half silent weeks means nothing scheduled touches it, and nobody opens the dashboard on +an ordinary day. At $15.10 for what is essentially five days of interactive work, it is the second +most expensive thing in the project. + +**By job / pipeline.** $21.49 is attributable (55% of Databricks spend); the unattributed $17.94 is +almost entirely the SQL warehouse, which carries no `job_id` — that reconciles, so the breakdown is +trustworthy as a picture of scheduled work only. Compared per *active* day rather than by raw +total: prod runs 31 days at $0.46/day across its three entities, while staging ran 5 days at +$1.35/day — **2.9× prod's daily burn**, because a staging day is a full deploy-and-verify cycle +rather than one scheduled pass. The batch-vs-SDP gap is the clearest signal in the table: +`job1_prod` costs $6.58 against `job1_sdp_prod`'s $4.03 for the same medallion tables, and staging +repeats it ($2.48 vs $1.62). A 35–39% saving in favour of the declarative pipeline, holding over 31 +days and across two environments, is a real finding rather than noise. The integration tests +deserve the attention the skill warns about: `job1_prod_integration` at $3.78 is 57% of the +pipeline it validates, and in staging the integration test ($2.67) actually *costs more* than +`job1_staging` ($2.48) itself. + +**Cross-cloud.** The correlation holds where it should: the five heaviest S3 days (07-03, 07-16, +06-24, 07-21, 06-23) are all among the six heaviest Jobs Serverless days, confirming that S3 I/O +tracks job activity. The instructive exception is 06-24 — the single biggest compute day of the +window, yet only third in S3, because most of that day's cost was SQL Serverless ($5.65), and +warehouse queries read Databricks-managed storage rather than our bucket. That is the mechanism +working exactly as expected, and it is why AWS spend can never be read as a measure of pipeline +cost: serverless SKUs bill the compute inside the DBU rate, so the EC2 that ran these jobs never +appears on the AWS invoice at all. diff --git a/reports/sql-diagram/job_spend_plan.mmd b/reports/sql-diagram/job_spend_plan.mmd new file mode 100644 index 0000000..689814a --- /dev/null +++ b/reports/sql-diagram/job_spend_plan.mmd @@ -0,0 +1,22 @@ +flowchart LR + n0[("SCAN system.lakeflow.pipelines
The `pipelines` table is a slow-changing dimension table (SCD2) that…")] + n1["AGGREGATE
GROUP BY pipeline_id
MAX_BY(name, change_time) AS name"] + n2[("SCAN pipe_names")] + n3[("SCAN system.billing.list_prices
The pricing table gives you access to a historical log of SKU…")] + n4[("SCAN system.billing.usage
The usage table gives you access to account-wide billable usage data.…")] + n5{{"JOIN 1 · LEFT · p
u.sku_name = p.sku_name
and u.usage_end_time >= p.price_start_time
and (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)"}} + n6{{"JOIN 2 · LEFT · n
u.usage_metadata.dlt_pipeline_id = n.pipeline_id"}} + n7["WHERE
u.usage_date >= CURRENT_DATE - INTERVAL '30' DAYS
NOT COALESCE(u.usage_metadata.job_id,…"] + n8["AGGREGATE
GROUP BY entity, kind, u.usage_unit
SUM(u.usage_quantity) AS quantity
SUM(`_a_0`) AS usd
COUNT(`_a_1`) AS active_days"] + n9["SORT
usd DESC"] + n10(["OUTPUT
entity
kind
quantity
usage_unit
usd
active_days"]) + n0 --> n1 + n1 --> n2 + n4 --> n5 + n3 --> n5 + n5 --> n6 + n2 --> n6 + n6 --> n7 + n7 --> n8 + n8 --> n9 + n9 --> n10 diff --git a/reports/sql-diagram/job_spend_plan.sql b/reports/sql-diagram/job_spend_plan.sql new file mode 100644 index 0000000..824b0ab --- /dev/null +++ b/reports/sql-diagram/job_spend_plan.sql @@ -0,0 +1,23 @@ +WITH pipe_names AS ( + SELECT pipeline_id, MAX_BY(name, change_time) AS name + FROM system.lakeflow.pipelines + GROUP BY pipeline_id +) +SELECT + COALESCE(u.usage_metadata.job_name, n.name, '(unnamed)') AS entity, + CASE WHEN u.usage_metadata.dlt_pipeline_id IS NOT NULL THEN 'pipeline' ELSE 'job' END AS kind, + SUM(u.usage_quantity) AS quantity, + u.usage_unit, + SUM(u.usage_quantity * p.pricing.effective_list.default) AS usd, + COUNT(DISTINCT u.usage_date) AS active_days +FROM system.billing.usage u +LEFT JOIN system.billing.list_prices p + ON u.sku_name = p.sku_name + AND u.usage_end_time >= p.price_start_time + AND (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time) +LEFT JOIN pipe_names n + ON n.pipeline_id = u.usage_metadata.dlt_pipeline_id +WHERE u.usage_date >= CURRENT_DATE() - INTERVAL 30 DAYS + AND COALESCE(u.usage_metadata.job_id, u.usage_metadata.dlt_pipeline_id) IS NOT NULL +GROUP BY entity, kind, u.usage_unit +ORDER BY usd DESC diff --git a/reports/sql-diagram/job_spend_plan.svg b/reports/sql-diagram/job_spend_plan.svg new file mode 100644 index 0000000..0b7a0bd --- /dev/null +++ b/reports/sql-diagram/job_spend_plan.svg @@ -0,0 +1,66 @@ + + + +job_spend_plan — logical plan + + + + + + + + + + + +SCAN system.lakeflow.pipelines +The `pipelines` table is a slow-changing +dimension table (SCD2) that contains the … + +AGGREGATE +GROUP BY pipeline_id +MAX_BY(name, change_time) AS name + +SCAN pipe_names + +SCAN system.billing.list_prices +The pricing table gives you access to a +historical log of SKU pricing. A record gets … + +SCAN system.billing.usage +The usage table gives you access to +account-wide billable usage data. The data … + +JOIN 1 · LEFT · p +u.sku_name = p.sku_name +and u.usage_end_time >= p.price_start_time +and (p.price_end_time IS NULL OR +u.usage_end_time < p.price_end_time) + +JOIN 2 · LEFT · n +u.usage_metadata.dlt_pipeline_id = +n.pipeline_id + +WHERE +u.usage_date >= CURRENT_DATE - INTERVAL '30' +DAYS +NOT COALESCE(u.usage_metadata.job_id, +u.usage_metadata.dlt_pipeline_id) IS NULL + +AGGREGATE +GROUP BY entity, kind, u.usage_unit +SUM(u.usage_quantity) AS quantity +SUM(`_a_0`) AS usd +COUNT(`_a_1`) AS active_days + +SORT +usd DESC + +OUTPUT +entity +kind +quantity +usage_unit +usd +active_days + diff --git a/scripts/project_costs.py b/scripts/project_costs.py index b7897fc..6916e74 100644 --- a/scripts/project_costs.py +++ b/scripts/project_costs.py @@ -2,7 +2,7 @@ Show AWS and Databricks spend for the last 30 days, week by week. Usage: uv run python scripts/project_costs.py [--days N] [--profile PROFILE] [--aws-profile PROFILE] -Also writes cost_report/YYYY-MM-DD.md (gitignored) holding the same tables in markdown, with an +Also writes reports/cost/YYYY-MM-DD.md (gitignored) holding the same tables in markdown, with an empty Analysis section for the /project-costs skill to fill in. The script never writes prose — it emits only numbers it pulled, so the report cannot drift from the data. """ @@ -21,7 +21,7 @@ _POLL_TERMINAL = {StatementState.SUCCEEDED, StatementState.FAILED, StatementState.CANCELED, StatementState.CLOSED} -_REPORT_DIR = Path(__file__).resolve().parent.parent / "cost_report" +_REPORT_DIR = Path(__file__).resolve().parent.parent / "reports" / "cost" # Native quantities span DBU / DSU / GB and are not comparable with each other, so the Quantity # column is never totalled. USD is the only figure that spans SKUs and clouds. @@ -465,8 +465,8 @@ def write_markdown( entity_df: pd.DataFrame | None, days: int, ) -> Path: - """Write the data tables to cost_report/YYYY-MM-DD.md, leaving Analysis for the skill.""" - _REPORT_DIR.mkdir(exist_ok=True) + """Write the data tables to reports/cost/YYYY-MM-DD.md, leaving Analysis for the skill.""" + _REPORT_DIR.mkdir(parents=True, exist_ok=True) path = _REPORT_DIR / f"{date.today()}.md" end = date.today() diff --git a/scripts/sql_diagram.py b/scripts/sql_diagram.py new file mode 100644 index 0000000..b326d73 --- /dev/null +++ b/scripts/sql_diagram.py @@ -0,0 +1,594 @@ +""" +Diagram a SQL query, in two modes, from the AST that `sqlglot` parses — never from model +inference, so what is drawn is exactly what the query says. + + --mode plan (default) the query's steps in order: one node per scan, one per *individual* + join with its type and keys, then the filter, aggregate and sort. CTEs appear + as their own sub-pipeline feeding the scan that reads them. + --mode lineage which source column feeds each output column. + +Writes three files to `reports/sql-diagram/`: the `.sql` that was analysed, `.mmd` +(renders on GitHub, diffable) and a standalone `.svg` for linking from prose where no +Mermaid renderer is available. Keeping the query beside the diagram is what makes the diagram +checkable — and regenerable. + +Usage: + python scripts/sql_diagram.py --file query.sql + cat query.sql | python scripts/sql_diagram.py --name job_costs --comments + python scripts/sql_diagram.py --file query.sql --mode lineage --stdout + +In lineage mode, columns that cannot be attributed to a table (unqualified references in a +multi-table join, where resolving them would need the table schemas) are grouped under +`(unqualified)` rather than guessed at. + +`--comments` additionally annotates each source table with its Unity Catalog comment. That is the +only part of this script that touches the network, so it is opt-in; a table without a comment, or +one the profile cannot read, is simply left unannotated rather than described from guesswork. +""" + +import argparse +import re +import sys +from dataclasses import dataclass +from html import escape +from pathlib import Path + +import sqlglot +from sqlglot import exp, planner +from sqlglot.lineage import lineage + +UNQUALIFIED = "(unqualified)" +FONT = "-apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif" +OUT_DIR = Path(__file__).resolve().parent.parent / "reports" / "sql-diagram" + + +def parse_select(sql: str, dialect: str) -> exp.Select: + """ + Parse to the SELECT both diagram modes work from. + + `CREATE TABLE … AS SELECT` and `INSERT … SELECT` wrap a SELECT that is perfectly diagrammable, + so they are unwrapped rather than rejected. Failures exit with one line instead of a traceback: + this is a CLI, and a stack trace tells the reader nothing they can act on. + """ + if not sql.strip(): + raise SystemExit("sql-diagram: no SQL given (empty input)") + try: + parsed = sqlglot.parse_one(sql, dialect=dialect) + except sqlglot.errors.ParseError as err: + raise SystemExit(f"sql-diagram: could not parse the SQL as {dialect} — {err}") from None + select = parsed.find(exp.Select) if parsed else None + if select is None: + raise SystemExit("sql-diagram: no SELECT found in the statement; nothing to diagram") + return select + + +def driving_table(select: exp.Select, dialect: str) -> str | None: + """ + The table in the FROM clause — the one every join hangs off. + + Found by scanning this SELECT's own args for the From node rather than by key: sqlglot names + it `from_` in v30 and `from` earlier, and a missing key returns None silently, which reads as + "no FROM clause" and quietly drops every join edge. Direct args only — `find()` would descend + into a CTE and return its FROM instead. + """ + source = next((v for v in select.args.values() if isinstance(v, exp.From)), None) + return exp.table_name(source.this) if source and isinstance(source.this, exp.Table) else None + + +def output_columns(select: exp.Select) -> list[str]: + """Names of the columns the query projects, in select order.""" + names = [e.alias_or_name for e in select.expressions] + if any(n == "*" for n in names): + raise SystemExit("sql-diagram: SELECT * cannot be traced without table schemas; name the columns") + # sqlglot's lineage() resolves a column by name, so `SELECT a.id, b.id` would trace both to + # a.id and draw a confident, wrong graph. Refusing is the only honest option: a lineage + # diagram that silently attributes a column to the wrong table is worse than no diagram. + duplicates = sorted({n for n in names if names.count(n) > 1}) + if duplicates: + raise SystemExit( + f"sql-diagram: output column name(s) {', '.join(duplicates)} appear more than once; " + "lineage cannot tell them apart — give each one a distinct alias (plan mode is unaffected)" + ) + return names + + +def source_columns(sql: str, dialect: str, column: str) -> set[tuple[str, str]]: + """(table, column) pairs the given output column ultimately reads from.""" + found = set() + for node in lineage(column, sql, dialect=dialect).walk(): + if node.downstream: # only leaves are real source columns + continue + table = UNQUALIFIED + if isinstance(node.source, exp.Table): + table = exp.table_name(node.source) + # strip the table alias qualifier: "u.sku_name" -> "sku_name" + found.add((table, node.name.split(".", 1)[-1])) + return found + + +@dataclass +class Stage: + """One step of the logical plan: a scan, a single join, a filter, an aggregate or a sort.""" + + id: str + kind: str + title: str + details: list[str] + deps: list[str] + source: str = "" # SCAN only: the table name, for an exact comment lookup + + +def build_plan(select: exp.Select, dialect: str) -> list[Stage]: + """ + Flatten sqlglot's logical plan into ordered stages. + + sqlglot models a multi-table join as one n-ary step; this splits it back into "JOIN 1", + "JOIN 2", … in the order the query writes them, which is the order a reader thinks in. The + result is the *logical* plan — what the SQL says. It is not the physical plan: Databricks is + free to reorder joins, broadcast a side, or prune columns, and `EXPLAIN FORMATTED` against a + warehouse is the only thing that shows what actually ran. + """ + stages: list[Stage] = [] + seen: dict[int, str] = {} + + def emit(kind: str, title: str, details: list[str], deps: list[str], source: str = "") -> str: + stages.append(Stage(f"n{len(stages)}", kind, title, details, deps, source)) + return stages[-1].id + + def sql_of(node) -> str: + return node if isinstance(node, str) else node.sql(dialect=dialect) + + def ordered(deps) -> list: + # planner.Step.dependencies is a set, so iteration order changes with the hash seed — + # without this the node numbering churns on every run and the .mmd is not diffable. + return sorted(deps, key=lambda d: (d.name or "", type(d).__name__)) + + def visit(step) -> str: + if id(step) in seen: + return seen[id(step)] + + if isinstance(step, planner.Join): + by_name = {dep.name: visit(dep) for dep in ordered(step.dependencies)} + current = by_name[step.name] # the FROM table; every join hangs off it in turn + for i, (source, ctx) in enumerate(step.joins.items(), 1): + keys = [ + f"{sql_of(a)} = {sql_of(b)}" for a, b in zip(ctx.get("source_key") or [], ctx.get("join_key") or []) + ] + # With a complex ON, sqlglot leaves source_key empty and puts everything in the + # residual, so "and" cannot be tied to the residual list — it belongs to every + # predicate after the first, wherever it came from. + predicates = keys + _and_parts(ctx.get("condition"), dialect) + details = predicates[:1] + [f"and {part}" for part in predicates[1:]] + # A join with neither keys nor a predicate is a cross product. Labelling that + # "INNER" would hide exactly the unconstrained join this diagram exists to expose. + side = (ctx.get("side") or "").upper() or ("CROSS" if not details else "INNER") + current = emit("JOIN", f"JOIN {i} · {side} · {source}", details, [current]) + stages[-1].deps.append(by_name[source]) + if step.condition: + current = emit("FILTER", "WHERE", _and_parts(step.condition, dialect), [current]) + seen[id(step)] = current + return current + + deps = [visit(dep) for dep in ordered(step.dependencies)] + if isinstance(step, planner.Scan): + table = exp.table_name(step.source) if isinstance(step.source, exp.Table) else step.name + node = emit("SCAN", f"SCAN {table}", [], deps, source=table) + elif isinstance(step, planner.Aggregate): + group = [sql_of(g) for g in (step.group or {}).values()] + details = ([f"GROUP BY {', '.join(group)}"] if group else []) + [sql_of(a) for a in step.aggregations] + node = emit("AGGREGATE", "AGGREGATE", details, deps) + elif isinstance(step, planner.Sort): + node = emit("SORT", "SORT", [sql_of(k) for k in step.key], deps) + else: + node = emit("STEP", type(step).__name__.upper(), [], deps) + + seen[id(step)] = node + return node + + try: + root = planner.Plan(select).root + except Exception as err: # sqlglot's planner raises assorted types on shapes it cannot model + raise SystemExit(f"sql-diagram: could not build a plan for this query ({type(err).__name__}: {err})") from None + last = visit(root) + emit("OUTPUT", "OUTPUT", [e.alias_or_name for e in root.projections], [last]) + return stages + + +def _and_parts(condition, dialect: str) -> list[str]: + """ + Split a conjunction into one string per top-level predicate. + + Done on the AST, never on generated text: splitting text on " AND " tears apart string + literals containing the word, and counting parentheses miscounts when a literal holds one. + Filtering `TRUE` here also removes the `TRUE AND …` padding sqlglot puts on a join's residual + predicate — a regex for that would delete a legitimate `TRUE AND` written inside a predicate. + """ + if condition is None: + return [] + if isinstance(condition, str): + try: + condition = sqlglot.parse_one(condition, dialect=dialect) + except sqlglot.errors.ParseError: + return [condition] + parts = condition.flatten() if isinstance(condition, exp.And) else [condition] + rendered = [] + for part in parts: + if isinstance(part, exp.Boolean) and part.this is True: + continue + text = part.sql(dialect=dialect) + # An OR shown as a bare conjunct reads as "(and X) OR Y" once the lines are prefixed — + # the opposite grouping. Parenthesise it so the line cannot be misread. + if isinstance(part, exp.Or) and not text.startswith("("): + text = f"({text})" + rendered.append(text) + return rendered + + +def _shorten(text: str, limit: int) -> str: + """Truncate on a word boundary so a long UC comment cannot blow out the layout.""" + text = " ".join(text.split()) + if len(text) <= limit: + return text + return text[:limit].rsplit(" ", 1)[0] + "…" + + +def _mermaid_safe(text: str) -> str: + """Quoted Mermaid labels cannot contain a raw double quote.""" + return text.replace('"', "#quot;") + + +def _wrap(text: str, width: int, max_lines: int) -> list[str]: + """Greedy wrap for SVG text, which has no automatic line breaking.""" + words = text.split() + lines, current = [], [] + for word in words: + if current and len(" ".join(current + [word])) > width: + lines.append(" ".join(current)) + current = [] + if len(lines) == max_lines: + break + current.append(word) + if current and len(lines) < max_lines: + lines.append(" ".join(current)) + if lines and len(" ".join(lines).split()) < len(words): + lines[-1] = lines[-1].rstrip(".,;") + " …" # say it was cut rather than end mid-sentence + return lines + + +def extract_joins(select: exp.Select, dialect: str) -> dict[str, str]: + """Source table -> the join that brings it in, e.g. "LEFT JOIN ON a.id = b.id".""" + joins = {} + for join in select.find_all(exp.Join): + if not isinstance(join.this, exp.Table): # subquery / derived table: no single name to key on + continue + kind = " ".join(filter(None, [join.side, join.kind, "JOIN"])) + on = join.args.get("on") + joins[exp.table_name(join.this)] = f"{kind} ON {on.sql(dialect=dialect)}" if on else kind + return joins + + +def fetch_comments(tables: list[str], profile: str) -> dict[str, str]: + """Unity Catalog comment per table. Any table that cannot be read is left out, not guessed.""" + from databricks.sdk import WorkspaceClient + + client = WorkspaceClient(profile=profile) + comments = {} + for table in tables: + if table.count(".") != 2: # not a three-level UC name — nothing to look up + continue + try: + comment = client.tables.get(table).comment + except Exception as err: + print(f"note: no comment for {table} ({type(err).__name__})", file=sys.stderr) + continue + if comment: + # UC comments are markdown; the diagram renders plain text, so unwrap [label](url) + comments[table] = re.sub(r"\[([^\]]+)\]\([^)]+\)", r"\1", comment).strip() + return comments + + +def build_graph( + sql: str, select: exp.Select, dialect: str +) -> tuple[list[tuple[str, set[tuple[str, str]]]], dict[str, list[str]]]: + """ + (output column, its source columns) in select order, plus source table -> its columns. + + Edges are a list, not a dict keyed by column name: `SELECT a.id, b.id` projects the name `id` + twice, and keying by name silently collapses the two — dropping a whole source table out of a + diagram whose only job is to show where columns come from. + """ + edges = [(col, source_columns(sql, dialect, col)) for col in output_columns(select)] + tables: dict[str, list[str]] = {} + for table, column in sorted({src for _, sources in edges for src in sources}): + tables.setdefault(table, []).append(column) + # A table can be joined purely to filter, projecting nothing. It still belongs on the diagram: + # without it the join edge has no node to leave from and would be dropped silently. + driving = driving_table(select, dialect) + if driving and driving not in tables: + tables[driving] = [] + return edges, tables + + +def to_mermaid( + edges: list[tuple[str, set[tuple[str, str]]]], + tables: dict[str, list[str]], + joins: dict[str, str] | None = None, + comments: dict[str, str] | None = None, + driving: str | None = None, +) -> str: + """Build a Mermaid flowchart of the query's column-level lineage.""" + joins, comments = joins or {}, comments or {} + ids = {(t, c): f"s{i}" for i, (t, c) in enumerate((t, c) for t, cols in tables.items() for c in cols)} + node = {table: re.sub(r"\W", "_", table) for table in tables} + + lines = ["flowchart LR"] + for table, columns in tables.items(): + label = table + if table in comments: + label += f"
{_mermaid_safe(_shorten(comments[table], 90))}" + # subgraph ids must be bare identifiers; the quoted label keeps the real name + lines.append(f' subgraph {node[table]}["{label}"]') + lines += [f' {ids[(table, column)]}["{column}"]' for column in columns] + lines.append(" end") + + lines.append(' subgraph out["output"]') + lines += [f' o{i}["{col}"]' for i, (col, _) in enumerate(edges)] + lines.append(" end") + + for i, (_, sources) in enumerate(edges): + lines += [f" {ids[src]} --> o{i}" for src in sorted(sources)] + + # The join is a relationship between tables, not between columns, so it is drawn as a dotted + # edge between the subgraphs — visually distinct from the solid column-lineage arrows. The + # driving table comes from the FROM clause: deducing it by elimination loses every join edge + # when the driving table happens to project no columns of its own. + for table, clause in joins.items(): + if table in node and driving and driving in node: + lines.append(f' {node[driving]} -. "{_mermaid_safe(clause)}" .-> {node[table]}') + return "\n".join(lines) + + +_SHAPES = {"SCAN": ("[(", ")]"), "JOIN": ("{{", "}}"), "OUTPUT": ("([", "])")} + + +def plan_to_mermaid(stages: list[Stage], comments: dict[str, str] | None = None) -> str: + """Mermaid flowchart of the logical plan, one node per stage.""" + comments = comments or {} + lines = ["flowchart LR"] + for stage in stages: + label = f"{_mermaid_safe(stage.title)}" + note = comments.get(stage.source) if stage.kind == "SCAN" else None + if note: + label += f"
{_mermaid_safe(_shorten(note, 70))}" + for detail in stage.details: + label += f"
{_mermaid_safe(_shorten(detail, 70))}" + open_, close = _SHAPES.get(stage.kind, ("[", "]")) + lines.append(f' {stage.id}{open_}"{label}"{close}') + for stage in stages: + lines += [f" {dep} --> {stage.id}" for dep in stage.deps] + return "\n".join(lines) + + +def plan_to_svg(stages: list[Stage], title: str, comments: dict[str, str] | None = None) -> str: + """ + Top-down rendering: each row is one depth of the plan, so the query reads scans-first, + output-last. Laid out vertically rather than left-to-right because a plan deep enough to be + worth drawing (this one is 8 levels) runs to several thousand pixels wide on one line. + """ + comments = comments or {} + box_w, col_gap, row_gap, lead, head = 250, 26, 34, 13, 26 + + depth: dict[str, int] = {} + for stage in stages: # stages are emitted dependency-first, so one pass suffices + depth[stage.id] = max((depth[d] + 1 for d in stage.deps), default=0) + + text: dict[str, list[str]] = {} + for stage in stages: + note = comments.get(stage.source) if stage.kind == "SCAN" else None + body = _wrap(note, 44, 2) if note else [] + for detail in stage.details: + body += _wrap(detail, 44, 2) + text[stage.id] = body + + rows: dict[int, list[Stage]] = {} + for stage in stages: + rows.setdefault(depth[stage.id], []).append(stage) + + widest = max(len(row) for row in rows.values()) + width = 24 + widest * (box_w + col_gap) - col_gap + 24 + + pos: dict[str, tuple[float, float]] = {} + y = 60 + for level in sorted(rows): + row = rows[level] + span = len(row) * (box_w + col_gap) - col_gap + x = (width - span) / 2 # centre each row, so the trunk of the plan stays vertical + for stage in row: + pos[stage.id] = (x, y) + x += box_w + col_gap + y += head + lead * max(len(text[s.id]) for s in row) + row_gap + height = y - row_gap + 24 + + curves, boxes = [], [] + for stage in stages: + x, y = pos[stage.id] + h = head + lead * len(text[stage.id]) + boxes.append(f'') + boxes.append(f'{escape(stage.title)}') + for i, line in enumerate(text[stage.id]): + boxes.append(f'{escape(line)}') + for dep in stage.deps: + dx, dy = pos[dep] + a = dy + head + lead * len(text[dep]) # bottom edge of the dependency + ax, bx = dx + box_w / 2, x + box_w / 2 + curves.append(f'') + + return "\n".join( + [ + f'', + "", + f'', + f'{escape(title)} — logical plan', + *curves, + *boxes, + "", + ] + ) + + +def to_svg( + edges: dict[str, set[tuple[str, str]]], + tables: dict[str, list[str]], + title: str, + joins: dict[str, str] | None = None, + comments: dict[str, str] | None = None, +) -> str: + """ + Render the same graph as a standalone SVG. + + Written by hand rather than shelled out to `mmdc`, which would drag a Node toolchain and a + headless Chromium into a Python repo — the same trade `scripts/star_history.py` makes. The + layout is a fixed two-column bipartite one, which is all a lineage graph needs. + """ + joins, comments = joins or {}, comments or {} + row, box_h, left_w, right_w, gap = 26, 20, 340, 210, 230 + lead = 14 # line height for the wrapped annotation text + width = 24 + left_w + gap + right_w + 24 + src_y, out_y = {}, {} + + y = 60 + rows = [] + for table, columns in tables.items(): + rows.append((y, f'{escape(table)}')) + y += row + for line in _wrap(comments.get(table, ""), 62, 2): + rows.append((y, f'{escape(line)}')) + y += lead + # no join glyph: U+2A1D is missing from enough system fonts to render as tofu + for line in _wrap(joins.get(table, ""), 58, 2): + rows.append((y, f'{escape(line)}')) + y += lead + if table in comments or table in joins: + y += 6 + for column in columns: + src_y[(table, column)] = y + rows.append((y, f'')) + rows.append((y, f'{escape(column)}')) + y += row + y += 8 # breathing room between tables + + oy = 60 + row # align the first output box with the first source box + for col in edges: + out_y[col] = oy + x = 24 + left_w + gap + # single class, not "node out": some SVG renderers do not resolve a multi-class cascade + rows.append((oy, f'')) + rows.append((oy, f'{escape(col)}')) + oy += row + + x1, x2 = 24 + left_w, 24 + left_w + gap + curves = [] + for col, sources in edges.items(): + for src in sorted(sources): + a, b = src_y[src] + box_h / 2, out_y[col] + box_h / 2 + curves.append(f'') + + height = max(y, oy) + 16 + return "\n".join( + [ + f'', + # One file for both themes: GitHub renders SVGs through , where the media query + # still resolves against the reader's colour scheme. + "", + f'', + f'{escape(title)}', + *curves, # behind the boxes, so a curve never cuts across a label + *[markup for _, markup in rows], + "", + ] + ) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--file", help="path to a .sql file (default: read from stdin)") + parser.add_argument("--dialect", default="databricks", help="sqlglot dialect (default: databricks)") + parser.add_argument("--name", help="output basename (default: the input filename, or 'query' on stdin)") + parser.add_argument("--stdout", action="store_true", help="print the Mermaid source instead of writing files") + parser.add_argument("--comments", action="store_true", help="annotate tables with their Unity Catalog comment") + parser.add_argument("--profile", default="dev", help="Databricks profile for --comments (default: dev)") + parser.add_argument( + "--mode", + choices=["plan", "lineage"], + default="plan", + help="plan: the query's steps (scans, each join, filter, aggregate, sort). " + "lineage: which source column feeds each output column.", + ) + args = parser.parse_args() + + sql = open(args.file).read() if args.file else sys.stdin.read() + select = parse_select(sql, args.dialect) + + # Each mode builds only what it renders. Sharing the work looked tidy but meant a plan run + # inherited lineage's constraints — a SELECT * query the plan draws fine used to die on the + # lineage tracer that never ran. + stages: list[Stage] = [] + edges: list[tuple[str, set[tuple[str, str]]]] = [] + tables: dict[str, list[str]] = {} + if args.mode == "plan": + stages = build_plan(select, args.dialect) + tables = {stage.source: [] for stage in stages if stage.kind == "SCAN" and stage.source} + else: + edges, tables = build_graph(sql, select, args.dialect) + + comments = fetch_comments(list(tables), args.profile) if args.comments else {} + + if args.mode == "plan": + mermaid = plan_to_mermaid(stages, comments) + else: + joins = extract_joins(select, args.dialect) + mermaid = to_mermaid(edges, tables, joins, comments, driving_table(select, args.dialect)) + + if args.stdout: + print(mermaid) + return + + name = args.name or (Path(args.file).stem if args.file else "query") + OUT_DIR.mkdir(parents=True, exist_ok=True) + # The query is written out verbatim alongside the diagram: a diagram without the SQL it came + # from cannot be checked or regenerated, and the source is often an f-string in a .py file + # that no longer reads the same once its placeholders are filled in. + (OUT_DIR / f"{name}.sql").write_text(sql.strip() + "\n") + (OUT_DIR / f"{name}.mmd").write_text(mermaid + "\n") + svg = plan_to_svg(stages, name, comments) if args.mode == "plan" else to_svg(edges, tables, name, joins, comments) + (OUT_DIR / f"{name}.svg").write_text(svg + "\n") + print(f"Wrote {OUT_DIR / name}.sql, .mmd and .svg") + + +if __name__ == "__main__": + main() diff --git a/specs/CHANGELOG.md b/specs/CHANGELOG.md index 863f80e..78466f1 100644 --- a/specs/CHANGELOG.md +++ b/specs/CHANGELOG.md @@ -2,6 +2,12 @@ --- +## [#50](https://github.com/andre-salvati/databricks-template/pull/50) · 2026-07-22 · feat: SQL query-plan diagrams, and a reports/ folder for generated artifacts + +Added `make sql-diagram` / `/sql-diagram`, which parses a query with `sqlglot` and draws either its execution steps (default) or its column-level lineage, emitting the analysed `.sql` beside a Mermaid `.mmd` and a hand-written `.svg` — no `mmdc`, so no Node toolchain in a Python repo. `sqlglot` models a multi-table join as one n-ary step, so the plan builder splits it back into `JOIN 1`, `JOIN 2`, … in written order, which is what makes an under-constrained join predicate visible — the bug class behind the 3.5× fan-out fixed in #47. Table annotations come from Unity Catalog comments through the `dev` profile (the MCP service principal lacks `USE SCHEMA` on `system.billing`), opt-in as the only network call. Round-tripping the emitted SQL exposed that `planner.Step.dependencies` is a set, so node numbering followed the process hash seed and the `.mmd` churned on every run; dependencies are now walked in a stable order. Generated artifacts also moved under `reports/`: `coverage`, `cost` and `sql-diagram`. + +--- + ## [#49](https://github.com/andre-salvati/databricks-template/pull/49) · 2026-07-21 · feat: attribute Databricks spend to jobs and pipelines Added a fourth table to `make project-costs`, Databricks by Job / Pipeline, attributing spend to the job or SDP pipeline that incurred it. Job names come off `usage_metadata.job_name`; only pipelines need a lookup, pre-collapsed per `pipeline_id` because joining those slowly-changing dimensions to usage otherwise fans each row out per revision and overcounted one job 21×. Only usage carrying a `job_id` or `dlt_pipeline_id` is attributable, so it breaks down scheduled work, not the whole bill. @@ -16,7 +22,7 @@ Replaced the README star-history chart with `scripts/star_history.py` + `make st ## [#47](https://github.com/andre-salvati/databricks-template/pull/47) · 2026-07-16 · feat: weekly USD cost pivots + generated markdown cost report -Reworked `scripts/project_costs.py` from per-day dumps into three tables — AWS by week × service, Databricks by week × SKU, and a combined by-service rollup carrying `Quantity`/`Unit`/`USD` — monetizing Databricks usage through a date-scoped `system.billing.list_prices` join on `pricing.effective_list.default`, which shows Databricks is ~99.5% of project spend ($33.91 of $34.06 at list over 30 days) and that AWS is only ever a proxy for job activity since serverless SKUs bill compute inside the DBU rate. Two properties of that join are load-bearing and easy to get wrong: it must be date-scoped because `list_prices` is a slowly-changing dimension (joining on `sku_name` alone fans out 3.5× on `JOBS_SERVERLESS_COMPUTE`), and it must be a LEFT join so an unpriced SKU shows blank rather than vanishing; per-day `ROUND()` was also dropped, since rounding then summing zeroed sub-cent SKUs like internet egress. The script now writes `cost_report/YYYY-MM-DD.md` — gitignored, with this first report force-added and linked from the README's Features list as a worked example — holding every table in markdown plus an `## Analysis` section that the `/project-costs` skill fills in, so numbers are never re-transcribed by hand. +Reworked `scripts/project_costs.py` from per-day dumps into three tables — AWS by week × service, Databricks by week × SKU, and a combined by-service rollup carrying `Quantity`/`Unit`/`USD` — monetizing Databricks usage through a date-scoped `system.billing.list_prices` join on `pricing.effective_list.default`, which shows Databricks is ~99.5% of project spend ($33.91 of $34.06 at list over 30 days). Two properties of that join are load-bearing and easy to get wrong: it must be date-scoped because `list_prices` is a slowly-changing dimension (joining on `sku_name` alone fans out 3.5× on `JOBS_SERVERLESS_COMPUTE`), and it must be a LEFT join so an unpriced SKU shows blank rather than vanishing; per-day `ROUND()` was also dropped, since rounding then summing zeroed sub-cent SKUs like internet egress. The script now writes a gitignored `cost_report/YYYY-MM-DD.md` holding every table plus an `## Analysis` section that the `/project-costs` skill fills in, so numbers are never re-transcribed by hand. --- @@ -28,13 +34,13 @@ Added a `## Code style: PySpark transformation chains` section to `specs/archite ## [#45](https://github.com/andre-salvati/databricks-template/pull/45) · 2026-07-03 · docs: add workflow spec + license, move dashboard docs to data-model -Added `specs/workflow.md` centralizing the development lifecycle (plan → ask-about-branch → hold-commits → PR), a research-backed PR description standard materialized as `.github/PULL_REQUEST_TEMPLATE.md`, and a production-table impact check with a schema-change alert table (add/remove/rename/type/cluster-key → risk → remediation, tied to the `overwriteSchema=false` guard); it also absorbs the former `specs/test-plan.md`, which was deleted. Consolidated the scattered dashboard documentation (README screenshot + `CLAUDE.md` deploy invariants + the latest-name binding) into a new `## Dashboard` section in `specs/data-model.md`, with the README section now linking to it. Added an Apache-2.0 `LICENSE` + `NOTICE`, a license badge and `## License` section in the README, and the SPDX `license` field in `pyproject.toml`, updating all cross-references (`README.md`, `CLAUDE.md`, `specs/README.md`, `specs/architecture.md`). +Added `specs/workflow.md` centralizing the development lifecycle (plan → ask-about-branch → hold-commits → PR), a research-backed PR description standard materialized as `.github/PULL_REQUEST_TEMPLATE.md`, and a production-table impact check with a schema-change alert table (add/remove/rename/type/cluster-key → risk → remediation, tied to the `overwriteSchema=false` guard); it also absorbs the former `specs/test-plan.md`, which was deleted. Consolidated the scattered dashboard documentation (README screenshot + `CLAUDE.md` deploy invariants + the latest-name binding) into a new `## Dashboard` section in `specs/data-model.md`, with the README section now linking to it. Added an Apache-2.0 `LICENSE` + `NOTICE`, a license badge and `## License` section in the README, and the SPDX `license` field in `pyproject.toml`, updating all cross-references. --- ## [#43](https://github.com/andre-salvati/databricks-template/pull/43) · 2026-06-23 · feat: freeze product_name instead of line_revenue -Removed the synthetic `line_revenue`/`unit_price_at_sale` columns — gold `total_value` is now `SUM(item_total)` (the line value the source already freezes on the order) — and re-pointed the silver insert-only-MERGE / streaming-table freeze at the mutable `product_name`, which the seed now changes by renaming 2 products per run (`Product N` → `Product N.k`); `unit_price` stays as a static attribute. The AI/BI "by product" chart and Product filter both identify a product by its latest name (consolidating by `product_id`, one line per physical product across renames, and filtering shows the full pre-/post-rename history); the frozen historical names remain in `report.order_agg` for audit. Also removed the batch `job1`'s standalone prod schedule (the SDP pipeline already had none) so `job1_prod_integration` is the single prod trigger orchestrating seed → batch + SDP, fixed `sdk_drop_tables.py` to fall back to `DROP TABLE` when a warehouse's parser rejects `DROP STREAMING TABLE`, added a `git_commit` (`${bundle.git.commit}`) deploy tag on every job so a deployed environment's exact commit is identifiable from `jobs list`, and updated unit/integration tests, schemas, the SDP pipeline, and docs. +Removed the synthetic `line_revenue`/`unit_price_at_sale` columns — gold `total_value` is now `SUM(item_total)`, the line value the source already freezes on the order — and re-pointed the silver insert-only-MERGE / streaming-table freeze at the mutable `product_name`, which the seed now changes by renaming 2 products per run (`Product N` → `Product N.k`); `unit_price` stays a static attribute. The AI/BI "by product" chart and Product filter both identify a product by its latest name (consolidating by `product_id`, one line per physical product across renames), while the frozen historical names remain in `report.order_agg` for audit. Also removed the batch `job1`'s standalone prod schedule so `job1_prod_integration` is the single prod trigger, fixed `sdk_drop_tables.py` to fall back to `DROP TABLE` when a warehouse's parser rejects `DROP STREAMING TABLE`, and added a `git_commit` (`${bundle.git.commit}`) deploy tag so a deployed environment's exact commit is identifiable from `jobs list`. --- @@ -44,91 +50,69 @@ Renamed the image-only `docs/` folder to `assets/` (it held no prose, only scree --- - ## [#41](https://github.com/andre-salvati/databricks-template/pull/41) · 2026-06-12 · feat: raw cost DataFrames + by-service/SKU aggregation in project_costs.py -`scripts/project_costs.py` now prints the raw per-day/service (AWS) and per-day/SKU (Databricks) DataFrames before each formatted daily table, plus an "AWS Costs by Service" rollup and a "Databricks Costs by SKU" rollup (quantity grouped by SKU and unit, since DBU/DSU/GB are not summable; NULL units kept, sorted within unit, AWS estimates flagged). Defaulted the `make project-costs` target to `--aws-profile costs` (the dedicated read-only Cost Explorer user), where `aws-profile=` now falls back to the default credential chain instead of hard-failing argparse. Also folded in code-review fixes: `scripts/sdk_drop_tables.py` now issues the kind-matched `DROP MATERIALIZED VIEW` / `DROP STREAMING TABLE` instead of a blind `DROP TABLE` (which aborted `make drop` mid-loop once the SDP pipeline had materialized its views/streaming tables), and `seed_sources.py` centralizes the product-category formula so the price-update path can't drift from the initial seed. +`scripts/project_costs.py` now prints the raw per-day/service (AWS) and per-day/SKU (Databricks) DataFrames before each formatted daily table, plus an "AWS Costs by Service" rollup and a "Databricks Costs by SKU" rollup (quantity grouped by SKU and unit, since DBU/DSU/GB are not summable; NULL units kept, sorted within unit, AWS estimates flagged). Defaulted the `make project-costs` target to `--aws-profile costs` (the dedicated read-only Cost Explorer user), where `aws-profile=` now falls back to the default credential chain instead of hard-failing argparse. Also folded in code-review fixes: `scripts/sdk_drop_tables.py` now issues the kind-matched `DROP MATERIALIZED VIEW` / `DROP STREAMING TABLE` instead of a blind `DROP TABLE` (which aborted `make drop` mid-loop once the SDP pipeline had materialized its views), and `seed_sources.py` centralizes the product-category formula so the price-update path can't drift from the initial seed. --- ## [#40](https://github.com/andre-salvati/databricks-template/pull/40) · 2026-06-11 · docs: reorganize specs into specs/ folder, slim README and CLAUDE.md -Extracted the deep technical documentation out of `README.md` and `CLAUDE.md` into a dedicated `specs/` folder (`architecture.md`, `data-model.md`, `test-plan.md`, an index, and `CHANGELOG.md` itself), leaving the README a landing page and `CLAUDE.md` working rules plus load-bearing invariants and pointers. -Converted the medallion diagram to inline Mermaid (GitHub-rendered, diffable) and deleted the superseded `docs/medallion_data_flow.png`; the CI/CD diagram stays a draw.io export (`docs/ci_cd.drawio` → `docs/ci_cd.png`) because the layout it needs isn't expressible in GitHub-rendered (dagre) Mermaid. -Added `specs/**` to the `onpush.yml` `paths-ignore` so docs-only changes don't trigger a CI run. +Extracted the deep technical documentation out of `README.md` and `CLAUDE.md` into a dedicated `specs/` folder (`architecture.md`, `data-model.md`, `test-plan.md`, an index, and `CHANGELOG.md` itself), leaving the README a landing page and `CLAUDE.md` working rules plus load-bearing invariants and pointers. Converted the medallion diagram to inline Mermaid (GitHub-rendered, diffable) and deleted the superseded `docs/medallion_data_flow.png`; the CI/CD diagram stays a draw.io export (`docs/ci_cd.drawio` → `docs/ci_cd.png`) because the layout it needs isn't expressible in GitHub-rendered (dagre) Mermaid. Added `specs/**` to the `onpush.yml` `paths-ignore` so docs-only changes don't trigger a CI run. --- ## [#39](https://github.com/andre-salvati/databricks-template/pull/39) · 2026-06-11 · feat: add --aws-profile flag to project_costs.py -Added an `--aws-profile` flag to `scripts/project_costs.py` so the AWS Cost Explorer query can target a dedicated AWS CLI profile (`aws_daily_costs` appends `--profile` only when set) instead of relying solely on the default credential chain. -This unblocks headless runs and lets the script use a scoped read-only Cost Explorer user rather than the expiry-prone browser `aws login` session that this workspace defaults to. -The flag defaults to `None`, so behavior is unchanged for anyone who does not pass it. +Added an `--aws-profile` flag to `scripts/project_costs.py` so the AWS Cost Explorer query can target a dedicated AWS CLI profile (`aws_daily_costs` appends `--profile` only when set) instead of relying solely on the default credential chain. This unblocks headless runs and lets the script use a scoped read-only Cost Explorer user rather than the expiry-prone browser `aws login` session that this workspace defaults to. The flag defaults to `None`, so behavior is unchanged for anyone who does not pass it. --- ## [#38](https://github.com/andre-salvati/databricks-template/pull/38) · 2026-06-11 · feat: move category to the product dimension, commit dashboard JSON -Moved product category off the order fact onto the product dimension — `external_source.product` now carries `category_id`/`category_name` and silver pulls them from the product join (both `job1` batch and `job1_sdp` paths) instead of deriving `"Category " + id` inline on the order, so a category rename flows correctly to gold and the dashboard. -Committed `resources/orders_dashboard.lvdash.json` as the canonical dashboard definition (dropping the `_build_dashboard_json` generator); since DABs 0.298.0 does not substitute `${var.catalog}` inside `.lvdash.json` content, the generator now writes a gitignored deploy copy with the catalog resolved at deploy time. -Verified end-to-end across unit, dev, staging, and prod; since the schema change is incompatible under `overwriteSchema=false`, replaced `make truncate`/`sdk_truncate_tables.py` (TRUNCATE, schema-preserving) with `make drop`/`sdk_drop_tables.py` (DROP all medallion tables so the next run recreates them with the new schema) and renamed the `make test` target to `make unit-test`. +Moved product category off the order fact onto the product dimension — `external_source.product` now carries `category_id`/`category_name` and silver pulls them from the product join (both `job1` batch and `job1_sdp` paths) instead of deriving `"Category " + id` inline on the order, so a category rename flows correctly to gold and the dashboard. Committed `resources/orders_dashboard.lvdash.json` as the canonical dashboard definition (dropping the `_build_dashboard_json` generator); since DABs 0.298.0 does not substitute `${var.catalog}` inside `.lvdash.json` content, the generator now writes a gitignored deploy copy with the catalog resolved at deploy time. Verified end-to-end across unit, dev, staging, and prod; since the schema change is incompatible under `overwriteSchema=false`, `make truncate`/`sdk_truncate_tables.py` was replaced with `make drop`/`sdk_drop_tables.py` (DROP all medallion tables so the next run recreates them), and `make test` was renamed to `make unit-test`. --- ## [#36](https://github.com/andre-salvati/databricks-template/pull/36) · 2026-06-09 · feat: price-freeze incremental silver, liquid clustering, readable labels -Added a mutable `external_source.product` dimension (daily `unit_price` MERGE) and frozen `line_revenue` in silver (`item_quantity × unit_price_at_sale`) so a later price change never restates booked revenue; `job1` freezes via an insert-only `MERGE`, `job1_sdp` via a streaming table + stream-static join — both carrying `product_name` and `category_name` labels through to gold. -Applied Delta liquid clustering to all accumulating tables via `BaseTask.cluster_by` (batch) and the `cluster_by=` decorator (SDP); full-overwrite `raw.*` are intentionally left unclustered, and `integration_setup.py` now matches the prod clustering layout set by `seed_sources`. -Added a medallion data flow diagram (`docs/medallion_data_flow.png`, also in README) showing both pipeline paths with write modes and clustering keys per table. +Added a mutable `external_source.product` dimension (daily `unit_price` MERGE) and frozen `line_revenue` in silver (`item_quantity × unit_price_at_sale`) so a later price change never restates booked revenue; `job1` freezes via an insert-only `MERGE`, `job1_sdp` via a streaming table + stream-static join — both carrying `product_name` and `category_name` labels through to gold. Applied Delta liquid clustering to all accumulating tables via `BaseTask.cluster_by` (batch) and the `cluster_by=` decorator (SDP); full-overwrite `raw.*` are intentionally left unclustered, and `integration_setup.py` now matches the prod clustering layout set by `seed_sources`. Added a medallion data flow diagram (`docs/medallion_data_flow.png`, also in README) showing both pipeline paths with write modes and clustering keys per table. --- ## [#35](https://github.com/andre-salvati/databricks-template/pull/35) · 2026-06-08 · feat: add /project-costs Claude command and make target -Added `scripts/project_costs.py`, which queries AWS Cost Explorer (daily, last 30 days) and the Databricks `system.billing.usage` system table via the SDK and prints two formatted cost tables, exposed through a `make project-costs` runner. -Added a `/project-costs` Claude slash command (`.claude/commands/project-costs.md`) that runs the script and analyzes the output for anomalies, spikes, period comparisons, and cross-cloud S3/egress-vs-DBU correlation; `.gitignore` now tracks `.claude/commands/` so project slash commands are shared with the team while personal settings stay local. -Hardened the script against credential leakage and runtime failures: sanitized AWS/SDK error messages (which can embed caller ARNs or the workspace URL), `FileNotFoundError` and JSON-decode guards, `StatementState.CLOSED` in the poll-exit set, a null-guard on `stmt.status.error` for canceled statements, and `--days >= 1` validation. +Added `scripts/project_costs.py`, which queries AWS Cost Explorer (daily, last 30 days) and the Databricks `system.billing.usage` system table via the SDK and prints two formatted cost tables, exposed through a `make project-costs` runner. Added a `/project-costs` Claude slash command (`.claude/commands/project-costs.md`) that runs the script and analyzes the output for anomalies, spikes, period comparisons, and cross-cloud S3/egress-vs-DBU correlation; `.gitignore` now tracks `.claude/commands/` so project slash commands are shared with the team while personal settings stay local. Hardened the script against credential leakage and runtime failures: sanitized AWS/SDK error messages (which can embed caller ARNs or the workspace URL), `FileNotFoundError` and JSON-decode guards, `StatementState.CLOSED` in the poll-exit set, a null-guard on `stmt.status.error` for canceled statements, and `--days >= 1` validation. --- ## [#34](https://github.com/andre-salvati/databricks-template/pull/34) · 2026-06-05 · feat: standardize silver/gold field names, fix dashboard KPIs, add total_orders -Dropped `ds_kpi` from the dashboard — all three KPI counters (Total Value, Total Orders, Number of Customers) now bind to `ds_orders` with aggregate expressions so all five filters update them; added a third KPI tile for Total Orders (`COUNT DISTINCT order_id`). -Standardized field names across silver (`curated.order_enriched`) and gold (`report.order_agg`) following four rules: `{entity}_id` suffix, entity-qualified names, `item_*` prefix for item-level fields, no abbreviations; `date` is now cast to `DateType` in silver. -Added `order_enriched_schema` and `order_agg_schema` to `commonSchemas.py` as canonical schemas for silver and gold; all tests and the integration validator import from there instead of inlining definitions. +Dropped `ds_kpi` from the dashboard — all three KPI counters (Total Value, Total Orders, Number of Customers) now bind to `ds_orders` with aggregate expressions so all five filters update them; added a third KPI tile for Total Orders (`COUNT DISTINCT order_id`). Standardized field names across silver (`curated.order_enriched`) and gold (`report.order_agg`) following four rules: `{entity}_id` suffix, entity-qualified names, `item_*` prefix for item-level fields, no abbreviations; `date` is now cast to `DateType` in silver. Added `order_enriched_schema` and `order_agg_schema` to `commonSchemas.py` as canonical schemas for silver and gold; all tests and the integration validator import from there instead of inlining definitions. --- ## [#33](https://github.com/andre-salvati/databricks-template/pull/33) · 2026-06-04 · feat: AI/BI dashboard, country in gold layer, randomized seed data -Added `country` to `curated.order_enriched` and `report.order_agg` (and SDP equivalents) so the gold layer carries the full customer dimension needed for country-based reporting. -Added an AI/BI (Lakeview) dashboard with three line charts (total value by date × country, product, and category) and a global filter page; dashboard JSON is generated by `sdk_generate_template_job.py` at deploy time with the target catalog embedded and is gitignored. -Improved seed data chart visibility with a non-uniform country distribution and `total_item` scaling with `prod_category_id` (category × $15 base + $10 noise), producing a ~6× spread across categories. +Added `country` to `curated.order_enriched` and `report.order_agg` (and SDP equivalents) so the gold layer carries the full customer dimension needed for country-based reporting. Added an AI/BI (Lakeview) dashboard with three line charts (total value by date × country, product, and category) and a global filter page; dashboard JSON is generated by `sdk_generate_template_job.py` at deploy time with the target catalog embedded and is gitignored. Improved seed data chart visibility with a non-uniform country distribution and `total_item` scaling with `prod_category_id` (category × $15 base + $10 noise), producing a ~6× spread across categories. --- ## [#31](https://github.com/andre-salvati/databricks-template/pull/31) · 2026-06-03 · feat: product dimensions, seed enrichment, truncate script, prod schedule -Added `product_id` (100 distinct) and `prod_category_id` (10 distinct) to the order schema and propagated them through extract → enrich → aggregate → SDP transforms; `report.order_agg` now groups by `name`, `date`, `product_id`, `prod_category_id`. -Enriched seed data: 2M orders spread over 365 days with varied totals ($10–$990), 10 countries, 500 customers, 5 000 incremental orders/day; raised DQX WARN limit to 1 000 to match the new range. -Added `scripts/sdk_truncate_tables.py` (with `make truncate env=X yes=--yes`) and `scripts/_sdk_sql.py` (shared `get_warehouse_id`/`run_sql` helpers extracted from `sdk_init_workspace.py`). -Added a 6 am BRT daily cron schedule to `job1_prod_integration`. +Added `product_id` (100 distinct) and `prod_category_id` (10 distinct) to the order schema and propagated them through extract → enrich → aggregate → SDP transforms; `report.order_agg` now groups by `name`, `date`, `product_id`, `prod_category_id`. Enriched seed data: 2M orders spread over 365 days with varied totals ($10–$990), 10 countries, 500 customers, 5 000 incremental orders/day; raised DQX WARN limit to 1 000 to match the new range. Added `scripts/sdk_truncate_tables.py` (with `make truncate env=X yes=--yes`) and `scripts/_sdk_sql.py` (shared `get_warehouse_id`/`run_sql` helpers extracted from `sdk_init_workspace.py`), plus a 6 am BRT daily cron schedule on `job1_prod_integration`. --- ## [#29](https://github.com/andre-salvati/databricks-template/pull/29) · 2026-05-29 · feat: scale load-test, rewrite seed_sources, add prod integration job -Scaled load-test to 500 customers / 2M orders / 6M order_items so the batch vs incremental timing gap is measurable; updated `_validate_load_test` assertions accordingly. -Rewrote `seed_sources`: initial load when `customer` count < 500, then daily append of 2000 orders + items and MERGE of 50 customer country updates; logs table totals after each run. -Added `job1_prod_integration` job (seed → run + run_sdp in parallel, no validate); removed `seed_sources` from `job1_prod` so the prod ETL is structurally identical to staging. +Scaled load-test to 500 customers / 2M orders / 6M order_items so the batch vs incremental timing gap is measurable; updated `_validate_load_test` assertions accordingly. Rewrote `seed_sources`: initial load when `customer` count < 500, then daily append of 2000 orders + items and MERGE of 50 customer country updates; logs table totals after each run. Added the `job1_prod_integration` job (seed → run + run_sdp in parallel, no validate) and removed `seed_sources` from `job1_prod` so the prod ETL is structurally identical to staging. --- ## [#28](https://github.com/andre-salvati/databricks-template/pull/28) · 2026-05-29 · feat: add load-test mode to integration tests via job parameter -Added a `load_test` job parameter (default `"false"`) to the integration test job. -When `"true"`, `Setup` seeds 200 customers / 500k orders / 200 order_items via `spark.range()` and `Validate` checks both `report.order_agg` and `report.order_agg_sdp` for deterministic aggregates. -No new task classes — the existing integration test job handles both modes. +Added a `load_test` job parameter (default `"false"`) to the integration test job. When `"true"`, `Setup` seeds 200 customers / 500k orders / 200 order_items via `spark.range()` and `Validate` checks both `report.order_agg` and `report.order_agg_sdp` for deterministic aggregates. No new task classes — the existing integration test job handles both modes. --- diff --git a/specs/tooling.md b/specs/tooling.md index 5e2fa32..7a70296 100644 --- a/specs/tooling.md +++ b/specs/tooling.md @@ -1,27 +1,81 @@ # Tooling: MCP servers, CLI, and skills -This project is developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutions/ai-dev-kit) -installed at the user level (`~/.ai-dev-kit/`). MCP servers and skills are **user-level tooling** — -`.mcp.json` and `.claude/` are gitignored and are **not** part of the repo. The one exception is -`.claude/commands/`, which `.gitignore` un-ignores: project-specific slash commands (e.g. -`/project-costs`) are committed so every developer gets them. This doc records what's wired up -locally so a session knows what to reach for; it does not provision anything. +This project is developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutions/ai-dev-kit). +The kit is **user-level tooling**: nothing it installs belongs in this repo, and none of it is +committed. This doc is the single source of truth for what's wired up locally and what to reach for; +`CLAUDE.md` carries only a short per-session decision list that links back here. Neither doc +provisions anything. -`CLAUDE.md` carries a short decision list for every session; this is the full reference. +## Install layout + +The kit lives at `~/.ai-dev-kit/` — a git clone (`repo/`, pinned to a release tag), its own `.venv/`, +and installer bookkeeping (`version`, `.skills-profile`, `.installed-skills`). Currently **v0.1.13**, +profile `all`. Upgrade by re-running the installer, which refreshes every tracked root: + +```bash +bash <(curl -sL https://raw.githubusercontent.com/databricks-solutions/ai-dev-kit/main/install.sh) +``` + +Skills install into three user-level roots, one per agent tool — all fed by that one installer: + +| Root | Tool | +|---|---| +| `~/.claude/skills/` | Claude Code | +| `~/.agents/skills/` | Codex | +| `~/.github/skills/` | Copilot | + +**Don't install the kit into this repo.** Skills placed in a project root silently take precedence +over the user-level set *and* are invisible to `install.sh`, so they never update — the drift is +undetectable from a session. This repo carried exactly that: a project-scoped install from +2026-05-26 left `.claude/skills/`, `.github/skills/`, and `.ai-dev-kit/` here, and the stale +`.claude/skills/` shadowed the user-level set for two months. It was removed on 2026-07-16; +`.github/skills/` and `.ai-dev-kit/` remain (gitignored, stale, and only relevant to Copilot). + +`.gitignore` keeps all of it out of the repo: `.mcp.json`, `.ai-dev-kit/`, `.github/skills/`, and +`.claude/*`. The one exception is `.claude/commands/`, which is un-ignored so project slash commands +(e.g. `/project-costs`) are committed and every developer gets them. ## MCP servers -Four MCP servers are configured in `.mcp.json` (all `defer_loading: true` — schemas load on demand): +Four servers are configured in `.mcp.json` (all `defer_loading: true` — schemas load on demand): | Server | Tools | Auth / env | Reach for it when… | |---|---|---|---| -| **databricks** | `mcp__databricks__*` (`manage_jobs`, `manage_job_runs`, `manage_uc_objects`, `execute_sql`, `manage_serving_endpoint`, `manage_workspace_files`, …) | `dev` profile in `~/.databrickscfg` | any workspace / Unity Catalog / Jobs / Pipelines / Apps / Serving / SQL operation. **Prefer these over `databricks` CLI shell-outs or hand-rolled SDK scripts.** | -| **aws-billing-cost** | `mcp__aws-billing-cost__*` (cost-explorer, pricing, cost-anomaly, cost-comparison, budgets, …) | `AWS_PROFILE=costs` (dedicated read-only IAM user) | analyzing project cloud spend, cost anomalies/spikes, or pricing. Pairs with `scripts/project_costs.py` and the `/project-costs` skill. Defaults: UnblendedCost, exclude credits/refunds. | +| **databricks** | `mcp__databricks__*` (`manage_jobs`, `manage_job_runs`, `manage_uc_objects`, `execute_sql`, `manage_serving_endpoint`, `manage_workspace_files`, …) | `DATABRICKS_CONFIG_PROFILE=DEFAULT` — see the identity note below | any workspace / Unity Catalog / Jobs / Pipelines / Apps / Serving / SQL operation. **Prefer these over `databricks` CLI shell-outs or hand-rolled SDK scripts.** | +| **aws-billing-cost** | `mcp__aws-billing-cost__*` (cost-explorer, pricing, cost-anomaly, cost-comparison, budgets, …) | `AWS_PROFILE=costs` (dedicated read-only IAM user) | analyzing project cloud spend, cost anomalies/spikes, or pricing. Pairs with `scripts/project_costs.py` and `/project-costs`. Defaults: UnblendedCost, exclude credits/refunds. | | **aws-documentation** | `mcp__aws-documentation__*` (search_documentation, read_documentation, recommend) | none | you need authoritative AWS docs (S3, IAM, external locations, Cost Explorer semantics). Cite the doc URL. | | **context7** | `mcp__context7__*` (resolve-library-id, query-docs) | none | you need **current** docs for a library / SDK / CLI (PySpark, Databricks SDK, uv, ruff, pytest). Prefer over web search for library docs — training data may be stale. Not for refactoring or business-logic debugging. | +The `databricks` server runs out of the kit's venv +(`~/.ai-dev-kit/.venv/bin/python ~/.ai-dev-kit/repo/databricks-mcp-server/run_server.py`), so it +breaks if `~/.ai-dev-kit/` is moved or removed. + +### MCP runs as the production service principal + +All four profiles in `~/.databrickscfg` point at the same workspace host, but they resolve to **two +different identities** (verified with `databricks current-user me --profile

`): + +| Profile | Auth | Resolves to | +|---|---|---| +| `dev` | `auth_type = databricks-cli` (user OAuth) | your user account | +| `DEFAULT`, `staging`, `prod` | `oauth-m2m` | the `template-sp` service principal | + +The `dev` profile also carries a `client_id`/`client_secret`, but they are **inert** — `auth_type = +databricks-cli` overrides them. Comparing `client_id` values across profiles is therefore +misleading; check the resolved identity with `databricks auth describe --profile

` instead. + +This matters because the `databricks` MCP server is pinned to `DATABRICKS_CONFIG_PROFILE=DEFAULT`, +which resolves to `template-sp` — **the same identity `prod` runs as**. MCP tool calls do not run as +your `dev` user and are not scoped to dev: an `execute_sql` through the server carries the service +principal's privileges and can read or write `prod` tables. Environment separation is *catalog-level*, +exactly as [`data-model.md`](data-model.md) describes — the guardrail is the catalog you name in the +query, not the profile you assume you're on. Name catalogs explicitly and check before mutating. + +Profiles still select the bundle target where a *script* maps profile → env (`make deploy env=prod` +uses `prod`), which is what `make whoami` reports on. + If MCP tools are unavailable in a session, fall back to the `databricks` CLI or `databricks-sdk` -directly (or `aws` CLI / web search for the AWS/context7 cases) — but flag the fallback to the user. +directly (or `aws` CLI / web search for the AWS and context7 cases) — but flag the fallback. ## Databricks CLI @@ -31,7 +85,8 @@ use `prod` for prod operations. To check or switch profiles, invoke the `databri ## Skills -Invoke via the Skill tool when the task matches: +Invoke via the Skill tool when the task matches. `databricks` (frontmatter name: **`databricks-core`**) +is the kit's entry point for CLI, auth, and bundle work — load it first, then the product skill. - **databricks-bundles** — editing `databricks.yml` / `resources/*.yml`, deploy/run. Note: this project **generates** `resources/jobs.yml` via `scripts/sdk_generate_template_job.py`; route job @@ -41,10 +96,40 @@ Invoke via the Skill tool when the task matches: - **databricks-python-sdk** — SDK code under `src/template/` and in `scripts/`. - **databricks-unity-catalog**, **databricks-aibi-dashboards**, **databricks-spark-declarative-pipelines**, etc. — invoke when the task is squarely in that area. -- **project-costs** — run `scripts/project_costs.py` and analyze the output for spikes/trends. + +Two gotchas. Some skills' frontmatter `name:` differs from their directory (`databricks` declares +`databricks-core`; `analyze-mlflow-trace` declares `analyzing-mlflow-trace`) — **invoke by directory +name**, which is what the session's skill list shows; the frontmatter name is not the handle. And +`/project-costs` is **not** a kit skill; it's this repo's own committed slash command +(`.claude/commands/project-costs.md`) wrapping `scripts/project_costs.py`. + +`databricks-core` also cross-references skills by their *post-migration* names — it points at +`/databricks-dabs` and `databricks-data-discovery`, neither of which is installed yet. Read those as +`databricks-bundles` and "not available" until the rename below lands. + +## Upcoming: skills move to the official Databricks set + +v0.1.13 is the **last release that installs skills from the kit repo's own files**. The next release +installs them from the official, engineering-supported Databricks skills set via the Databricks CLI; +`install.sh` stays the front-end, so the upgrade command doesn't change. The **MCP server and Builder +App stay in the kit repo** — but the MCP server drops to *best-effort maintenance as issues are +filed*, which is worth knowing given this project depends on it daily. + +Most skill names carry over. The exceptions will break references in this doc and `CLAUDE.md`, so +update both when the release lands: + +| Today | Official set | +|---|---| +| `databricks-bundles` | `databricks-dabs` | +| `databricks-spark-declarative-pipelines` | `databricks-pipelines` | +| `databricks-lakebase-autoscale`, `databricks-lakebase-provisioned` | `databricks-lakebase` (merged) | +| `databricks-config` | folded into `databricks-core` | ## Conventions -- Use the `dev` profile unless told otherwise; `prod` for prod jobs/SQL/pipelines. -- Do **not** install the Dev Kit into this repo or commit MCP config — it's user-level tooling. +- Use the `dev` profile unless told otherwise; `prod` for prod jobs/SQL/pipelines — but see + [MCP runs as the production service principal](#mcp-runs-as-the-production-service-principal): + MCP calls bypass your `dev` identity entirely. The catalog is the guardrail, not the profile. +- Do **not** install the Dev Kit into this repo or commit MCP config — it's user-level tooling, and + a project-scoped install silently shadows the maintained set. - Prefer MCP tools over CLI shell-outs when both are available; flag any fallback. diff --git a/specs/workflow.md b/specs/workflow.md index 67d161d..34098b9 100644 --- a/specs/workflow.md +++ b/specs/workflow.md @@ -19,7 +19,10 @@ before starting any change. For what the code does, see [architecture.md](archit ## Branches & commits -- Branch from an up-to-date `main`; one subject per branch/PR. +- Branch from an up-to-date `main`; one subject per branch/PR. Check where you are first + (`git branch --show-current`): if you're on `main` or on a stale/already-merged branch, run + `git checkout main && git pull && git checkout -b ` before editing. Starting from a + diverged base causes conflicts and can silently regress work from a merged PR. - Commit messages end with the `Co-Authored-By: Claude Opus 4.8 ` trailer. - **Never commit generated / local-state files** (all gitignored): `resources/jobs.yml`, `resources/orders_dashboard_deploy.lvdash.json`, `.databricks-resources.json`. @@ -73,14 +76,31 @@ default for schema migrations) / *rebuild* / *leave as-is*. On `staging`/`prod`, ## CHANGELOG discipline -`specs/CHANGELOG.md` is **append-only** — add a new entry before every merge, never edit old ones. -Each entry is **at most 3 sentences**. Header format: +`specs/CHANGELOG.md` is **append-only** — add a new entry at the top before every merge, and never +edit or reformat an existing one. This section is the authority on the rule; `CLAUDE.md` only points +here. + +**Write it at merge time — not before.** Don't draft the entry when you cut the branch, when you +commit, or when you open the PR. A branch's scope almost always grows, and an entry written early +just gets rewritten every time it does. Write it once, against the branch's final scope, when merge +is imminent. Forgetting isn't the risk: the `require-changelog-entry.sh` hook blocks `gh pr merge` +when the branch adds no entry, so that block is the reminder. + +**Size — around 1000 characters**, written as one unwrapped paragraph. The cap is a length budget, +not a wrap width: don't hard-wrap the entry, and don't pad a short one to reach it. Character count +is the only limit — an earlier "exactly 3 sentences" rule was dropped because it constrained nothing +(three sentences had drifted into 17-line run-ons; see the first draft of #48). Name what changed +and the one fact a future reader needs — exhaustive detail belongs in the PR description and the +commit message, which is where it survives anyway. + +**Header format** — use the PR URL directly, since the PR number is known once it's open: ``` -## [#NN] · · YYYY-MM-DD · +## [#NN](https://github.com/andre-salvati/databricks-template/pull/NN) · YYYY-MM-DD · <title> ``` -Replace `<branch>` with the PR URL after the PR is merged. +The existing entries were normalized to this style once, in #49. That was a deliberate one-off to +give the file a single voice; append-only applies from there on. ## Keep docs in sync @@ -110,7 +130,7 @@ uv run pytest tests/job1/unit_test.py uv run pytest tests/job1/unit_test.py::test_enrich_orders ``` -Coverage: `make unit-test` writes a report under `coverage_reports/` (uploaded as a CI artifact, +Coverage: `make unit-test` writes a report under `reports/coverage/` (uploaded as a CI artifact, 14-day retention). | Test (batch — `unit_test.py`) | Asserts |