chore: upgrade Databricks CLI to 1.9.0, DQX to 0.15.0, and track uv.lock

.github/workflows/onpush.yml

@@ -40,12 +40,20 @@ jobs:
           python-version: '3.12'
 
       - name: Install uv
-        uses: astral-sh/setup-uv@v7
+        # Pinned to the full release tag: setup-uv stopped publishing moving major
+        # tags (v1..v7) after v7, so `@v9` does not resolve — use the exact tag.
+        uses: astral-sh/setup-uv@v9.0.0
         with:
-          version: "0.9.8"
+          version: "0.11.31"
           enable-cache: true
           cache-dependency-glob: "uv.lock"
 
+      # Fail if a PR edited pyproject.toml without re-running `uv lock`. `make sync`
+      # would silently re-resolve and discard the update in CI's throwaway checkout,
+      # leaving the committed uv.lock stale. This read-only check keeps them in sync.
+      - name: Verify lockfile is current
+        run: uv lock --check
+
       - name: Install dependencies
         run: make sync
 
@@ -62,7 +70,7 @@ jobs:
 
       # Pin the CLI to a tagged release so an upstream change can't silently break CI.
       - name: Install Databricks CLI
-        uses: databricks/setup-cli@v0.298.0
+        uses: databricks/setup-cli@v1.9.0
 
       - name: Deploy on staging
         run: make deploy env=staging

.gitignore

@@ -11,7 +11,9 @@ build/
 reports/coverage/
 src/template.egg-info/
 *.pyc
-*.lock
+# uv.lock IS tracked — it pins transitive deps (notably databricks-sdk, which arrives
+# via DQX under a wide `~=0.73` range) so CI resolves the same graph every run. The old
+# blanket `*.lock` ignore was a leftover from the 2024 pipenv era and silently excluded it.
 .coverage*
 resources/jobs.yml
 resources/orders_dashboard_deploy.lvdash.json
@@ -20,6 +22,10 @@ resources/orders_dashboard_deploy.lvdash.json
 .ai-dev-kit/
 .github/skills/
 .mcp.json
+.mcp.json.bak
+# Dev Kit 0.1.14 also writes Codex/Copilot agent configs into the project dir
+.agents/
+.codex/
 # Generated spend reports — local artifacts containing account cost figures.
 # A report that has been committed stays tracked (.gitignore only governs untracked files), so
 # new dated reports are ignored by default; `git add -f` a specific one to keep it.

.pre-commit-config.yaml

@@ -2,7 +2,7 @@ default_language_version:
     python: python3
 repos:
 -   repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.15.15
+    rev: v0.15.22
     hooks:
     -   id: ruff
         args:

README.md

@@ -59,7 +59,7 @@ This project template demonstrates how to:
 - utilize [Databricks Unity Catalog](https://www.databricks.com/product/unity-catalog) to manage permissions and get data lineage.
 - enforce production guardrails out of the box — identity-locked CI deploys, a health-check task, wheel version pinning, per-task timeouts, schema-drift guards, queued runs, and on-call alerting.
 - track project cloud spend in USD across AWS (Cost Explorer) and Databricks ([`system.billing`](https://docs.databricks.com/aws/en/admin/system-tables/pricing)) with `make project-costs` — see an [example report](reports/cost/2026-07-22.md).
-- diagram any SQL query with `make sql-diagram sql=<file>` — [`sqlglot`](https://github.com/tobymao/sqlglot) parses the AST and writes a Mermaid flowchart plus a standalone SVG to `reports/sql-diagram/`, either as the query's execution steps (each scan, each join with its keys, filter, aggregate, sort — [example](reports/sql-diagram/job_spend_plan.svg)) or as column-level lineage (`mode=lineage`), so what is drawn is what the query says rather than what a model guessed.
+- diagram any SQL query with [sqlglot](https://github.com/tobymao/sqlglot) — see an [example](reports/sql-diagram/job_spend_plan.svg).
 - utilize serverless job clusters on [Databricks Free Edition](https://docs.databricks.com/aws/en/getting-started/free-edition) to deploy your pipelines.
 
 

databricks.yml

@@ -1,7 +1,7 @@
 # See https://docs.databricks.com/dev-tools/bundles/index.html for documentation.
 bundle:
   name: default_python
-  databricks_cli_version: ">=0.298.0"
+  databricks_cli_version: ">=1.9.0"
   engine: direct
 
 artifacts:

pyproject.toml

@@ -9,21 +9,25 @@ readme = "README.md"
 license = "Apache-2.0"
 requires-python = "==3.12.*"
 dependencies = [
-    "databricks-labs-dqx==0.12.0", 
+    "databricks-labs-dqx==0.15.0",
 ]
 
 [project.optional-dependencies]
 dev = [
+    # Frozen to mirror serverless environment version 5 (the runtime the wheel executes
+    # on): Spark 4.1.0 via Databricks Connect 18, plus its exact pandas/numpy/pyarrow.
+    # Do not bump these to the newest PyPI release — that puts unit tests ahead of the
+    # runtime. They move only when `client=` in scripts/sdk_generate_template_job.py does.
     "numpy==2.1.3",
     "pandas==2.2.3",
     "pyarrow==21.0.0",
-    "pydantic==2.10.6",
-    "coverage==7.6.1",
-    "pre-commit==4.0.1",
-    "pytest==8.3.5",
-    "pytest-cov==5.0.0",
     "pyspark==4.1.0",
-    "databricks-bundles>=0.298.0",
+    "pydantic==2.13.4",
+    "coverage==7.15.2",
+    "pre-commit==4.6.1",
+    "pytest==9.1.1",
+    "pytest-cov==7.1.0",
+    "databricks-bundles>=1.9.0",
     "sqlglot==30.13.0",
 ]
 

specs/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ---
 
+## [#51](https://github.com/andre-salvati/databricks-template/pull/51) · 2026-07-23 · chore: upgrade Databricks CLI to 1.9.0, DQX to 0.15.0, and track uv.lock
+
+Bumped the deploy toolchain — Databricks CLI and `databricks-bundles` `0.298.0` → `1.9.0` — DQX `0.12.0` → `0.15.0`, and the dev/test pins (ruff, uv, pytest 9, coverage, pre-commit, pydantic). The serverless-mirror pins (`pyspark` 4.1.0, pandas, numpy, pyarrow) stay frozen to environment version 5, the runtime the wheel executes on, and now carry a comment saying so — bumping them past PyPI would put unit tests ahead of production, and there is no Spark 4.2 serverless environment yet. DQX 0.15 widens the quarantine `_errors`/`_warnings` structs with `rule_fingerprint`, `rule_set_fingerprint` and `skipped`; since every medallion write uses `overwriteSchema=false`, that fails against tables created under 0.12, so `raw.order_quarantine` needs `make drop` before the first run on each env (validated on staging: drop → deploy → run). CLI v1.0.0 also moved OAuth tokens to the OS keyring, so `databricks-cli` profiles need one `databricks auth login`; service-principal profiles and CI are unaffected. Finally, `uv.lock` is now tracked — a stale `*.lock` ignore from the 2024 pipenv migration had let transitive deps like `databricks-sdk` float on every CI run — guarded by a `uv lock --check` step.
+
+---
+
 ## [#50](https://github.com/andre-salvati/databricks-template/pull/50) · 2026-07-22 · feat: SQL query-plan diagrams, and a reports/ folder for generated artifacts
 
 Added `make sql-diagram` / `/sql-diagram`, which parses a query with `sqlglot` and draws either its execution steps (default) or its column-level lineage, emitting the analysed `.sql` beside a Mermaid `.mmd` and a hand-written `.svg` — no `mmdc`, so no Node toolchain in a Python repo. `sqlglot` models a multi-table join as one n-ary step, so the plan builder splits it back into `JOIN 1`, `JOIN 2`, … in written order, which is what makes an under-constrained join predicate visible — the bug class behind the 3.5× fan-out fixed in #47. Table annotations come from Unity Catalog comments through the `dev` profile (the MCP service principal lacks `USE SCHEMA` on `system.billing`), opt-in as the only network call. Round-tripping the emitted SQL exposed that `planner.Step.dependencies` is a set, so node numbering followed the process hash seed and the `.mmd` churned on every run; dependencies are now walked in a stable order. Generated artifacts also moved under `reports/`: `coverage`, `cost` and `sql-diagram`.