docs: local star-history chart, README refresh, and doc-consistency fixes

.pre-commit-config.yaml

@@ -7,5 +7,5 @@ repos:
     -   id: ruff
         args:
           - --fix
-        files: ^src/
+        files: ^(src|scripts)/
     -   id: ruff-format

CLAUDE.md

@@ -8,10 +8,10 @@ A production-ready PySpark/Databricks ETL pipeline template using medallion arch
 
 ## Tooling: MCP servers, CLI, skills → see [`specs/tooling.md`](specs/tooling.md)
 
-Developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutions/ai-dev-kit) (user-level, `~/.ai-dev-kit/`). MCP config (`.mcp.json`) and `.claude/` are gitignored — user-level tooling, not committed. Quick decision list (full reference in `specs/tooling.md`):
+Developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutions/ai-dev-kit) (user-level, `~/.ai-dev-kit/`). MCP config (`.mcp.json`) and `.claude/` are gitignored — user-level tooling, not committed; the exception is `.claude/commands/`, which is un-ignored and committed. Quick decision list (full reference in `specs/tooling.md`):
 
 - **Workspace / UC / Jobs / Pipelines / Apps / Serving / SQL** → prefer `mcp__databricks__*` tools over `databricks` CLI shell-outs or hand-rolled SDK scripts.
-- **Bundle / job changes** → `databricks-bundles` / `databricks-jobs` skills, but route job edits through `scripts/sdk_generate_template_job.py` + `make deploy` (never hand-edit `resources/jobs.yml`).
+- **Bundle / job changes** → `databricks-bundles` / `databricks-jobs` skills, and route job edits through `scripts/sdk_generate_template_job.py` + `make deploy`.
 - **Library/SDK docs** (PySpark, Databricks SDK, uv, ruff) → `context7` MCP, not memory or web search.
 - **Cloud spend / cost analysis** → `aws-billing-cost` MCP (`AWS_PROFILE=costs`) + `/project-costs` skill. **AWS docs** → `aws-documentation` MCP.
 - Use the `dev` profile unless told otherwise (`prod` for prod ops). If MCP tools are unavailable, fall back to CLI/SDK and flag it.
@@ -27,6 +27,9 @@ make init              # One-time workspace bootstrap (SP, catalogs, schemas, gr
 make deploy env=dev    # Generate resources/jobs.yml (jobs + SDP pipeline) + deploy bundle to target env (dev/staging/prod)
 make run env=dev       # Run integration test job on a target env (dev or staging)
 make drop env=dev      # Drop all medallion tables in a target env (schema migrations; staging/prod need yes=--yes)
+make whoami            # Print the identity the env's profile authenticates as (runs implicitly before deploy/run/drop)
+make project-costs     # AWS + Databricks spend report (--aws-profile costs); backs the /project-costs skill
+make star-history      # Regenerate the README star-history SVGs (assets/star_history*.svg) from the GitHub API
 ```
 
 Run a single test file:
@@ -43,7 +46,7 @@ uv run pytest tests/job1/unit_test.py::test_enrich_orders
 
 The detailed specs live in [`specs/`](specs/) — read the relevant one **before** working in that area:
 
-- [`specs/architecture.md`](specs/architecture.md) — execution flow, CLI surface, key classes, jobs DAG, job **generation** (`scripts/sdk_generate_template_job.py` → `resources/jobs.yml`; never hand-edit), CI/CD, job-level params, deploy-time env vars, logging, production guardrails, adding a new job.
+- [`specs/architecture.md`](specs/architecture.md) — execution flow, CLI surface, key classes, jobs DAG, job **generation**, CI/CD, job-level params, deploy-time env vars, logging, production guardrails, adding a new job.
 - [`specs/data-model.md`](specs/data-model.md) — plain-words pipeline overview, catalog/schema isolation, medallion flow, table schemas, **field naming conventions**, product-name freeze, liquid clustering, DQX/quarantine, lineage.
 - [`specs/workflow.md`](specs/workflow.md) — the development lifecycle (plan → branch → PR), PR description standard, production-table impact check, and the unit / integration / load test plan.
 - [`specs/tooling.md`](specs/tooling.md) — MCP servers (Databricks, AWS billing/docs, context7), CLI, and skills: what to reach for and when.
@@ -54,7 +57,6 @@ The detailed specs live in [`specs/`](specs/) — read the relevant one **before
 - **Catalog-level isolation** — env separation is at the *catalog* level (`dev_<user>` / `staging` / `prod`); the same medallion schemas (`external_source`/`raw`/`curated`/`report`/`ops`) exist in each. Staging/prod catalogs+schemas are owned by `make init`, not the runtime wheel.
 - **Product-name freeze** — silver freezes `product_name` onto each order line at sale time: batch via an insert-only `MERGE`, SDP via a **streaming table** (a materialized view would *restate* the name; a streaming table appends once and *freezes*). A later rename never relabels booked orders. `unit_price` is static; `total_value` in gold is `SUM(item_total)`.
 - **Generated job config** — `resources/jobs.yml` is generated by `scripts/sdk_generate_template_job.py`; never hand-edit it (it's gitignored).
-- **Schema-drift guard** — all medallion writes use `overwriteSchema=false` (the only exception is `ops._health`).
 
 ## Constraints (things that broke us)
 
@@ -70,7 +72,7 @@ The detailed specs live in [`specs/`](specs/) — read the relevant one **before
 - **Ask "should I open a new branch?" before executing a plan**, and **never commit directly to `main`** — cut a feature branch and land via PR (a hook blocks direct commits and pushes to `main`).
 - **Hold commits until asked.** Before merging, update the PR description (a hook uses it as the merge commit message body) following the What / Why / How / Validation / **Impact in prod** template in [`.github/PULL_REQUEST_TEMPLATE.md`](.github/PULL_REQUEST_TEMPLATE.md); any table schema/data change needs the production-table impact check.
 - **Keep docs in sync in the same commit.** Don't ship changes to the CLI surface (`main.py:arg_parser`), runtime env vars, catalog/schema model, or production guardrails without updating `README.md`, the relevant doc under `specs/`, and this file (`CLAUDE.md`) together.
-- **Add a `specs/CHANGELOG.md` entry before merging a PR** — append-only (never edit old ones), each entry **at most 3 sentences**.
+- **Add a `specs/CHANGELOG.md` entry immediately before merging a PR** (not while the work is in progress — scope grows, and an early entry just gets rewritten). Append-only (never edit old ones). Each entry is **exactly 3 sentences** and **at most ~5 rendered lines** (~475 chars); keep it one unwrapped paragraph — the line cap is a length budget, not a wrap width.
 
 ## Keep It Simple
 
@@ -79,8 +81,8 @@ Favor solutions with less code, fewer classes, and fewer abstractions. When two
 - Don't reintroduce `--user`, `--debug`, or `--schema` CLI args. They were removed deliberately — see PR #21.
 - Don't add `funcy` (or any decorator-based timing utility) to the dependencies. Use the structured logger.
 - Don't add `CREATE CATALOG` or `CREATE SCHEMA` calls outside the `args.env == "dev"` branch in `config.py`. Staging/prod catalogs and schemas are owned by `make init`; runtime jobs run without those privileges.
-- Don't commit `resources/jobs.yml` (gitignored — regenerated on every deploy).
+- Don't commit or hand-edit `resources/jobs.yml` — it's gitignored and overwritten on every deploy. Change `scripts/sdk_generate_template_job.py` instead (it generates jobs, the SDP pipeline, and the dashboard resource stanza).
 - Don't commit `.databricks-resources.json` (gitignored — local provisioning state, diverges per developer).
-- Don't commit or hand-edit `resources/orders_dashboard_deploy.lvdash.json` (gitignored — regenerated from `orders_dashboard.lvdash.json` on every deploy with `${var.catalog}` resolved, since DABs can't substitute bundle vars inside `.lvdash.json` content). Edit the source `orders_dashboard.lvdash.json` instead.
-- Don't hand-edit `resources/jobs.yml` — it is overwritten on every deploy. Change `scripts/sdk_generate_template_job.py` instead (it generates jobs, the SDP pipeline, and the dashboard resource stanza into `resources/jobs.yml`). `resources/orders_dashboard.lvdash.json` is committed and editable directly.
+- Don't commit or hand-edit `resources/orders_dashboard_deploy.lvdash.json` (gitignored — regenerated on every deploy). Edit the committed `resources/orders_dashboard.lvdash.json` instead — mechanics in [`specs/data-model.md#dashboard`](specs/data-model.md#dashboard).
+- Don't hand-edit `assets/star_history*.svg` — they're generated (and committed, unlike `resources/jobs.yml`) so the README renders from this repo instead of a third-party chart service. Regenerate with `make star-history`; change `scripts/star_history.py` to alter the chart.
 - For PySpark transformation chains, let `ruff format` shape any chain that's already multi-line (don't hand-tune it) and keep chains that fit on one line unbroken — see the convention in [`specs/architecture.md`](specs/architecture.md#code-style-pyspark-transformation-chains).

Makefile

@@ -36,3 +36,8 @@ drop: whoami
 project-costs: aws-profile ?= costs
 project-costs:
 	uv run python ./scripts/project_costs.py $(if $(aws-profile),--aws-profile $(aws-profile),)
+
+# Regenerate the README star-history chart from the GitHub API. The SVGs are committed,
+# so the README renders from this repo rather than a third-party chart service.
+star-history:
+	uv run python ./scripts/star_history.py

README.md

@@ -1,4 +1,4 @@
-# databricks-template — agentic development for Databricks + production-ready ETL
+# Databricks PySpark ETL Template — production-ready pipelines, built agentically with Claude Code
 
 ![Databricks](https://img.shields.io/badge/platform-Databricks-orange?logo=databricks)
 ![PySpark](https://img.shields.io/badge/pyspark-4.1+-brightgreen?logo=apache-spark)
@@ -9,7 +9,7 @@
 
 ## 🚀 Overview
 
-> Stop spending weeks on boilerplate. This PySpark project template for Databricks gives you medallion architecture, Python packaging, unit + integration + load tests, CI/CD via Declarative Automation Bundles, DQX data quality, and service-principal-based production deploys — all wired together and ready to ship. Whether you're starting a new Databricks ETL project or looking for a reference implementation of production-ready PySpark pipelines, fork this and go.
+> Stop spending weeks on boilerplate. This PySpark project template for Databricks gives you medallion architecture, Python packaging, unit + integration + load tests, CI/CD, DQX data quality, service-principal-based production deploys, and much more — all wired together and ready to ship. Whether you're starting a new Databricks ETL project or looking for a reference implementation of production-ready PySpark pipelines, fork this and go.
 
 If this saves you time, a star helps others find it. Let's [connect on LinkedIn](https://www.linkedin.com/in/andresalvati/).
 
@@ -27,7 +27,7 @@ If this saves you time, a star helps others find it. Let's [connect on LinkedIn]
 - Claude Code
 - PySpark 4.1
 - Spark Declarative Pipelines (SDP)
-- Python 3.12+
+- Python 3.12
 - GitHub Actions
 - Pytest
 
@@ -134,7 +134,7 @@ run:
 
         make deploy env=dev
 
-8) Configure CI/CD automation with the service principal ID and secret. Configure [GitHub Actions repository secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions) (DATABRICKS_HOST, DATABRICKS_PRINCIPAL_ID, DATABRICKS_SECRET).
+8) Configure CI/CD automation with the service principal ID and secret. Configure [GitHub Actions repository secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions): `DATABRICKS_HOST`, `DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET`, and `TEMPLATE_ALERT_EMAILS` (comma-separated on-call addresses).
 
 9) (Optional) You can also execute unit tests from your preferred IDE. Here's a screenshot from [VS Code](https://code.visualstudio.com/) with [Microsoft's Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python) installed.
 
@@ -146,10 +146,12 @@ Licensed under the [Apache License 2.0](LICENSE). See [`LICENSE`](LICENSE) and [
 
 ## Star History
 
-<a href="https://www.star-history.com/?repos=andre-salvati%2Fdatabricks-template&type=date&legend=top-left">
+<a href="https://github.com/andre-salvati/databricks-template/stargazers">
  <picture>
-   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=andre-salvati/databricks-template&type=date&theme=dark&legend=top-left" />
-   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=andre-salvati/databricks-template&type=date&legend=top-left" />
-   <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=andre-salvati/databricks-template&type=date&legend=top-left" />
+   <source media="(prefers-color-scheme: dark)" srcset="assets/star_history_dark.svg" />
+   <source media="(prefers-color-scheme: light)" srcset="assets/star_history.svg" />
+   <img alt="Star History Chart" src="assets/star_history.svg" width="800" />
  </picture>
 </a>
+
+Generated from the GitHub API by `make star-history` — see [`scripts/star_history.py`](scripts/star_history.py).