🎉 Charts as first-class citizens in ETL: layered approach with grapher

.claude/skills/chart-editing/SKILL.md

@@ -1,162 +1,164 @@
 ---
 name: chart-editing
-description: Edit and preview .chart.yml files for OWID graph steps. Use when user wants to edit chart config, preview charts, change chart appearance, or work with graph step chart files.
+description: Create or edit an ETL-authored Grapher chart — a single-chart `.config.yml` in `etl/steps/export/multidim/`. Use when the user wants to author a chart from ETL, edit one, change its title/subtitle/colors/map settings, or preview an ETL-authored chart on staging. For charts with dropdowns (multi-dimensional), use the `create-multidim` skill instead.
 metadata:
   internal: true
 ---
 
-# Chart Editing
+# Chart Editing (ETL-authored single charts)
 
-Edit `.chart.yml` files that define OWID Grapher chart configurations, then preview changes.
+ETL-authored single charts are stored as zero-dimension mdim collections — a `.config.yml` with `dimensions: []` and exactly one view. ETL pushes them to Grapher's `chart_configs.etlConfig` column; admin edits land in `chart_configs.patch`; the two layers never collide.
 
-## File Locations
+This skill covers creating and editing those `.config.yml` files, pushing them to staging, and previewing the result.
 
-- Chart configs: `etl/steps/graph/<namespace>/<version>/<short_name>.chart.yml`
-- Schema: `schemas/chart-schema.json` (read this to understand valid fields)
-- DAG definitions: `dag/graph/<namespace>.yml` (maps graph steps to data dependencies)
+## File layout
 
-## Before Editing
+```
+etl/steps/export/multidim/<namespace>/latest/<short_name>.config.yml
+```
 
-1. **Read the schema** at `schemas/chart-schema.json` to understand all valid fields, enums, and nested structures
-2. **Read the target `.chart.yml`** to understand current config
-3. **Check the DAG** in `dag/graph/<namespace>.yml` to understand which dataset the chart depends on
+Reference example in this repo:
 
-## Chart Config Formats
+- `etl/steps/export/multidim/animal_welfare/latest/banning_of_chick_culling.config.yml`
 
-### Simple format (single indicator)
+The chart's public slug is auto-derived from the short name with underscores replaced by dashes (`banning_of_chick_culling` → `banning-of-chick-culling`).
 
-Used for charts with one indicator and no dimension toggles:
+## Minimum viable config
 
 ```yaml
-title: Which countries have banned chick culling?
-tab: map
-hasMapTab: true
-chartTypes: []
-yAxis:
-  min: auto
-map:
-  hideTimeline: true
-  colorScale:
-    customCategoryColors:
-      Banned: '#4881c6'
-      No laws: '#b6a28c'
-    customNumericColorsActive: true
-$schema: https://files.ourworldindata.org/schemas/grapher-schema.010.json
-originUrl: /animal-welfare
-dimensions:
-  - property: y
-    catalogPath: status
+topic_tags:
+  - "Animal Welfare"
+dimensions: []
+views:
+  - dimensions: {}
+    indicators:
+      y:
+        - catalogPath: "<dataset_short_name>#<indicator_short_name>"
+    config:
+      $schema: "https://files.ourworldindata.org/schemas/grapher-schema.009.json"
+      title: "Your chart title"
+      subtitle: "One-line context for the chart."
+      note: "Any caveats, sources of bias, methodology notes."
+      originUrl: "/your-topic-page"
+      tab: "chart"
+      chartTypes:
+        - "LineChart"  # or StackedArea, DiscreteBar, etc.
+      yAxis:
+        min: 0
+      selectedEntityNames:
+        - "United States"
 ```
 
-Key fields: `dimensions[].property` (y/x/size/color) and `dimensions[].catalogPath` (column name in the dataset).
+Key fields:
 
-### Multi-dimensional format (explorer-like)
+- `dimensions: []` and exactly one view → this YAML pushes as a single chart, not an mdim page.
+- `views[0].indicators.y` — list of indicator catalog paths. For multi-series, list more than one.
+- `views[0].config` — the grapher config that becomes the chart's `etlConfig` in `chart_configs`. Same shape as a chart-admin export.
+- No top-level `title:` or `default_selection:` block — those exist only for multidim data pages and are ignored for single charts.
 
-Used for charts with multiple indicator combinations selectable via dropdowns:
+## DAG entry
+
+Add to `dag/<namespace>.yml`:
 
 ```yaml
-slug: covid/covid#covid_cases
+  #
+  # <Chart description> — chart authored in ETL.
+  #
+  export://multidim/<namespace>/latest/<short_name>:
+    - data://grapher/<namespace>/<version>/<dataset_short_name>
+```
 
-definitions:
-  common_views:
-    - config:
-        tab: map
-        originUrl: ourworldindata.org/coronavirus
+The dependency is the upstream `grapher` step whose dataset contains the indicators referenced in `views[0].indicators.y`.
 
-title:
-  title: COVID-19 confirmed cases
-  title_variant: ""
+## Indicators with custom display names
 
-default_selection:
-  - World
-  - Europe
-  - Asia
+The legend label defaults to the indicator's full title. For better legends, pass each indicator as an object with `display.name`:
 
-topic_tags:
-  - COVID-19
-
-dimensions:
-  - slug: period
-    name: Period
-    choices:
-      - slug: weekly
-        name: Weekly
-      - slug: biweekly
-        name: Biweekly
-
-  - slug: metric
-    name: Indicator
-    choices:
-      - slug: absolute
-        name: Absolute number
-      - slug: per_capita
-        name: Per million people
+```yaml
+indicators:
+  y:
+    - catalogPath: "<dataset>#<indicator_a>"
+      display:
+        name: "Short label A"
+    - catalogPath: "<dataset>#<indicator_b>"
+      display:
+        name: "Short label B"
+```
 
-views:
-  - dimensions:
-      period: weekly
-      metric: absolute
-    indicators:
-      y:
-        - catalogPath: weekly_cases
+Other useful `display` fields: `unit`, `shortUnit`, `numDecimalPlaces`, `roundingMode`, `numSignificantFigures`, `tolerance`, `zeroDay`.
 
-  - dimensions:
-      period: weekly
-      metric: per_capita
-    indicators:
-      y:
-        - catalogPath: weekly_cases_per_million
-```
+## Common chart-config edits
 
-Key differences from simple: `title` is an object `{title, title_variant}`, `dimensions` defines UI facets (not indicators), actual indicators are in `views[].indicators.y[].catalogPath`.
+Inside `views[0].config`:
 
-## Previewing Charts
+| What | Field | Notes |
+|---|---|---|
+| Chart type | `chartTypes: ["LineChart"]` | `LineChart`, `ScatterPlot`, `StackedArea`, `DiscreteBar`, `StackedDiscreteBar`, `SlopeChart`, `StackedBar`, `Marimekko` |
+| Default tab | `tab: "chart"` | `chart`, `map`, `table`, `line`, `slope`, `discrete-bar`, `marimekko` |
+| Map tab visible? | `hasMapTab: true` | Set with `tab: "map"` for map-by-default charts |
+| Y-axis range | `yAxis: { min: 0, max: 100 }` | Use `"auto"` for auto-scaling |
+| Default entities | `selectedEntityNames: ["United States"]` | List of country / region names |
+| Footer note | `note: "..."` | Caveats, methodology, source notes |
+| Origin URL | `originUrl: "/topic-page-slug"` | Links the chart to its topic page |
+| Map colors | `map.colorScale.customCategoryColors: {...}` | For categorical indicators on a map |
+| Color scheme | `map.colorScale.baseColorScheme: "BinaryMapPaletteA"` | See grapher schema for valid values |
+| Hide map timeline | `map.hideTimeline: true` | For point-in-time map charts |
+| Topic page tag | `topic_tags: ["Animal Welfare"]` | Top-level field, outside `views` |
 
-Charts are previewed via the staging server. The staging URL follows this pattern:
+For the authoritative list, the grapher schema is at the URL in `$schema:` — currently `grapher-schema.009.json`.
 
-```
-http://staging-site-{branch}/grapher/{slug}
+## Pushing to staging
+
+```bash
+.venv/bin/etlr export://multidim/<namespace>/latest/<short_name> --export --force --only
 ```
 
-where `{branch}` is the current git branch name (with `/._` replaced by `-`, truncated to 28 chars).
+`--force --only` re-pushes even when nothing changed (useful when iterating on the YAML). The step prints `admin_url=http://staging-site-<branch>/admin/charts/<id>/edit` on success.
 
-### PNG screenshot (for visual inspection)
+## Previewing
 
-Fetch the PNG directly from the staging server using `WebFetch`:
+Use the `check-chart-preview` skill — its `get_chart_png_url.py` helper resolves the chart slug to a draft-friendly PNG URL via `/grapher/by-uuid/<UUID>.png` (works for unpublished charts):
 
-```
-http://staging-site-{branch}/grapher/{slug}.png?nocache
+```bash
+URL=$(.venv/bin/python .claude/skills/check-chart-preview/get_chart_png_url.py <slug>)
+curl -o ai/chart_preview.png "$URL"
 ```
 
-Add `&tab=map` or `&tab=chart` to control which tab is shown.
+Pass extra grapher query params with `--key=value`:
 
-**Prerequisite**: The chart must be pushed to staging first. If the user has the VSCode chart preview extension open, this happens automatically via `etlr --watch`. Otherwise push manually:
 ```bash
-.venv/bin/etlr graph://<namespace>/<version>/<slug> --graph --graph-push --private
+.venv/bin/python .claude/skills/check-chart-preview/get_chart_png_url.py <slug> --tab=map
+.venv/bin/python .claude/skills/check-chart-preview/get_chart_png_url.py <slug> --tab=chart --time=2020 --country=USA~GBR~FRA
 ```
 
-## Editing Workflow
+Read the resulting PNG with the `Read` tool to view the chart.
+
+## Editing workflow
+
+1. Read the current `.config.yml` and the upstream dataset's `.meta.yml` (so you know what indicators exist and their default titles/units).
+2. Edit the YAML using the `Edit` tool. Preserve comments with `ruamel` if needed (see `etl.files.ruamel_load/dump`).
+3. Push: `.venv/bin/etlr export://multidim/<namespace>/latest/<short_name> --export --force --only`.
+4. Preview the PNG (see above) and iterate.
+5. Once the chart looks right, commit the `.config.yml` (and the DAG entry if newly added) on the working branch.
+
+## Admin edits coexist with ETL edits
+
+Once a chart is on staging, an admin (human) can edit it in the chart editor. Those edits land in `chart_configs.patch` and survive subsequent ETL pushes — the layered model is exactly:
+
+```
+chart_configs.full = merge(variableETL, etlConfig, patch)
+```
 
-1. Read the chart file and schema
-2. Make edits using the Edit tool (preserve YAML comments with ruamel if needed)
-3. Push to staging: `.venv/bin/etlr graph://<namespace>/<version>/<slug> --graph --graph-push --private`
-4. Fetch the PNG from `http://staging-site-{branch}/grapher/{slug}.png?nocache` to verify visually
-5. If the chart looks wrong, read the schema for correct field names/values
+Admin overrides always win on a per-field basis. To "unlink" a field back to the ETL-authored value, click the chip next to the field in the admin editor — it clears that field from `patch`.
 
-## Common Edits
+## When NOT to use this skill
 
-- **Change default tab**: Set `tab` to `chart`, `map`, `table`, `line`, `slope`, `discrete-bar`, or `marimekko`
-- **Change selected entities**: Edit `selectedEntityNames` array
-- **Change colors**: Edit `map.colorScale.customCategoryColors` or `colorScale.customCategoryColors`
-- **Change color scheme**: Set `baseColorScheme` or `map.colorScale.baseColorScheme` (see schema for valid values)
-- **Hide/show map tab**: Set `hasMapTab: true/false`
-- **Add chart note**: Set `note` field
-- **Change axis**: Edit `yAxis` or `xAxis` with `min`, `max`, `scaleType`, `label`
+- **Chart with dropdowns / dimension selectors** → use `create-multidim`. Single-chart `.config.yml` files have `dimensions: []`; multi-dim ones don't.
+- **Brand-new chart from scratch and you want the structure auto-generated** → use `create-multidim` even for single charts; it writes the YAML skeleton, and you set `dimensions: []` after.
+- **Editing a chart that exists only in the admin (no `.config.yml`)** → adopt it into ETL first, then edit here. Adoption tooling (`chart_pull` CLI) is a Phase 1 follow-up.
 
-## Validation
+## Related skills
 
-Chart configs should conform to `schemas/chart-schema.json`. Key constraints:
-- `tab` enum: `chart`, `map`, `table`, `line`, `slope`, `discrete-bar`, `marimekko`
-- `chartTypes` enum items: `LineChart`, `ScatterPlot`, `StackedArea`, `DiscreteBar`, `StackedDiscreteBar`, `SlopeChart`, `StackedBar`, `Marimekko`
-- `dimensions[].property` enum: `y`, `x`, `size`, `color`, `table`
-- `addCountryMode` enum: `add-country`, `change-country`, `disabled`
+- `check-chart-preview` — for previewing the rendered chart (PNG or browser screenshot).
+- `create-multidim` — for charts with dropdowns.
+- `chart-preview` VSCode extension — interactive preview pane while you edit.

.claude/skills/check-chart-preview/SKILL.md

@@ -11,9 +11,9 @@ Visually verify that a chart or multidimensional indicator page renders correctl
 
 ## Prerequisites
 
-- The `agent-browser` skill must be available (for browser automation)
 - A staging server must be running for the current git branch (check via `curl -s -o /dev/null -w "%{http_code}" http://<container>/`)
 - The chart/mdim must have been pushed to staging already (either via `etlr --watch` in VSCode or manually)
+- For non-trivial UI checks (MDIM dropdowns, link/unlink chips, etc.) the `agent-browser` skill is needed; for plain PNG snapshots it isn't
 
 ## Getting the Staging URL
 
@@ -56,26 +56,40 @@ For export multidim: derive from path `etl/steps/export/multidim/<namespace>/<ve
 
 ## Checking the Preview
 
-Use the `agent-browser` skill to:
+### Recommended: PNG via `/grapher/by-uuid/<UUID>.png` (works for drafts)
 
-1. **Navigate** to the staging URL
-2. **Wait** for the chart to render (look for the Grapher SVG or iframe content to load)
-3. **Verify**:
-   - No error banner or "Chart not found" message
-   - Chart title is visible
-   - Data points / map regions are rendered (not an empty chart)
-   - For mdim: dimension dropdowns are present and functional
-4. **Take a screenshot** and save to `ai/` directory
+The fastest, most reliable way to grab a chart's rendered image — works for **drafts as well as published charts**, no browser, no auth required. Uses the same render pipeline OWID's public site uses for `/grapher/<slug>.png`, but addresses the chart by its `chart_configs.id` UUID, which is published to R2's `byUUID/` directory for every chart (whether or not the chart itself is `isPublished`).
 
-### Quick PNG check (non-mdim charts only)
+```bash
+# Resolve slug → by-uuid PNG URL, then fetch
+URL=$(.venv/bin/python .claude/skills/check-chart-preview/get_chart_png_url.py <slug-or-id>)
+curl -o ai/chart_preview.png "$URL"
+```
+
+Pass extra grapher query params with `--<key>=<value>`:
+
+```bash
+.venv/bin/python .claude/skills/check-chart-preview/get_chart_png_url.py <slug> --tab=map
+.venv/bin/python .claude/skills/check-chart-preview/get_chart_png_url.py <slug> --tab=chart --time=earliest..2020
+```
+
+The helper queries the staging MySQL via `etl.config.OWID_ENV.read_sql` (which auto-routes to `staging-site-<branch>`), so it works as long as the staging server is up and the chart has been pushed.
+
+Use the resulting PNG by reading it directly with the `Read` tool (Claude can view images).
+
+### Fallback: browser preview for `mdim` pages / interactive UI
+
+The by-uuid PNG route renders a single chart frame. For MDIM data pages with dropdowns, or for verifying interactive UI (controls, link/unlink chips, etc.), use the `agent-browser` skill to navigate to the staging URL from `get_staging_url.py` and take an in-browser screenshot.
+
+### Quick PNG check (published charts only)
 
-For simple (non-mdim) charts, you can fetch a PNG directly without a browser:
+For charts that are already published, the slug-based PNG route also works:
 
 ```bash
 curl -o ai/chart_preview.png "http://<container>/grapher/<slug>.png?nocache"
 ```
 
-Add `&tab=map` or `&tab=chart` to control which tab is shown. This does NOT work for mdim charts.
+This 404s for drafts — use the by-uuid route above instead.
 
 ## Pushing to Staging First