diff --git a/README.md b/README.md index fe435062..df5384b9 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,11 @@ The `environment.yml` file should define the dependencies of the environment req See [detailed instructions](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#sharing-an-environment) for creating a `environment.yml` file. +> **Note:** notebooks are automatically tested in CI and made runnable in Google +> Colab. Before opening a PR, see **[Adding a notebook: CI, Colab, and the +> exclusion lists](docs/adding-notebooks.md)** for the required Colab-bootstrap +> install cell, how the CI test works, headless-execution gotchas, and the +> `.github` exclusion lists. Feel free to reach out on the [DANDI helpdesk](https://github.com/dandi/helpdesk/issues/new/choose) with any questions. diff --git a/docs/adding-notebooks.md b/docs/adding-notebooks.md new file mode 100644 index 00000000..4f64c5ab --- /dev/null +++ b/docs/adding-notebooks.md @@ -0,0 +1,225 @@ +# Adding a notebook: CI, Colab, and the exclusion lists + +This guide explains what happens to a notebook after you open a Pull Request — +how it is tested, how it is made runnable in Google Colab, and the three +`.github/*.txt` lists that control that behavior. For the basic file layout and +`environment.yml`, see the [README](../README.md#submission-instructions); +this doc picks up where that leaves off. + +## TL;DR — checklist for a new notebook + +- [ ] Place it under `///` with a `README.md` + and `environment.yml` (see main README). +- [ ] Prepend the **Colab-bootstrap cells** (badge → install intro → pinned + install cell → restart admonition). See [below](#the-colab-bootstrap-cells). +- [ ] Generate the install cell's pins with `uv pip compile`, constrained to + Colab's versions. See [generating the install cell](#generating-the-install-cell). +- [ ] **Stream data directly from the DANDI Archive** (remfile/fsspec) — don't + download large files or hardcode local paths. See + [streaming data](#stream-data-from-the-dandi-archive). +- [ ] Make sure it runs **headlessly** — no `fig.show()` on a default plotly + renderer, no `cv2.imshow`, no `%matplotlib widget`, no `input()`. See + [headless gotchas](#headless-gotchas). +- [ ] Open the PR. CI runs the changed notebook end-to-end. Green = good. +- [ ] If it genuinely can't run in CI (or shouldn't get a Colab button), add it + to the right [exclusion list](#the-three-github-lists) with a one-line reason. + +## How CI tests a notebook + +Two workflows execute notebooks: + +| Workflow | Trigger | Scope | +|---|---|---| +| [`test-changed-notebooks.yml`](../.github/workflows/test-changed-notebooks.yml) | PR touching `**.ipynb` (or the test harness) | Only the notebooks **added/modified** in the PR | +| [`test-all-notebooks-weekly.yml`](../.github/workflows/test-all-notebooks-weekly.yml) | Mondays 06:00 UTC + manual dispatch | Every testable notebook; opens an issue on failure | + +Both call [`.github/scripts/run_notebook.py`](../.github/scripts/run_notebook.py) +on an `ubuntu-latest` runner with Python 3.12. That script: + +1. Finds the **install cell** (the first code cell containing + `!uv pip install --system`) and extracts the pinned package list from it. +2. Installs those exact pins with `uv pip install --system`. +3. Runs any `!curl`/`!wget` helper-file fetches from that cell. +4. **Stubs out** the install cell, converts the notebook to a script with + `nbconvert --to script`, and runs it under `ipython` (sidestepping the + flaky ZMQ kernel). +5. Exits non-zero — failing CI — on any install error, helper-fetch failure, or + notebook exception. + +> **Key consequence:** CI is a *fresh-env* test. It installs **only** what the +> install cell pins — there is no preinstalled safety net. If the notebook +> imports something the install cell doesn't pin, CI fails (even if it "works on +> Colab," where that package happens to be preinstalled). + +`test-changed-notebooks.yml` only tests notebooks **changed in the PR**. A +notebook already on `master` that would fail today is not re-run until it is +touched again (or until the weekly sweep catches it). Don't assume "it's on +master, so it's green." + +A brand-new notebook **without** an install cell fails by design — CI can't +know what to install. Add the bootstrap, or add the notebook to an exclusion +list with a reason. + +## The Colab bootstrap cells + +Every testable notebook begins with four cells (the pattern established in +[PR #149](https://github.com/dandi/example-notebooks/pull/149)): + +1. **Colab badge** (markdown): + ```markdown + [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dandi/example-notebooks/blob/master/.ipynb) + ``` +2. **Install intro** (markdown): one paragraph explaining the install cell. +3. **Install cell** (code) — the one CI keys off: + ```python + #@title Installing requirements (click ▶ to run) { display-mode: "form" } + # Colab provides Python 3.12. We install with `uv --system` because Colab's + # kernel runs outside a virtualenv. All versions (direct + transitive) are + # pinned below so the notebook is reproducible regardless of resolver drift. + !pip install -q uv + !uv pip install --system \ + "package-a==1.2.3" \ + "package-b==4.5.6" \ + ... + ``` + The `{ display-mode: "form" }` collapses the cell to a one-line bar in Colab + so users aren't faced with ~100 lines of pins. If the notebook needs a + colocated helper `.py`, fetch it at the end of this cell: + `!curl -sL -o helper.py https://raw.githubusercontent.com/dandi/example-notebooks/master//helper.py` +4. **Restart admonition** (markdown): tells the user to restart the runtime + after install, because upgrading C-extension packages mid-session requires it. + +### Why fully pinned? + +Pinning **all** transitive deps (not just direct ones) makes the notebook +reproducible forever — it can't break later when an upstream release changes a +default. It's also what CI installs, so a green CI run means the exact pinned +set works on Python 3.12 / linux. + +## Generating the install cell + +Resolve a full, pinned lock with [`uv`](https://github.com/astral-sh/uv), +constrained to **Colab's preinstalled versions** so the install is a no-op for +packages Colab already ships (faster, and avoids the "RESTART RUNTIME" prompt +that changing `numpy` and other C-extensions triggers): + +```bash +uv pip compile requirements.in \ + --python-version 3.12 \ + --python-platform linux \ + --constraint .github/colab-preinstalled.txt \ + -o pins.txt +``` + +where `requirements.in` lists the notebook's **direct** imports (e.g. `dandi`, +`pynwb`, `remfile`, `matplotlib`, `pandas`). Then format each `pkg==ver` line +into the `!uv pip install --system \` block. + +[`.github/colab-preinstalled.txt`](../.github/colab-preinstalled.txt) is a +pip-freeze of the current Colab Python 3.12 runtime (numpy 2.0.2, etc.). Using +it as a constraint keeps your pins aligned with Colab. When a notebook's needs +are genuinely incompatible with a Colab version, the resolver falls back to a +non-Colab version for that package — the user will then get a restart prompt, +which is the correct trade-off. + +> **nbformat gotcha:** cell `id` fields require `nbformat_minor >= 5`. If you +> prepend cells and validation complains about an unexpected `id`, bump the +> notebook's `nbformat_minor` to 5. + +## Stream data from the DANDI Archive + +Notebooks should **stream their data directly from the DANDI Archive** rather +than downloading files or reading from a local path. Streaming reads only the +bytes a cell actually needs, so the notebook runs unattended in CI and in Colab +with no manual data setup, no multi-GB download, and no machine-specific paths. + +Resolve the asset's S3 URL through the DANDI API and open it with +[`remfile`](https://github.com/magland/remfile) (or `fsspec`): + +```python +from dandi.dandiapi import DandiAPIClient +import remfile, h5py +from pynwb import NWBHDF5IO + +with DandiAPIClient() as client: + asset = client.get_dandiset("000000", "draft").get_asset_by_path( + "sub-XX/sub-XX_ecephys.nwb" + ) + s3_url = asset.get_content_url(follow_redirects=1, strip_query=False) + +io = NWBHDF5IO(file=h5py.File(remfile.File(s3_url), "r"), load_namespaces=True) +nwbfile = io.read() +``` + +Guidelines: + +- **Don't** `dandi download` a dandiset, read `../path/to/local.nwb`, or assume a + file exists on disk — that fails in CI and gives Colab users a broken notebook. + (Hardcoded local paths are a common reason a notebook lands on the + [exclusion lists](#the-three-github-lists).) +- **Do** wrap the remote file in a cache so re-runs don't re-fetch: + `remfile.File(s3_url, disk_cache=remfile.DiskCache("nwb-cache"))`. +- Pin `remfile` (and/or `fsspec`/`s3fs`) in the [install cell](#the-colab-bootstrap-cells). +- For Zarr-based assets, stream with `fsspec` instead of `remfile`. + +## Headless gotchas + +CI runs notebooks with **no display and no browser**. Most plotting is fine; +a few patterns are not. Before opening a PR, make sure none of these slip in: + +| Pattern | Headless? | Notes | +|---|---|---| +| `plt.show()` (matplotlib) | ✅ fine | Falls back to the Agg backend. | +| `IPython.display.IFrame(...)` | ✅ fine | Produces an inert HTML repr; never fetches or opens anything. | +| `fig.show()` (**plotly**) | ❌ **fails** | Default renderer is `"browser"` → calls `webbrowser.open()` → `Error: could not locate runnable browser`. Set a notebook renderer once near the top: `import plotly.io as pio; pio.renderers.default = "iframe"` (renders in Colab, safe headless). | +| `cv2.imshow` / `cv2.namedWindow` | ❌ fails | Needs a GUI window. | +| `%matplotlib notebook` / `widget` / `tk` / `qt` | ❌ fails | Interactive backends need a frontend. Use `%matplotlib inline`. | +| `input()` / `getpass()` | ❌ hangs | No stdin in CI. | +| `webbrowser.open(...)` | ❌ fails | No browser on the runner. | + +The plotly case is the most common surprise: a notebook works in Colab (which +has a notebook renderer + preinstalled plotly) but fails in CI. The fix is the +one-line `pio.renderers.default` setting above — keep it in CI **and** Colab. + +## The three `.github` lists + +These three files control different things and are **independent** of each +other. Add a notebook to whichever applies, always with a one-line reason. + +### `colab-preinstalled.txt` +[`.github/colab-preinstalled.txt`](../.github/colab-preinstalled.txt) — **not** +an exclusion list. It's a snapshot of Colab's preinstalled package versions, +used as the `uv pip compile --constraint` when generating install-cell pins (see +[above](#generating-the-install-cell)). Refresh it when Colab bumps its runtime. + +### `notebook-test-exclusions.txt` → "skip in CI" +[`.github/notebook-test-exclusions.txt`](../.github/notebook-test-exclusions.txt) +— notebooks the CI test sweep should skip. Add a notebook here when it can't run +cleanly on a headless CI runner: needs a database/credentials, hits a +headless-incompatible API the slim runner image lacks, a pre-existing content +bug, or an upstream break not yet fixed. Lines are fnmatch globs relative to the +repo root. Removing the line re-enables testing once the issue is fixed. + +### `notebook-colab-exclusions.txt` → "no Colab button" +[`.github/notebook-colab-exclusions.txt`](../.github/notebook-colab-exclusions.txt) +— notebooks that should **not** show an "Open in Colab" button on the index +page (consumed by `collect_and_render.py`). Add a notebook here when clicking +"Open in Colab" would give a broken experience even on a fresh Colab runtime: +stale data paths, missing creds, removed upstream APIs, wheels lacking a needed +feature, or runtimes too long for a tutorial. + +### They are independent + +A notebook can be on one list but not the other: + +- **Test-excluded but Colab-OK:** `read_avi.ipynb` uses OpenCV's `libxcb`, which + the slim CI image lacks but Colab has → on test-exclusions, **not** on + colab-exclusions. +- **Colab-excluded but test-OK:** rare, but e.g. a notebook that runs in CI yet + points users at data they can't access interactively. +- **Both:** the DataJoint examples need a MySQL server that neither CI nor Colab + provides → on both lists. + +## Where to get help + +Reach out on the [DANDI helpdesk](https://github.com/dandi/helpdesk/issues/new/choose).