End-to-end sponsor example: catalog → shardmap → dual aggregation → cached reads

[espg]

🤖 from Claude

(espg-directed, in-session)

Tracking issue for the sponsor-facing end-to-end example: two notebooks plus the executed pipeline behind them, on real public data.

Deliverables

• notebooks/sponsor_end_to_end.ipynb — the full pipeline (assumes AWS + EDL credentials), every run step with copy-paste instructions:
  1. CMR → STAC catalog query for the NEON AOP SERC AOI (tests/data/benchmark/AOP_NEON.geojson, ATL03 v007), saved as stac-geoparquet (build-once pattern, per the add 88 south stress test shard + benchmark #148/88S stress shard: pins, 900s timeout, chunk-boundary extraction, a-priori read plan, streaming tdigest (issue #148) #152 precedent).
  2. Shardmap built from the saved catalog with the strict-AOI aoi_mask payload (Optional strict-AOI polygon clip for shardmap output (sub-shard clipping) #101), visualized in-notebook (shardmap_viewer approach).
  3. Two aggregations from the same shardmap (reuse story, Reuse shardmaps across configs with different aggregators (relax signature guard to spatial-only) #89): a t-digest config and a gain_bias config, both carrying the AOI-mask column and the virtual-index cache (index: block — sidecar / on_miss / write_back semantics per Virtual index entry point: pluggable chunk-index backends for the read path #160; the cache built here is the Virtual chunk-index backends for the read path (issue #160) #163→High-resolution morton location channel for ragged/CSR fields #165 machinery). Outputs to s3://sliderule-public-cors/zagg-examples/serc/; sidecar manifests under the canonical zagg-index/ATL03/007/ prefix.
  4. Read-back: open the Zarr outputs, spatial cell map, t-digest → PDF reconstruction compared against the gain_bias 128-bin waveform.
• notebooks/sponsor_read_only.ipynb — binder-runnable, no credentials: anonymous reads of the published Zarr stores (plus a sidecar-manifest fetch demo), reproducing the read-back plots. Binder badge wired per repo convention.

Environment premise

Notebooks are written against zagg ≥ 0.14 (main + #163 virtual-index backends + #165 morton location channel — both now merged; 0.14.0 is the release the notebooks assume) and h5coro-hidefix ≥ 0.2. Anything not yet released is flagged in a provisional cell. Lambda invocation cells document the deployed path; the actual execution run is local/in-process until 0.14.0 deploys.

Phases

• 1. Scaffolding + catalog & shardmap sections (executed)
• 2. Dual aggregation runs (executed, outputs in bucket)
• 3. Read-back / PDF / waveform sections
• 4. Notebook B + binder wiring
• 5. Polish + full top-to-bottom re-execution of both notebooks + self-review folds