Pyramid/overview generation sweep (second pass): overview zarrs, MOC regen, optional interop materialization

[espg]

🤖 from Claude

Design record: docs/design/sparse_coverage.md §7 (D11/D12), tracked on #198. discuss — the write-time layout (#199) is settled; this second-pass process has genuinely open design questions.

Settled framing (from the design discussion)

• Overview generation is a separate post-process sweep, never write-time — it aggregates across worker-shard boundaries, so shard workers can't produce it anyway.
• Overview zarrs live at ancestor nodes of the hive tree, explicitly marked (role: overview + source order + aggregation method in attrs). Never inferred from position: a shallow zarr may equally be coarse source in a sparse region.
• Full pyramid storage cost is geometric ~1/3 extra (4 children per order).
• The sweep is idempotent and non-load-bearing: it can fail or lag without corrupting anything. It also owns MOC regeneration and any optional derived interop artifacts (e.g. a zarr.open-able root hierarchy or consolidated index — the D12 escape hatch, generated as a cache, never required).

To discuss

1. Aggregation semantics per field: mean/count compose trivially up-tree; t-digests merge natively; which fields degrade (e.g. located/ragged CSR fields — summarize, subsample, or omit at overview orders?).
2. Level schedule: every order up to base? Sparse levels (e.g. every 2–3 orders)? Declared in morton_hive.json's pyramid section — who writes that declaration, template time or sweep time?
3. Orchestration: same Lambda fan-out machinery over ancestor cells (bottom-up levels, each level reading the one below), or a coarser local/batch job? Reuses the leaves-to-root construction argument from docs/design/architecture.md.
4. Trigger: manual CLI, end-of-run hook from the dispatcher, or scheduled?
5. Walker/read interplay: display readers may stop at an overview; analysis readers must not conflate it with source — is the role attr enough, or does the manifest also enumerate overview orders for zero-open filtering?

[espg]

🤖 from Claude

Positions on the five open questions, grounded in docs/design/sparse_coverage.md §7 / D11 / D12, then risks.

(1) Degrading fields: omit by default, per-field override. Composable stats (count/sum/min/max, mean via sum+count) compose exactly and t-digests merge natively — those populate overviews. Located/ragged CSR fields should be omitted at overview orders by default: subsampling invents a lineage problem (which observations survive, and why?) and re-inflates the ~1/3 geometric storage bound with exactly the fields that dominate leaf size. Offer a per-field summarize override that emits derived scalars only (e.g. observation count / density per cell). Since every overview field carries its aggregation method in attrs (D11), omission is discoverable, and a reader needing raw observations is by definition a source-order reader.

(2) Schedule: sparse — every 2 orders — declared at template time; keep the manifest static. Every-order pyramids buy only a 4× reduction per step at the full ~1/3 storage cost; every-2-orders (16× per step) covers zoom/browse at roughly 1/15 extra. On "template time or sweep time": declare the schedule + methods in morton_hive.json at template time (it is config, and §3's zero-request reader bootstrap argues the manifest should be complete up front); the sweep should not rewrite the manifest — actuals (generated_at, source order, method) live in each overview zarr's own attrs. That keeps the manifest write-once and the sweep free of shared mutable state, the same discipline as D5.

(3) Orchestration: reuse the Lambda fan-out, bottom-up level-by-level, with backend="local" as the small-store path. Each overview cell within a level is independent (read its descendant leaves, aggregate, write one leaf + commit stamp) — exactly the existing worker shape; levels are sequential barriers the dispatcher already knows how to run as successive fan-outs. A separate batch-job path would be a second orchestrator to maintain for no capability gain.

(4) Trigger: manual CLI subcommand in round one. The sweep is idempotent and non-load-bearing (§7), so lagging is harmless; an end-of-run dispatcher hook (output.sweep: true) is a cheap follow-on once per-run cost is measured; scheduling belongs to ops tooling, not zagg.

(5) Walker/read interplay: let the MOC do the filtering; role stays the per-object truth. coverage.moc is built from the dispatcher's source-shard completion list (D8), so arithmetic readers driven by manifest+MOC never visit overview nodes at all — zero opens spent filtering. The manifest's pyramid section additionally enumerates which orders carry overviews (a useful walker hint), but it cannot be authoritative per-object: an order can host both overviews and coarse source in sparse regions (D11), so walkers at overview-carrying orders still check role on open. Two artifacts, two jobs: MOC = source domain; role = per-object classification.

Carried over from #194 (which I've recommended closing as superseded): if the D12 escape hatch ever materializes a consolidated index, it happens in this sweep — and the #180 measurement (~4,100 serial zarr.json reads ≈ 70 s) says build that step with a bounded concurrent reader from day one.

Risks/unknowns: (a) per-level sweep dispatch should shuffle like #197 — gentler than the leaf fan-out but same S3 partition physics; (b) quantile overviews require t-digests to exist at the leaves for those fields — fields aggregated by non-mergeable methods can only overview via re-reading child values, which is what the composable-stats-only rule avoids; (c) the incremental-store MOC question raised on #200 (merge vs rebuild) also decides whether this sweep is the sole MOC producer for stores written across multiple runs.

---

Generated by Claude Code