Design: sparse coverage convention + cross-resolution read path (MOC domain, hive stores, xdggs extension)

[espg]

🤖 from Claude

Design-doc tracker. The full document lives at docs/design/sparse_coverage.md and is iterated in git; this issue is the discussion thread and pointer. (Doc is staged locally at time of filing; it lands on main with the next docs commit.)

Summary

Three converging pressures, one set of primitives. Grew out of the hive-layout discussion on espg/mortie#48:

1. S3 write contention at global scale → hive-partitioned output stores keyed by morton decimal ids ({root}/{±base}/{d1}/…/{full_id}.zarr, one digit per level, full-id leaf).
2. Multi-dataset, multi-resolution joins (ICESat-2 ~12 m cells, GEDI one order coarser, Sentinel-2 finer) → per-dataset stores; interop between any pair is morton truncation (string-prefix = ancestor), so cross-resolution lookup is arithmetic, not I/O.
3. xdggs materializes dense full-sphere coordinates — intractable for sparse-coverage data → a store-level MOC as the domain declaration ("data exists on these cells"), keeping coordinates sparse with a lazily fabricated dense view.

Key design points (rationale + decision registry in the doc):

• Write path stays dumb: prefix tree + vanilla zarr v3 leaves, zero metadata above the leaf, commit-stamp via final root-attrs update (this is not consolidated metadata — no per-store aggregation, one tiny PUT per shard).
• Store MOC replaces consolidated metadata for extent/discovery: built from the dispatcher's own completion list (no S3 walk — milliseconds, not the 70 s/worker consolidation cost), one small object, regenerable cache; the strongly-consistent delimiter-LIST tree walk remains ground truth.
• Reader flow: manifest GET → MOC intersect against query AOI → arithmetic hive paths → leaf opens; LIST-walk only as fallback/discovery (termination: a node with no digit children has nothing finer).
• Pyramid/overview generation is a second-pass sweep (role: overview zarrs at ancestor nodes), never write-time.
• Upstream trajectory: this is the broad scope of what should eventually land in xdggs (or a standalone xarray extension) — sparse DGGS domain (MOC) + morton coordinate (relates to Upstream (xdggs): support MortonIndexDtype as a native DGGS coordinate #72).

Relationship to other work

• Hive layout spec itself is frozen mortie-side (Version 1.0 Sweep (Follow up to new morton datatype) espg/mortie#48 discussion → v1.0 docs: specification & conventions page (bit-layout + resolution/Δ table + frozen-convention statement) espg/mortie#62 spec page); this doc owns the zagg/reader/xarray side.
• Shuffle shard dispatch order before fan-out to spread S3 prefix load #197 (dispatch shuffle) is the orthogonal throughput fix.
• The read-path implementation issue(s) will be carved out of this doc once it stabilizes.

Proposed label: design (new — sits one level above discuss: iterate the doc, no code).