Write store coverage MOC at end of run (dispatcher-side domain declaration)

[espg]

🤖 from Claude

Design record: docs/design/sparse_coverage.md §4 (D8/D9), tracked on #198. Blocked by #199 (needs the store-root conventions: morton_hive.json, root-object exception to the node invariant).

What

At end of run, the dispatcher writes {store_root}/coverage.moc — the store's populated-shard coverage as a MOC (mortie's compressed multi-order coverage). This is the roll-our-own replacement for consolidated metadata (disabled by default; measured +70 s/worker), answering "where does data exist?" in one GET.

How (the fast part)

No S3 walk. The dispatcher already tracks per-shard completion via the status channel poller (runner.py _async_result_poll); build the MOC from that completion list in memory (existing morton_coverage/MOC machinery — milliseconds) and PUT one small object. Spatially coherent coverage compresses well (~50k CONUS shards → few KB).

Semantics

• Cache, not truth (D9): stamped with generated_at + source (dispatcher vs sweep); regenerable from a tree walk. The strongly-consistent delimiter-LIST walk remains ground truth — a run that crashes before writing its MOC degrades to walking, never to wrong answers.
• Readers: GET manifest → GET MOC → moc_and against query AOI → arithmetic hive paths. Zero LISTs on the hot path (D10).

Open questions (from the design doc)

• O1: serialization format for .moc (JSON nested-range pairs? packed-word .npy?) — needs freezing alongside the mortie spec; FITS/IVOA explicitly out (RangeMOC / moc-crate adoption decision (split from #35 Phase 3) espg/mortie#50).
• O2: mortie MOC depth ceiling MAX_DEPTH = 18 (v1.0: mixed-order MOC semantics  espg/mortie#61) — fine for shard-order coverage (≤ 11 today), blocks cell-order domains at 19+.
• O7: staleness policy — do readers warn, re-walk, or trust when the MOC predates recent writes?

[espg]

O1: serialization format for .moc (JSON nested-range pairs? packed-word .npy?) — needs freezing alongside the mortie spec; FITS/IVOA explicitly out (espg/mortie#50).

lean on this? mine is slightly as a np array, but could be convinced on a json nested-range pairs (especially since we don't have a numpy morton datatype yet)

O2: mortie MOC depth ceiling MAX_DEPTH = 18 (v1.0: mixed-order MOC semantics espg/mortie#61) — fine for shard-order coverage (≤ 11 today), blocks cell-order domains at 19+.

we need to support cell orders past order 19... is this a question on shard order / zarr.json metadata order limit? I doubt we'll have zarr stores / chunks past order 18 anytime soon... but I'd probably allow for it in the future. Can we set a cieling / max depth param as an over-ridable default?

O7: staleness policy — do readers warn, re-walk, or trust when the MOC predates recent writes?

good question-- I have some thoughts, what are yours?

[espg]

🤖 from Claude

Taking the three in turn (your comment):

O1 (format): recommend JSON nested-range pairs for round one — with the hedges that make it cheap to change your mind later. (1) Compactness isn't the constraint: spatially coherent coverage is a few KB either way (§4). JSON buys curl-able/greppable debugging next to morton_hive.json and — your own point — avoids freezing a dtype commitment while the numpy morton dtype (#72) is unsettled; an .npy of packed words bakes today's in-memory repr into the wire format. (2) One trap to spec around either way: packed u64 endpoints exceed 2^53, so raw JSON numbers get silently mangled by any float-based parser — range endpoints must be strings (decimal-morton strings would make ranges human-readable and consistent with D1's "strings are the external form"). (3) D9 makes this a two-way door: the MOC is a regenerable cache behind a versioned envelope (spec: morton-moc/1), so migrating to packed binary later costs one sweep regen, not a store migration. If a profile ever shows MOC parse time mattering, that's the moment .npy wins — I don't expect it at this size.

O2 (depth ceiling): to your clarifying question — no, this isn't the shard order or anything in zarr.json; it's mortie's MOC machinery itself (MAX_DEPTH = 18, espg/mortie#61) capping the orders representable in a MOC. Shard-order coverage (≤11 today) never gets near it; it binds only if a store wants to declare a cell-order domain (order-19 ICESat-2 cells) as a MOC — the §6 "domain = MOC" use, not coverage.moc. Concur on the overridable ceiling param: zagg-side, the MOC envelope should carry its order/depth explicitly rather than assume a constant; the actual >18 lift is mortie-side and can trail. Round one dodges it entirely by declaring shard-order coverage only.

O7 (staleness), my position: trust silently on the hot path, detect lazily, regenerate explicitly.

• Envelope: generated_at, source: dispatcher|sweep, producing run id.
• The MOC is rewritten as the last act of every producing run (that's D8's construction), so steady state is never stale; staleness means a crashed run or out-of-band writes.
• Readers trust silently by default. The only failure mode is a false negative — a shard exists that the MOC doesn't list — which degrades to "reader doesn't see the newest run," never a wrong answer (D9).
• Lazy detection over proactive verification: if a reader opens a commit-stamped leaf the MOC says shouldn't exist (or a walk finds one), warn and suggest regen. No automatic re-walk on the hot path — that's exactly what D10 forbids.
• Explicit escape hatches: reader-side refresh=True rebuilding from the tree walk, and the §7 sweep regenerating unconditionally.
• One knob I would not add: a wall-clock staleness horizon ("warn if older than N days"). Campaign stores sit legitimately untouched for months; age isn't evidence here — missing completions are.

The place O7 gets sharp, and where I'd like your thoughts: incremental runs. The dispatcher builds from its own completion list, so a second run appending to an existing store would clobber prior coverage if it blindly overwrites. Options: (a) GET existing MOC, union, PUT (one extra GET, still no walk); (b) always rebuild via walk at end of run (safe, slow); (c) dispatcher writes union per (a), sweep remains the authoritative rebuilder. I lean (c).

---

Generated by Claude Code