π€ from Claude
Combined layout migration (per espg's call: one issue, phased β both phases change S3 key shapes, so landing them together is one wire-format break instead of two). Design record: docs/design/sparse_coverage.md Β§2βΒ§3 (D1βD6, D12), tracked on #198; convention spec frozen mortie-side (espg/mortie#48 β espg/mortie#62).
Blocked by espg/mortie#104 β needs the decimal emit / scalar / hive_path surface, released to PyPI (we pin mortie>=0.8.5; this will bump).
Phase 1 β decimal shard-id strings (existing flat layout)
Shard ids surface externally as raw packed-word integers today; route them through the decimal repr:
- CSR subgroup paths:
{group_path}/{field}/{shard_key}/β¦ (processing/write.py:583-598)
- Async status objects:
<store>.status/<run>/{shard_key}.json (runner.py:1403)
- Worker/dispatcher logging (
worker.py:208 etc.)
--morton-cell / catalog stringified-key round-trips (runner.py:899-926)
Hard break, no migration shims (pre-1.0; existing test stores become debris).
Phase 2 β hive tree behind a config flag
Acceptance
Related: #197 (dispatch shuffle, orthogonal), #198 (design doc), follow-on MOC write issue (blocked by this).
π€ from Claude
Combined layout migration (per espg's call: one issue, phased β both phases change S3 key shapes, so landing them together is one wire-format break instead of two). Design record:
docs/design/sparse_coverage.mdΒ§2βΒ§3 (D1βD6, D12), tracked on #198; convention spec frozen mortie-side (espg/mortie#48 β espg/mortie#62).Blocked by espg/mortie#104 β needs the decimal emit / scalar /
hive_pathsurface, released to PyPI (we pinmortie>=0.8.5; this will bump).Phase 1 β decimal shard-id strings (existing flat layout)
Shard ids surface externally as raw packed-word integers today; route them through the decimal repr:
{group_path}/{field}/{shard_key}/β¦(processing/write.py:583-598)<store>.status/<run>/{shard_key}.json(runner.py:1403)worker.py:208etc.)--morton-cell/ catalog stringified-key round-trips (runner.py:899-926)Hard break, no migration shims (pre-1.0; existing test stores become debris).
Phase 2 β hive tree behind a config flag
{store_root}/{sign+base}/{d1}/β¦/{full_id}.zarrβ one digit per level to shard order, full-id leaf (D2/D3).morton_hive.jsonwritten once at template time:spec: morton-hive/1, dataset identity, cell/shard orders, split schedule, pyramid declaration (D6).group.attrs.update(...)(completion flag + cell count + timestamp + granule count). A.zarr/prefix without the stamp is debris, safe to overwrite on retry (D4). This is not consolidated metadata β one tiny PUT on an object that exists anyway.[1-4]/children and*.zarrobjects (D5 β zero metadata above the leaf; see Sharded CSR writes issue a per-array metadata-op storm β upstream ask: a no-probe / batch array-creation path in zarr-pythonΒ #189/Parallelize the opt-in metadata-consolidation path (deferred from #191/PR #193)Β #194 for why).Acceptance
Related: #197 (dispatch shuffle, orthogonal), #198 (design doc), follow-on MOC write issue (blocked by this).