Skip to content

Adopt morton decimal shard ids + hive-partitioned output layout (manifest, commit stamp)Β #199

Description

@espg

πŸ€– from Claude

Combined layout migration (per espg's call: one issue, phased β€” both phases change S3 key shapes, so landing them together is one wire-format break instead of two). Design record: docs/design/sparse_coverage.md Β§2–§3 (D1–D6, D12), tracked on #198; convention spec frozen mortie-side (espg/mortie#48 β†’ espg/mortie#62).

Blocked by espg/mortie#104 β€” needs the decimal emit / scalar / hive_path surface, released to PyPI (we pin mortie>=0.8.5; this will bump).

Phase 1 β€” decimal shard-id strings (existing flat layout)

Shard ids surface externally as raw packed-word integers today; route them through the decimal repr:

  • CSR subgroup paths: {group_path}/{field}/{shard_key}/… (processing/write.py:583-598)
  • Async status objects: <store>.status/<run>/{shard_key}.json (runner.py:1403)
  • Worker/dispatcher logging (worker.py:208 etc.)
  • --morton-cell / catalog stringified-key round-trips (runner.py:899-926)

Hard break, no migration shims (pre-1.0; existing test stores become debris).

Phase 2 β€” hive tree behind a config flag

Acceptance

Related: #197 (dispatch shuffle, orthogonal), #198 (design doc), follow-on MOC write issue (blocked by this).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions