Benchmarks: full-AOI NEON runs, re-pin temporal to full catalog (→ 2026-03-15), reset live series, CONUS cost model

[espg]

🤖 from Claude

Filed from the hive-layout discussion (espg/mortie#48 → #198/#199); espg will follow up with more detail. This issue also absorbs the write-throughput acceptance criterion from #199 (validating shuffle #197 + hive layout under real concurrency belongs here, with the benchmarks, not as a #199 checklist item).

Context

The live matrix (#193, just landed — only 1–2 datapoints recorded so far, so now is the cheap moment to reset) measures one pinned densest shard per target (tests/data/benchmark/targets.json; "densest = most granules, lowest-key tiebreak"), with the temporal pin 2018-10-13 → 2025-06-01. That isolates code deltas well, but it can't answer the questions we now need: what does a full AOI cost, and what would CONUS cost.

Scope

1. Full-AOI NEON target(s). Dispatch all shards over the AOP_NEON.geojson box, not just the pinned densest cell. Record: total wall / lambda-seconds / cost, the per-shard distribution (granules, runtime, cost, RSS — this feeds the regression in (4)), and AOI-mask build time as its own recorded number.
2. Temporal re-pin to the full catalog. 2018-10-13 → 2026-03-15 (current end 2025-06-01 truncates the mission). Rebuild the NEON shardmaps over the new window; pinned densest shards and granule counts will shift (the drift test already tolerates tie reselection — extend as needed). 88S stress pins (add 88 south stress test shard + benchmark #148) keep their own temporal unless espg says otherwise.
3. Reset the live series. Clear the 1–2 datapoints on the just-updated matrix/figure and start the corrected-metrics series from zero, so the tracked history is internally consistent from day one.
4. CONUS cost model. The deliverable is a written, realistic estimate before anyone runs it:
   • Regression from the NEON full-AOI per-shard data: granules-per-shard → lambda-seconds / cost (and memory, for sizing).
   • Build the CONUS shardmap over the full temporal window → per-shard granule counts for the real domain (~50k order-9 shards).
   • Apply the regression → estimated total lambda-seconds + dollar cost, plus operational costs (S3 PUT/GET volume, CMR/catalog build, status-channel traffic).
5. Write-throughput validation (from Adopt morton decimal shard ids + hive-partitioned output layout (manifest, commit stamp) #199): the full-AOI runs — and eventually the scoped CONUS run — exercise dispatch shuffle (Shuffle shard dispatch order before fan-out to spread S3 prefix load #197) + hive layout (Adopt morton decimal shard ids + hive-partitioned output layout (manifest, commit stamp) #199) at real concurrency. Record throttling/503 SlowDown counts alongside the cost numbers.

Coordination notes

• The pinned shard_key values in targets.json are raw packed-word integers (e.g. 5347394812217655307); under Adopt morton decimal shard ids + hive-partitioned output layout (manifest, commit stamp) #199 phase 1 these become morton decimal strings — sequence the re-pin so the shardmap rebuild only happens once.
• Full-AOI runs are credentialed + billed — gated dispatch, not CI-automatic, same as the existing matrix.

Questions for espg

1. Does full-AOI replace the one-shard pins, or run alongside them? (One-shard isolates code deltas from data drift; full-AOI measures cost truth. Keeping both — one-shard for the regression-tracking series, full-AOI recorded per release or on demand — seems defensible.)
2. Which live-matrix axes carry into the reset series (the Skip zarr metadata consolidation by default (opt-in) #193 inline/sidecar A/B over o9/o10? tdigest only, or gain_bias too)?
3. What counts as "operational cost" for the CONUS estimate — S3 requests + CMR + CloudWatch/logs, or a broader accounting?

[espg]

Does full-AOI replace the one-shard pins, or run alongside them? (One-shard isolates code deltas from data drift; full-AOI measures cost truth. Keeping both — one-shard for the regression-tracking series, full-AOI recorded per release or on demand — seems defensible.)

yeah, seems reasonable; full AOI recorded per release. We do need a separate plot that's just the release full neon aoi numbers in benchmark.md though.

Which live-matrix axes carry into the reset series (the #193 inline/sidecar A/B over o9/o10? tdigest only, or gain_bias too)?

sidecar vs inline, AOI mask vs no AOI mask, o9 size shards only. All the other options are fixed (sharded codec, K, tdigest). Per release plot for the neon AOI, and per merge to main plot for the single shard case we keep.

What counts as "operational cost" for the CONUS estimate — S3 requests + CMR + CloudWatch/logs, or a broader accounting?

definitely lambda GB/second. Yes, include the other items too as there own columns in the estimate... and this is an estimate, not a benchmark (i.e., we aren't running conus yet). include summary stats (sq km, granules, shards, etc), and the polygon reference that's used to build the shard map (which we can build)

[espg]

🤖 from Claude

Per espg (approved in-session): this issue also absorbs the shard-MOC sizing measurement that decides O8 on #200 (stamp-riding MOC representation + budget — see the design amendment). Adding it as scope item (6). It is pure local/offline work — no credentialed dispatch, no fleet run — so it can execute before items (1)–(5), by a local agent with the offline mission-length ATL03 STAC geoparquet.

(6) Shard-MOC sizing measurement (decides #200 O8, informs O9)

Question to answer: for the per-shard cell-level coverage MOC that rides the D4 commit stamp, pick between (a) byte-budgeted coarsen-to-fit MOC ranges and (b) a snap relative-depth bitmap — and pick the budget/depth constant.

Shard selection (from the offline mission geoparquet):

• SERC/NEON: the pinned densest o9 shard from tests/data/benchmark/targets.json (local granule cache from the PR End-to-end sponsor example: catalog → shardmap → dual aggregation → cached reads #169 work should cover it).
• 88S worst case: the o9 and o10 shards with maximum granule count on the 88°S ring (reuse the add 88 south stress test shard + benchmark #148 catalog snapshots if present; otherwise select via the geoparquet). Track data at the pole hole is the adversarial case for range fragmentation — the whole point of measuring.

Per shard, from real photon data (h5coro reads, local caches where available):

1. Compute cell-order (order-19) morton words for all photons — the same geo2mort path the worker uses — and dedupe to the occupied-cell set. Record photon count and occupied-cell count.
2. Occupancy by depth: occupied-cell counts at relative depths d = 3…(19 − shard_order), i.e. absolute orders shard+3 … 19.
3. Exact MOC per depth (mortie 0.9.0 compressed multi-order coverage — the depth ceiling is lifted, order 19 is representable): member count and serialized size two ways — (i) JSON nested-range pairs with string endpoints (the O1 convention), (ii) packed-word array bytes. Measure the size as it would sit in zarr.json attrs (JSON/base64 inflation counts against the budget, not raw bytes).
4. Coarsen-to-fit: for byte budgets {1, 2, 4, 8, 16} KB — the deepest depth whose MOC fits, and the resulting over-coverage ratio (conservative-superset area ÷ true occupied area at cell order). That ratio is the read-side filtering power we keep at each budget.
5. Bitmap alternative: raw sizes are analytic (4^d bits: shard+5 → 128 B, shard+7 → 2 KB, …); report zstd-compressed sizes at each d for the real occupancy patterns so compressed-bitmap vs ranges is an honest comparison.
6. Worker-side build cost: wall time from in-memory morton coordinate array → serialized payload (must be negligible vs shard processing; confirm ms-scale).
7. Optional, cheap: union timing of 4 sibling MOCs at each representation (the sweep's composition step — expected microseconds; one line confirming so).

Decision rule sketch (so the numbers land in a recommendation, not just a table): if the 88S worst case still fits ≤4 KB as ranges at a useful depth (≥ shard+5), budgeted ranges win — human-readable, mortie-native algebra, consistent with O1. If 88S range lists blow past sane budgets at shallow depths while compressed bitmaps stay small and fixed, the snap bitmap wins. The over-coverage ratios set the floor depth below which the MOC stops being a useful pre-filter.

Deliverable: table + recommendation posted on #200 (cross-linked here), with the measurement script committed somewhere rerunnable (.github/scripts/ next to run_benchmark.py seems natural — espg's call).

O9 rider for the fleet runs later: when item (1)'s full-AOI runs happen, record the end-of-run root-MOC Event-invoke round trip (dispatcher-side post latency + invoke duration) so the O9 default-on/off call has its wall-time number.

---

Generated by Claude Code

[espg]

🤖 from Claude

Three espg-approved additions to scope item (6) (the shard-MOC sizing measurement):

1. Report over-coverage in STAC simplification-error terms: alongside the raw ratio (covered ÷ true occupied area), report the cell-edge length at the achieved depth as the error-tolerance analog — the same way stactools reports footprint simplification — so the numbers translate directly if a zagg → STAC bridge materializes.
2. Tier-0 morton-box baseline: per shard, also compute the minimal ≤ 4-member MOC ("morton box" — see the tier design on #200 thread addendum) and report its over-coverage ratio. This is the baseline the budgeted MOC tier must beat at each byte budget — the deliverable becomes "how much filtering power does each KB buy over the 32-byte box," not just a size survey.
3. Budget grid: evaluate 1024 B and 2032 B explicitly as the candidate budgets (2032 = the PostgreSQL TOAST inline threshold, adopted as a portability bound; the boundary is inclusive — exactly 2032 stays inline in Postgres terms). Keep the {1, 2, 4, 8, 16} KB sweep for the curves, but the recommendation should land on one of those two.

---

Generated by Claude Code

[espg]

🤖 from Claude

Correction to item (6)'s budget grid, per espg: the budget is matched to S3/cloud conventions, not database-page conventions — drop the earlier "recommendation should land on 1024 B or 2032 B" directive (PostGIS is retained only as a portability footnote in the design doc). Expected scale is much larger: extend the sweep to {1, 4, 16, 64, 256} KB and 1 MB, plus the exact/uncapped size, and let the box-baseline comparison and over-coverage curves drive the recommendation with no anchor.

One consequence to report on explicitly: the budget picks the carrier. A KB-scale payload rides the commit-stamp attrs (zero extra objects, but bloats the leaf zarr.json that every metadata open GETs); a hundreds-of-KB tier becomes its own object inside the leaf with only the 32 B morton box + achieved depth + pointer in attrs (one extra GET, paid only by readers that opt in). So alongside the size table, report the leaf zarr.json size at each in-attrs budget — that's the read-side cost the carrier fork trades against.

Design-doc side is landed as draft PR #206 (O8/O9 recorded open, wired here); this issue's PR updates the doc as the numbers come in.

---

Generated by Claude Code

[espg]

🤖 from Claude

NEON-box granule selection: 158-vs-59 decomposed (issue #202, refs #65, #36)

Full trace + a reproducible three-way count comparison, a planar-vs-sphere-correct membership check, and the MOC-order sweep espg asked for. Scripts on branch claude/202-selection-investigation under bench/neon_*.

TL;DR

• Production was never wrong and is not drifting. make_shardmap is CMR-gated: CMRSource().fetch(query) sends the AOI bbox to CMR-STAC, which server-side subsets, and ShardMap.build only ever intersected that already-subset set. Rebuilding the pin from the CMR fetch reproduces 59 exactly (densest o9 shard = 59). So ShardMap.build does not over-assign on the CMR subset.
• The reported "158" is a planar-shapely antimeridian artifact, not a ShardMap.build count. A local shapely "footprint-polygon intersects AOI bbox" test returns 158 — but 103 of those are footprints whose stored WKB, read as a planar polygon, spans lon[-180, 180] (area 2300–3100 deg²) because the granule crosses the antimeridian/pole. This is exactly the WGS84-planar bug that got shapely removed as an intersection backend (Why are we using shapely as a backend at all? #36). The sphere-correct answer is far lower.
• Sphere-correct, the honest local-vs-CMR gap is only +10 (69 vs 59), and bbox-vs-polygon shape contributes 0 (box ≈ the 0.1° AOI).
• The order sweep espg asked for plateaus, it does not collapse: 99 granules flat at orders 9/12/13/16 while wall time roughly doubles per step. So for this AOI the residual over-assignment is not MOC-rasterization coarseness — it's the coarse Tighter catalog filtering: decompose CMR swath polygon into per-beam tracks #65 swath-envelope footprint plus the o9 coverage cells being wider than the 0.1° box. Raising the order buys nothing here except runtime.

(Q1) Drift or not — NOT drift

measurement	granules
(A) CMR bbox query	59
(C) CMR-fetch → ShardMap.build (the real production path)	59 (densest shard 59)
(B) full local catalog → ShardMap.build, auto order (13)	99
planar shapely footprint ∩ AOI bbox (the "158")	158 ← antimeridian artifact
sphere-correct footprint ∩ AOI bbox (mortie, order 14)	69
sphere-correct footprint ∩ AOI polygon (mortie, order 14)	69

(C) == (A) == the committed pin. The 158/99 patterns appear only when the full local catalog is fed to ShardMap.build directly — a new path the local catalog enables, not something production ever did.

(Q2) Fixable locally — the local parquet DOES reproduce CMR, sphere-correctly

• CMR's 59 is not "footprint bbox-intersects AOI bbox": that planar test gives 158 (wrong, antimeridian) and even the sphere-correct footprint∩bbox gives 69, a superset of CMR (all 59 ⊆ 69, missing=0). CMR searches on a finer internal orbit geometry than the coarse quarter-orbit envelope it emits in the STAC item — even a CMR-selected granule's stored footprint spans lat[27,59] (~5 deg²). So the residual +10 is the Tighter catalog filtering: decompose CMR swath polygon into per-beam tracks #65 coarse-envelope over-assignment, not a test-choice bug.
• "Intersects AOI bbox" vs "covers this o9 shard cell" answer different questions. A correct shard map needs the latter — a granule belongs in shard S iff its photons can land in S — so per-cell assignment is the right semantics, and it is arguably more correct than CMR's bbox membership. The o9 build's superset is therefore superset-by-design (o9 cells are ~13 km, wider than the 0.1° box) plus the Tighter catalog filtering: decompose CMR swath polygon into per-beam tracks #65 envelope slack, not a bug.
• A local Catalog source for make_shardmap (read parquet + build directly, no CMR round-trip) is viable: it reproduces CMR to within the Tighter catalog filtering: decompose CMR swath polygon into per-beam tracks #65 envelope slack as long as the intersection is sphere-correct (mortie or spherely) — never planar shapely. Sketch: add a Catalog/parquet fetch alongside CMRSource that a Query-equivalent bbox pre-filter narrows, then ShardMap.build as today.

(Q3) MOC-order sweep — it PLATEAUS; NO mortie change needed

Full local catalog → NEON o9 shard map, mortie_order swept (region = AOI polygon; floor of this build is the o9-cell region, ~13 km cells wider than the box):

 order  granules  densest   wall_s     (CMR pin = 59)
     9        99       82     2.01
    12        99       82    14.83
    13        99       82    30.05      <- production default (chunk_order)
    16        99       82   251.96

Contrary to the "raise the order and it collapses to 59" guess, the count plateaus at 99 for every order while wall time roughly doubles per step (and order 16 on these quarter-orbit envelopes costs ~250 s / ~125× order 9 for zero count benefit). So NEON is in the plateau regime: the residual over-assignment is not MOC-rasterization coarseness (a granule that genuinely overlaps the o9 region overlaps at least one of the 4 o9 cells at any order — refining the order only shuffles which of the 4 it lands in, not whether it's in the map). The lever that does move it is the footprint geometry: footprint="beams" (the #65 per-beam-corridor stopgap) narrows the coarse swath envelope so a granule stops being assigned to cells its beams never cross.

Decomposing the 99 → 59 gap:

• 99 → 69 (~30 granules): the o9 coverage cells span ~26 km (4 cells), wider than the 0.1° (~10 km) AOI box — mechanism (a). This is correct per-cell shard-assignment semantics (a granule assigned to an o9 shard whose cells the AOI only partly covers), just wider than a bbox query.
• 69 → 59 (~10 granules): the coarse Tighter catalog filtering: decompose CMR swath polygon into per-beam tracks #65 swath-envelope footprint (even a CMR-selected granule's stored polygon spans lat[27,59], ~5 deg²) reaches the AOI cells though CMR's finer internal orbit geometry excludes it.

mortie is fine (0.9.0, #103 fixed — experiments ran clean). No exact-polygon∩cell test or finer MOC would help — raising the order was shown not to reduce the count. The over-assignment is entirely (i) zagg's o9-region-vs-bbox granularity and (ii) the coarse CMR-provided swath-envelope footprint (#65); neither is a mortie defect. A mortie MOC-set-intersection helper would be a convenience, not a correctness requirement.

Recommendation

The local catalog is usable as-is for shard-map building, provided two rules hold: (1) intersect sphere-correctly (mortie/spherely) — never planar shapely, which the antimeridian wrap corrupts (#36); (2) accept that the result is a correct superset of a CMR bbox query — ~+40 here, ~30 from the o9-cell region being wider than the box (right shard-assignment semantics) and ~10 from the #65 swath envelope (tightened by footprint="beams", not by raising mortie_order — the sweep shows order doesn't move the count). No mortie change is required.

[espg]

🤖 from Claude

Beams@16 confirmation (the line I said was still running). footprint="beams" over the full local catalog at order 16:

footprint='beams' @order16: granules=95  densest=56  wall=627.32s

vs swath@16 (99 / densest 82 / 252 s) and the CMR pin (densest 59). Two things:

• The total barely moves (99 → 95) but the densest shard — the benchmark-relevant quantity — drops 82 → 56, i.e. below CMR's 59. Per-beam corridors drop granules whose swath envelope grazed the densest o9 cell but whose actual beams never cross it; a few re-land in neighbor shards, so the total is nearly unchanged while the peak collapses to CMR-level. This is the Tighter catalog filtering: decompose CMR swath polygon into per-beam tracks #65 mechanism working exactly as intended, and arguably more correct than CMR bbox membership on that cell.
• It cost 627 s (~10.5 min), ~2.5× swath@16 and ~300× order-9 swath, for the full candidate set. So beams is the right correctness lever for the densest-shard pin but is expensive at high order — reinforcing that the practical recipe is a modest mortie_order (the plateau means 9-13 give the same counts) with beams only when the peak-shard tightness matters.