Sharded CSR writes issue a per-array metadata-op storm — upstream ask: a no-probe / batch array-creation path in zarr-python

[espg]

🤖 from Claude

Summary

The sharded ragged (CSR) write path generates far more S3 requests than its payload requires, because zarr-python's array-creation API re-probes and re-ensures the entire ancestor hierarchy on every open_array(mode="w") call. This was the request-rate fuel behind the 503 SlowDown failures root-caused on #186. PR #188 fixed the failure mode itself (paced retry backoff — decided as the (b) option on PR #188: retry-only, raise the op storm upstream). This issue tracks the op storm and the upstream ask.

Measurements (zarr 3.2.1, op-counting spy store)

One CSR subgroup write (write_csr: 3 arrays — values/offsets/cell_ids) is 33 store ops:

op	count	notes
GET	15	5 per open_array — target existence + a probe of every ancestor (zarr.json, {group}/zarr.json, {group}/{field}/zarr.json, {group}/{field}/{key}/zarr.json)
conditional PUT (set_if_not_exists)	12	4 per open_array — implicit parent group docs re-ensured per array, not per subgroup
PUT	5	3 array metadata docs + 2 chunk objects
DELETE	1	write_empty_chunks=False: an all-fill chunk (e.g. cell_ids == [0]) deletes instead of writing

18 of the 33 ops hit the same three shared hot keys (zarr.json, {group}/zarr.json, {group}/{field}/zarr.json). The o9 NEON shard that failed 3/3 on #186 has 256 subgroups: ~8,400 ops per shard, ~4,600 of them on those three keys — per worker, and the #186 reproduction ran multiple concurrent arms. A 4th array (locations, issue #87 located fields) raises all per-array counts proportionally.

What doesn't work on zarr's public API (both measured)

1. Caching an opened parent group and creating arrays through the handle (group.create_array(...)): identical per-create ancestor probing (same 4 GETs/array), and open_group(mode="a") additionally PUTs a group doc. There is nothing to cache — the probes are inside the per-create path.
2. zarr.create_hierarchy batches metadata properly (one probe pass + concurrent PUTs), but it writes explicit group docs for all implicit parents including the root — with overwrite=True it would clobber the template's root metadata written by emit_template. Layout-changing, not a drop-in.

What does work (prototyped, not landed — decided against for now)

Building ArrayV3Metadata locally from an in-memory template and PUTting metadata + chunk directly: 2 PUTs per array, zero GETs, byte-identical array docs and chunk objects. But it requires zarr.core internals (AsyncArray-from-metadata, StorePath, zarr.core.sync.sync) with no stability guarantee, skips zarr's ancestor-conflict checks (they become an unchecked invariant of zagg's layout), and takes ownership of subtle semantics (write_empty_chunks DELETE behavior, parent-doc creation). Per the decision on PR #188, zagg stays on the public API unless fleet evidence forces the escalation.

The upstream ask (zarr-python)

A public way to create many arrays without per-create ancestor traffic. Any of:

• an ensure_parents=False / check_ancestors=False flag on create_array (caller asserts the hierarchy exists — zagg writes into a pre-templated store);
• a create_hierarchy mode that creates only the named nodes, leaving implicit parents implicit (no root/parent group-doc writes);
• honoring an already-open parent Group handle as the hierarchy authority, so group.create_array skips re-probing the ancestors the handle already proved.

Filing this needs a maintainer call on framing (feature request vs. perf issue) and is a human-owned outbound step — parking the drafted context here so it can be lifted into a zarr-python issue verbatim.

Escalation trigger

If fleet runs still hit SlowDown exhaustion with PR #188's paced retries in place (the failure is loud: ragged (CSR) write failed ... 503 SlowDown after ~180 s of paced retries), revisit the internals-based writer (option (a) on PR #188) — the prototype and its byte-identity test approach are described there.

Refs #186 (root cause), PR #188 (retry fix + decision), #147 (parallel CSR writes), #108/#133 (ShardingCodec / sharded layout), #87 (located fields' 4th array).

[espg]

...how does this interact with #191 ?

[espg]

🤖 from Claude

Both this issue and #191 are the same root phenomenon seen from opposite ends of a metadata object's life: the sharded ragged/CSR write layout emits ~1,025 zarr metadata objects per shard (256 CSR subgroups × their arrays, plus the main arrays — 4,102 total on the 4-shard o9 NEON store).

• This issue (Sharded CSR writes issue a per-array metadata-op storm — upstream ask: a no-probe / batch array-creation path in zarr-python #189) = the write side. Every open_array(mode="w") re-probes and re-ensures the whole ancestor hierarchy → 33 store ops per 3-array CSR subgroup, ~8,400 ops/shard, ~4,600 of them on three shared hot keys. That request rate is the 503 SlowDown fuel (PR Paced S3 retry backoff for 503 SlowDown #188 paced it via retry backoff; Sharded CSR writes issue a per-array metadata-op storm — upstream ask: a no-probe / batch array-creation path in zarr-python #189 tracks the op storm itself + the upstream zarr-python ask for a no-probe / batch array-creation path).
• Skip zarr metadata consolidation by default: ~70s/run finalize tax no reader uses #191 = the read side. consolidate_metadata reads all 4,102 zarr.json objects serially (~70 s finalize tax). Skip zarr metadata consolidation by default: ~70s/run finalize tax no reader uses #191 (merged) skips consolidation by default.

How they interact:

1. Same scaling law. Skip zarr metadata consolidation by default: ~70s/run finalize tax no reader uses #191 notes the cost scales with (K inner chunks × ragged fields), not shard count; Sharded CSR writes issue a per-array metadata-op storm — upstream ask: a no-probe / batch array-creation path in zarr-python #189's op storm scales identically. A structural fix that cuts the CSR metadata-object count (fewer subgroups / batched array docs) shrinks both — fewer objects to probe-on-write and fewer to read-on-consolidate.
2. Complementary, and neither blocks the other. Skip zarr metadata consolidation by default: ~70s/run finalize tax no reader uses #191 already removed the consolidation read entirely by default, so the finalize half of the proliferation cost is gone. It leaves Sharded CSR writes issue a per-array metadata-op storm — upstream ask: a no-probe / batch array-creation path in zarr-python #189 untouched: even with consolidation skipped, workers still create the same ~1,025 metadata objects/shard, which is exactly what Sharded CSR writes issue a per-array metadata-op storm — upstream ask: a no-probe / batch array-creation path in zarr-python #189 targets. Skip zarr metadata consolidation by default: ~70s/run finalize tax no reader uses #191 is a zagg-side default flip; Sharded CSR writes issue a per-array metadata-op storm — upstream ask: a no-probe / batch array-creation path in zarr-python #189 is an upstream ask (plus PR Paced S3 retry backoff for 503 SlowDown #188's landed pacing).
3. One alignment consequence. Skip zarr metadata consolidation by default: ~70s/run finalize tax no reader uses #191 makes the non-consolidated store the default read path (its parity test covers it). So if Sharded CSR writes issue a per-array metadata-op storm — upstream ask: a no-probe / batch array-creation path in zarr-python #189's upstream ask ever lands a batch/no-probe create path that changes how metadata is written, the non-consolidated read path is the one that matters — the consolidated blob is no longer produced. The two changes point the same direction rather than conflict.

Net: #189 (write op storm, upstream) and #191 (finalize consolidation, merged) are two faces of the sharded-CSR metadata-object count. They land independently, and the highest-leverage move against both is reducing that per-shard object count itself.

---

Generated by Claude Code