Parallelize the opt-in metadata-consolidation path (deferred from #191/PR #193)

[espg]

🤖 from Claude

Summary

Follow-up to #191 / PR #193, which made zarr.consolidate_metadata opt-in (output.consolidate_metadata: true, default false). When it is requested, consolidation still reads the store's ~4,102 zarr.json metadata objects serially (~17 ms/GET → ~70 s on the o9 NEON store). This tracks making the opt-in path fast so it isn't a foot-gun.

Why it was deferred (not done in PR #193)

zarr==3.2.1's consolidate_metadata(store, path=None, zarr_format=None) exposes no concurrency parameter — the only lever is the global zarr.config['async.concurrency'] (default 10). Whether raising that actually parallelizes the ~4,102-object serial-GET path can't be verified without the production store the #180 measurements ran against. Per #191's own guidance ("if it doesn't expose concurrency cleanly, don't fight it — note the finding and defer; do not add a dependency"), PR #193 shipped the default-skip and deferred this. The opt-in path is off by default, so today this is latent, not a live foot-gun.

Options to evaluate

1. Measure zarr.config['async.concurrency'] against a real consolidated run — does bumping it from 10 collapse the ~70 s? If so, the fix is a one-line config set inside the opt-in branch (no new code, no dep). Cheapest if it works.
2. Bounded-pool reimplementation — enumerate the metadata objects and fetch them through a ThreadPoolExecutor (or a concurrent obstore batch), then assemble the consolidated blob. More code, but independent of whatever consolidate_metadata does internally. Reuses the store/obstore stack already in the repo — no new dependency.
3. Upstream — if zarr's own path is the bottleneck and async.concurrency doesn't reach it, note it for a zarr-python feature ask (a concurrency arg on consolidate_metadata), alongside the metadata-op-storm ask on Sharded CSR writes issue a per-array metadata-op storm — upstream ask: a no-probe / batch array-creation path in zarr-python #189.

Acceptance (when picked up)

• output.consolidate_metadata: true still produces a valid consolidated store, now via parallel metadata reads.
• Measured wall-time drop on a representative store (target ~70 s → single-digit seconds).
• No new dependency added without discussion (§4).
• Read-parity unaffected (the consolidated blob's contents are unchanged; only how it's assembled).

Refs: #191 (opt-in decision), PR #193 (the default-skip that landed), #180 (the ~70 s serial-read measurement + wall-time breakdown), #189 (the related write-side metadata-op storm).

Filed for triage — no action label applied; label implement/plan/discuss to bring it into scope.

[espg]

I feel like this is irrelevant and can be closed given #201 and #198 -- thought (disagree, concur)?

[espg]

🤖 from Claude

Concur — close it (your question).

• This issue's premise was that consolidate_metadata is a finalize-path product worth making fast. Under the Design: sparse coverage convention + cross-resolution read path (MOC domain, hive stores, xdggs extension) #198 design that premise is gone: D8 replaces consolidated metadata for extent/discovery with the store MOC (the +70 s/worker measurement is cited in the design doc as the reason), and D12 defers any zarr-native hierarchy or consolidated index to the §7 sweep as an optional derived cache — round one ships without it.
• The one thing worth preserving: if the Pyramid/overview generation sweep (second pass): overview zarrs, MOC regen, optional interop materialization #201 sweep ever does materialize a consolidated index (the D12 escape hatch), the Granule-level read concurrency: the worker's serial granule x group loop is the dominant read-wall term #180 measurement (~4,100 serial zarr.json reads ≈ 70 s) transfers to that sweep step. I've carried that note into my Pyramid/overview generation sweep (second pass): overview zarrs, MOC regen, optional interop materialization #201 comment so it isn't lost — it's an implementation note there, not a live issue here.
• None of this issue's options ((1) async.concurrency bump, (2) bounded-pool reimplementation, (3) upstream zarr-python concurrency ask) is worth spending on a path that is opt-in, default-off, and slated to become a post-process artifact.

Recommend closing as superseded by #198/#201 — the close itself is yours. Note the write-side twin (#189) has a different answer under the new design; posted there.

---

Generated by Claude Code