Skip to content

Prepare shard files ahead of writes by default - #290

Open
nclack wants to merge 2 commits into
mainfrom
shard-lookahead
Open

nclack wants to merge 2 commits into
mainfrom
shard-lookahead

Conversation

@nclack

@nclack nclack commented Sep 18, 2026

Copy link
Copy Markdown
Member

Prepare the first shard generation before stream creation returns, then prepare one generation ahead on a separate filesystem worker. Preparation is enabled by default for supported sinks across CPU, GPU, multiarray, and NGFF streams. config.disable_shard_preparation = 1 opts out; unsupported sinks, including S3, retain their existing behavior automatically.

Keep at most one unused prepared file per slot and respect finite stream limits. A private preparation module owns the worker, file adoption, and cleanup, retaining shared-directory ownership across separate cancellations. Cleanup continues across arrays after a failure, preserves existing files and directories, and reports failures. Preparation and ordinary writes share the capacity calculation.

Validation: all 55 non-S3 CPU tests passed, including readback and new lifecycle regressions. Preparation tests also passed with AddressSanitizer and UndefinedBehaviorSanitizer.

@nclack

nclack commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

This PR is stacked on #288 (borrow-shard-footers) to preserve the tested
footer implementation. Its diff contains the shard-preparation option and
its lifecycle, portability, documentation, and regression coverage.

Preparation uses a separate worker and is excluded from write, metadata,
and footer completion fences. Opening a shard adopts the prepared file;
it waits if preparation has not finished. Flush stops future preparation
before draining delivery, then waits for and removes unused owned files.
Cancellation preserves existing files and directories, reports failures,
and can be retried.

Validation:

  • The initial implementation passed 53 CPU tests and all 29 GPU tests.
  • After correcting compressed prepared capacity, the final implementation
    passed three focused CPU tests and two GPU tests, including Zarr
    readback and the raw/compressed initial-resize regressions.
  • Tests cover empty, partial, complete, finite, and unbounded streams;
    explicit flush/close and destruction; prepared-file adoption; queue
    independence; existing-file preservation; and retryable cleanup.
  • Every submitted implementation file matches that final tested version.

@codecov

codecov Bot commented Sep 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.59504% with 114 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/zarr/shard_preparation.c 68.51% 23 Missing and 28 partials ⚠️
src/zarr/zarr_array.c 59.64% 12 Missing and 11 partials ⚠️
src/zarr/shard_pool_fs.c 71.42% 6 Missing and 8 partials ⚠️
src/multiarray/stream.c 72.22% 2 Missing and 3 partials ⚠️
src/zarr/shard_delivery.c 75.00% 2 Missing and 3 partials ⚠️
src/zarr/io_backend.fs.c 63.63% 2 Missing and 2 partials ⚠️
src/cpu/stream.c 40.00% 1 Missing and 2 partials ⚠️
src/ngff/ngff_multiscale.c 82.35% 1 Missing and 2 partials ⚠️
src/platform/platform_io.posix.c 70.00% 0 Missing and 3 partials ⚠️
src/cpu/stream.body.c 71.42% 1 Missing and 1 partial ⚠️
... and 1 more

❌ Your patch check has failed because the patch coverage (68.59%) is below the target coverage (70.00%). You can increase the patch coverage or adjust the target coverage.

Files with missing lines Coverage Δ
src/platform/platform_io.h 100.00% <ø> (ø)
src/zarr/shard_write_plan.c 64.39% <100.00%> (-0.03%) ⬇️
src/writer.c 85.00% <80.00%> (-0.46%) ⬇️
src/cpu/stream.body.c 82.35% <71.42%> (-0.41%) ⬇️
src/cpu/stream.c 81.61% <40.00%> (-0.39%) ⬇️
src/ngff/ngff_multiscale.c 64.20% <82.35%> (+1.28%) ⬆️
src/platform/platform_io.posix.c 72.46% <70.00%> (+0.79%) ⬆️
src/zarr/io_backend.fs.c 75.82% <63.63%> (-0.79%) ⬇️
src/multiarray/stream.c 78.22% <72.22%> (+1.59%) ⬆️
src/zarr/shard_delivery.c 73.91% <75.00%> (+0.10%) ⬆️
... and 3 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Base automatically changed from borrow-shard-footers to main September 20, 2026 23:44
@nclack nclack changed the title Prepare shard files ahead of writes Prepare shard files ahead of writes by default Sep 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant