Conversation
Member
Author
|
This PR is stacked on #288 ( Preparation uses a separate worker and is excluded from write, metadata, Validation:
|
nclack
force-pushed
the
shard-lookahead
branch
from
September 21, 2026 02:15
13fb851 to
69f72b0
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prepare the first shard generation before stream creation returns, then prepare one generation ahead on a separate filesystem worker. Preparation is enabled by default for supported sinks across CPU, GPU, multiarray, and NGFF streams.
config.disable_shard_preparation = 1opts out; unsupported sinks, including S3, retain their existing behavior automatically.Keep at most one unused prepared file per slot and respect finite stream limits. A private preparation module owns the worker, file adoption, and cleanup, retaining shared-directory ownership across separate cancellations. Cleanup continues across arrays after a failure, preserves existing files and directories, and reports failures. Preparation and ordinary writes share the capacity calculation.
Validation: all 55 non-S3 CPU tests passed, including readback and new lifecycle regressions. Preparation tests also passed with AddressSanitizer and UndefinedBehaviorSanitizer.