Implement ZipStore delete via archive rewrite (fixes #828)#4085
Open
Akash-t25 wants to merge 3 commits into
Open
Implement ZipStore delete via archive rewrite (fixes #828)#4085Akash-t25 wants to merge 3 commits into
Akash-t25 wants to merge 3 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4085 +/- ##
==========================================
- Coverage 93.47% 93.31% -0.17%
==========================================
Files 90 90
Lines 11967 11994 +27
==========================================
+ Hits 11186 11192 +6
- Misses 781 802 +21
🚀 New features to boost your workflow:
|
…#828) Add tests for the auto-open branches in delete()/delete_dir(), the prefix-normalization branch in delete_dir(), and the temp-file cleanup path in _rewrite_without() (both when the temp exists and when it is already gone), bringing patch coverage to 100%. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
I worry that this will have counter-intuitive performance characteristics, especially since we don't have a transactional API that would support batching many deletes into one atomic archive resurrection. IF we want to support this delete implementation, it should probably be exposed via runtime store configuration. cc @mkitti |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements
delete()anddelete_dir()forZipStore, resolving #828.ZIP files don't support in-place deletion, so this PR uses an archive-rewrite strategy: copy every surviving member into a fresh temporary ZIP, drop the deleted entries, then atomically swap it in using
os.replace.Changes
src/zarr/storage/_zip.pysupports_deletes: bool = False→True_rewrite_without(should_delete)private helper that rewrites the archive atomicallydelete(key)— removes the key (missing key = no-op) instead of raisingNotImplementedErrordelete_dir(prefix)— single archive rewrite dropping everything under the prefixtests/test_store/test_zip.pytest_api_integration— chunk deletion anddel root["bar"]now succeedtest_store_supports_deletes,test_delete_compacts_duplicates,test_delete_then_setStoreTests.test_delete,test_delete_dir,test_delete_nonexistent_key_does_not_raisenow run and passtests/test_codecs/test_sharding.pyfilterwarnings("ignore:Duplicate name")marker for the now-active zip casechanges/828.feature.mdTest Results
test_zip.py: 74 passed, 5 skippedtest_sharding.py(zip): 49 passedtest_core.py+test_api.py(zip): 28 passed, 3 skippedNote:
*_syncdelete tests are skipped —ZipStoredoes not implementSupportsDeleteSync, which is out of scope for this PR.Checklist
docs/user-guide/*.mdchanges/Closes #828