Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support zarr write_empty_chunks for zarr-python 3 and up #10177

Merged
merged 6 commits into from
Mar 30, 2025

Conversation

ianhi
Copy link
Contributor

@ianhi ianhi commented Mar 26, 2025

Prevent passing write_empty_chunks from raising an error. Prior to this PR Dataset.to_zarr(...,write_empty_chunks=) would raise:

TypeError: Group.create_array() got an unexpected keyword argument 'write_empty_chunks'

This seems like the shortest path to prevent this bug without changing the xarray api at all, in the future it might be good to better align the xarray api with the zarr api. I also added order to this fix for create array as it underwent a similar deprecation.

@sotosoul I independently encountered this issue and set out to fix it, I hadn't seen the fix in your fork. Can you try this out and see if it fixes it for you.

Test Changes

The test that would have caught this issue was not enabled for zarr v3. I enabled it but had to make some modifications to have it work for both format 2 and 3. I did a few things

  1. changed the file listing to work for both format 2 and format 3
  2. test the expected files after the first write and again after the append
    • This helped me understand what was happening in the test and is a bit more throrough
  3. temporarily don't test some files when the zarr-python library is version 3 due a zarr bug: Appending to re-opened zarr array does not respect write_empty_chunks zarr-developers/zarr-python#2931

@ianhi ianhi changed the title Zarr3 write empty fix Support zarr write_empty_chunks for zarr-python 3 and up Mar 26, 2025
Copy link
Contributor

@dcherian dcherian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, Ian. LGTM.

@dcherian dcherian enabled auto-merge (squash) March 30, 2025 00:07
@dcherian dcherian merged commit f25637d into pydata:main Mar 30, 2025
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants