Zarrista overview & goals

[kylebarron]

Human written by @kylebarron

Background

Zarr is the pre-eminent data format for storing N-dimensional data. Zarr-Python is the primary Python library for reading/writing this data; Zarrs is the primary Rust library for reading/writing Zarr, and is potentially much faster than Zarr-Python.

@d-v-b recently prototyped vibe-coded Zarrs bindings for Zarr-Python in zarr-developers/zarr-python#4064. After some discussion, I decided to start prototyping a standalone Zarrs binding for Python, with the goal of providing another option for zarr-developers/zarr-python#4064. That is, Zarrista should be a low-level binding which Zarr-Python could potentially wrap in its higher level APIs in the future.

@d-v-b found that zarr-developers/zarr-python#4064 had vastly improved performance:

I'm seeing ~15x throughput improvement, looks good.

This gives strong motivation for the potential performance improvements of a Python Zarr library built on Zarrs.

Goals

Zarrista should be both directly usable by intermediate-to-advanced users, but should also be built so that Zarr-Python could theoretically build on top of it in the future.

Zarrista should not add logic beyond what already exists in Zarrs. Similar to Obstore, the scope should be limited to only what is already implemented upstream. This keeps maintainability high.

Zarrista should expose as many APIs as possible from Zarrs. Array and Group will be medium-high level APIs, but ideally all lower level APIs (if stable) should also be exposed, so that downstream libraries can choose the most performance route for them.

• Create a minimal but complete Python binding of Zarrs
  • Include parallel sync and async stores, arrays, and groups
  • Support chunk reading
  • Support array reading
  • Support chunk writing
  • Support array writing
• Various store support
  • Icechunk integration
  • ObjectStore/Obstore integration
  • Filesystem integration
  • Memory store integration (for testing only)
• Zero copy data exchange between Rust and Python
  • Primitive, fixed width types:
    • Buffer protocol/numpy
    • Arrow
    • DLPack
  • Variable width types:
    • Buffer protocol/numpy
    • Arrow
    • DLPack
  • Masked types:
    • Buffer protocol/numpy
    • Arrow
    • DLPack

Partner and Stakeholders

Partner with @d-v-b as needed for general project direction, and to ensure that public-facing API is usable

Keeping the scope limited to what is already implemented upstream keeps this project focused to what my (@kylebarron's) strengths are. Given my experience from obstore, async-tiff, and similar projects, I'm really good at creating high-performance, Pythonic APIs from Rust libraries. I have considerably less experience with Zarr itself. This means that I'm ill-equipped to design a higher-level Zarr API myself; that task should be left to @d-v-b and others in higher-level wrapper libraries (Zarr-Python or other).

Milestones

cc @developmentseed/cng-island

[d-v-b]

i think this is the future of zarr + python! let's go!

[maxrjones]

Thanks for writing this up! I don't see zarrs developers in the partners and stakeholders section. Is that representative and/or intentional?

I have a lot of optimism about this as the future of zarr + python. Given that, I think we should be explicit about the long-term maintenance and ownership plans for this library even though it's not a fun part of the conversation.

[kylebarron]

Thanks for writing this up! I don't see zarrs developers in the partners and stakeholders section. Is that representative and/or intentional?

I definitely don't want to exclude anyone. I'd be happy to engage with zarrs and zarr-python maintainers. It's only a question of at what stage should we engage people more broadly. Should we first get to an 0.1 release and then engage people once we have something concrete they can fully test out end to end?

I think we should be explicit about the long-term maintenance and ownership plans for this library

That's fair. In terms of ownership, I think having it in the developmentseed org is good for now. If/when it's seriously evaluated for Zarr-Python to depend on Zarrista, I would expect Zarrista would need to be transferred into the zarr-developers org, and I'd support that.

Long term maintenance of Zarrs is another important discussion. But for now I think we can think of that separately from the bindings.

What do you think?