defining a zarr schema and validating meets that schema

[hardingnj]

Hi everyone,

We use zarr as a backend to store data generated by a pipeline. These data take the form of counts, and metadata.

We typically use xarray to write and load zarr stores into xarray.Dataset objects.

As we begin to formalise, we realise we need a way to validate that a zarr store is what we are expecting. Specifically:

• it contains the arrays we expect
• the dtypes are as expected
• the shapes are consistent, ie array B shares dimension 0 with dimension 1 of array A.
• attributes are provided and meet expectations.
• all blocks are present (no missing data)

I've looked at pydantic and AVRO, but they don't seem to do exactly what I want. Is anyone aware of any other solutions that would enable this? Or able to provide an example?

[d-v-b]

I solve this problem by making pydantic models for my zarr arrays / groups: see https://github.com/janelia-cellmap/pydantic-zarr. I'd be curious to hear your feedback on that library. it's definitely been extremely helpful for my own work.

[hardingnj]

Thanks! Looks really helpful

[jhamman]

Since you are using Xarray, you may also want to take a look at xarray-schema.

[hardingnj]

Great suggestion- wasn't aware of this...