[bendichter]

Closes #463.

This lets zarrita read arrays with the struct data type, which holds records of named fields like a NumPy structured dtype.

let arr = await zarr.open.v3(root.resolve("trials"), { kind: "array" });
arr.is("struct"); // true
let { data } = await zarr.get(arr);
data.get(0); // { idx_start: 0, count: 10, timeseries: "/acquisition/ts" }

What It Adds

• StructArray, next to BoolArray and the string arrays. It is a view of the packed records, with the same buffer, byteOffset, byteLength, and BYTES_PER_ELEMENT that the indexing code copies bytes with, so get and slicing needed no change. Each element reads through a DataView as an object with a value for every field. Fields are packed with no padding, so a record can be 13 bytes and start at any offset, which a typed array could not view.
• Field types: the numeric types, bool, fixed_length_utf32, zarr-python's null_terminated_bytes, and a nested struct. Any other field type raises InvalidMetadataError.
• Byte order: for big-endian data the bytes codec reverses each field on its own, on a copy. A StructArray is little-endian in memory.
• Fill values: an object with a value for each field. int64 fields become bigint, "NaN" and the infinities become numbers, and a byte string field is decoded from base64.
• The earlier name: zarr-python wrote these arrays as structured before struct was registered, with fields as [name, data_type] pairs, a base64 fill value, and a bytes codec without endian. The registry asks readers to accept that form. It is converted to struct when the metadata is read, so arr.dtype is the same for both.
• Struct joins the DataType union, with StructScalar as its scalar, and arr.is("struct") narrows to it.

Any other data type that is an object in the metadata, such as numpy.datetime64, now raises InvalidMetadataError with the type in the message. It was the same TypeError before.

What It Leaves Out

Writing is not covered. set treats any object value as a chunk, so a struct scalar cannot be set, and create cannot encode a fill value that holds a bigint or a byte string. I kept this to reading.

Fixtures

generate-v3.py pins zarr-python with exclude-newer, and that version writes structured. I used it for the earlier form (1d.chunked.structured) and added generate-v3-struct.py, which needs zarr-python 3.4, for 1d.chunked.struct, 1d.chunked.struct.be, and 2d.chunked.compressed.struct. All four have chunks that were never written, to cover the fill value.

I added only these four arrays. Running update-fixtures rewrites the compressed bytes of every existing fixture, so I left those and the zip archives as they are.

Testing

Unit tests cover StructArray and the bytes codec with little- and big-endian records. The fixture tests read the same records in all three forms and compare them with what zarr-python wrote. I also read a compound column from an NWB file written by hdmf-zarr by hand (int32, int32, and a 2048-byte fixed_length_utf32, zstd-compressed), and got the values zarr-python reads.

Would you prefer the second fixture script, or moving the pin in generate-v3.py forward and regenerating everything?

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: 171cf1f

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR