Shim new streaming backend

[nclack]

chucky is a new streaming backend for acquire.

It adds support for gpu acceleration and there are some improvements to cpu-based streaming as well.

Getting it in is going to take a few steps.

I'm starting with this PR by creating a "shim". The shim is a thin layer implementing the public c api of acquire using chucky. I will use that to gradually walk through acquire-zarr's tests, changing chucky as necessary.

This should allow us to make some standalone libraries/wheels we can test for a bit and identify when to switch the back-ends more permanently.

[nclack]

Added almost all functionality. I'm trying very hard to maintain existing acquire-zarr behavior. One thing that will have to change is some of the downsampling behavior.

acquire-zarr treats x and y in a special way, where chucky does not. This impacts some rules determining when to stop downsampling certain dimensions as we get to higher (coarser) levels of detail (lod).

If the user doesn't want to allow the aspect of ratios of dimensions to change, then we must stop downsampling as soon as any dimension becomes too small; "Too small" means it's one chuck or fewer in that dimension.

If aspect ratios can change, then we just stop downsampling dimensions that are too small and continue with the others until there are no dimensions left - everything fits in one chunk.

This is almost what acquire-zarr does. The aspect ratio of x and y are held fixed, but the others can change. Chunk sizes also vary.

In chucky, variable chunk size causes all kinds of headaches - so I'll constrain to a constant chunk size. I'll add an option letting a user select whether they want a variable aspect ratio or not. I've already got a max lod parameter. The lod's will be computed until any/all dimensions are 1 or fewer chunks wide or the max lod is hit.

[aliddell]

Couple of things need looking at.

[aliddell]

Still needed?

[nclack]

not when things are working :D I vote for keeping it.

[aliddell]

lol fair enough

[aliddell]

.claude is already on line 87

[aliddell]

This is going to get out of date very quickly (if it isn't already) and it probably won't be relevant after merge. Consider deleting this file and moving the contents to the PR description instead.

[aliddell]

Rejecting NULL-valued metadata_key is a behavior change. It should be accepted if and only if we have just the one array.

[aliddell]

Seems wrong to return Success with a NULL-valued *key.

On closer inspection, the original code dedupes and regularizes the keys, so the behavior is changed here.

[nclack]

Packaging note: chucky drop-in wheels (shim/python, shim/python-gpu)

The two shim Python packages now both declare distribution name acquire-zarr (was acquire-zarr-cpu / acquire-zarr-gpu):

• shim/python → acquire-zarr (CPU) — commit 3692f4c
• shim/python-gpu → acquire-zarr (GPU) — commit a2a6470

Why: this makes the chucky wheels a transparent drop-in for the baseline package, so downstream acquire-zarr>=… requirements (e.g. ome-writers[acquire-zarr]) resolve to chucky without consumers editing their dependency names. Both build dist acquire-zarr but ship as separate wheels (CPU vs GPU) — never on the same index, so there's no name collision.

Trial wheels (cp313 / linux_x86_64) published as pre-releases on the fork, built natively via a uv py3.13 venv + python -m build --no-isolation shim/python[-gpu] (self-contained):

Release	Backend	Deps
chucky-cpu-cp313	CPU	vcpkg static (aws-c-s3/blosc/lz4/zstd/openssl)
chucky-gpu-cp313	GPU sm_120	CUDA 13.2 + nvcomp 5.2 (static); needs only libcuda.so.1 at runtime

Consumed by czbiohub-sf/livescreen-acquisition#169.

⚠️ GPU compression limitation: the GPU backend supports only zstd / uncompressed — blosc-lz4 and blosc-zstd are rejected at stream-create (shim/chucky/src/stream/config.c: "max_output_size is 0 (unsupported codec?)"). The downstream trial works around this by defaulting to zstd. Flagging in case GPU CODEC_BLOSC_* support is in scope for this PR (the CPU backend handles blosc fine).