11# Release notes
22
3- ## Changes from 4.11.0 to 4.11.1
3+ ## Changes from 4.11.0 to 4.12.0
44
5- XXX version-specific blurb XXX
5+ This release focuses on efficient remote arrays. Blosc2 containers can now be
6+ read and written through fsspec URLs, while lazy proxies fetch only the required
7+ blocks, overlap requests and reuse validated caches. Caterva2 arrays gain
8+ block-range reads and concurrent chunk writers. UTF-8 FULL-index lookups also
9+ use substantially less memory by bisecting their vocabulary on disk.
610
711### Improvements
812
913* New ` blosc2[fsspec] ` extra: ` blosc2.open() ` , ` save_array() ` and ` save_tensor() `
1014 accept any [ fsspec] ( https://filesystem-spec.readthedocs.io ) URL — ` s3:// ` ,
11- ` gs:// ` , ` https:// ` , ` zip:// ` , chained ones like
12- ` zip://inner.b2nd::s3://bucket/a.zip ` .
13- ` open() ` reads the container whole, or through a staleness-checked local copy
14- with ` cache_storage= ` (which is what covers ` .b2d ` stores, sparse frames,
15- ` offset ` and ` mmap_mode ` ), or a piece at a time with ` lazy=True ` , which
16- leaves a huge frame where it is and fetches only what a slice touches
17- through the new ` blosc2.FsspecNDSource ` . The two combine: ` lazy=True ` with a
18- ` cache_storage= ` keeps what was fetched there, so a later run starts from
19- it. Protocol drivers (` s3fs ` , ` gcsfs ` ...) and credentials stay the caller's
20- business. On the write side ` NDArray.save() ` and ` blosc2.save() ` upload the
21- whole array as one object; containers cannot be * backed* by a URL while they
22- are written (the C layer rewrites a frame's header and offsets as chunks land,
23- which an object store has no way to serve), so constructors given a URL now say
24- that instead of failing deep in C.
15+ ` gs:// ` , ` https:// ` , ` zip:// ` and chained URLs. ` open() ` can download the
16+ container, keep a validated local copy with ` cache_storage= ` , or fetch only
17+ requested slices with ` lazy=True ` and the new ` blosc2.FsspecNDSource ` ; lazy
18+ reads can also use a persistent cache. Saves upload one complete object, but
19+ URL-backed mutable containers are not supported. Protocol drivers and
20+ credentials remain the caller's responsibility.
2521
2622* A ` C2Array ` can be written to a chunk at a time, which is how several
2723 processes fill one remote array at once: ` update_chunk() ` (and its async
@@ -70,49 +66,13 @@ XXX version-specific blurb XXX
7066 with the traffic, since a fetch in flight is now a block rather than a chunk.
7167 ` bench/ndarray/fsspec-block-granularity.py ` measures both on any array.
7268
73- * A ` Proxy ` over a ` C2Array ` reads ** blocks** too, straight out of the stored
74- frame over HTTP byte ranges. Caterva2 serves a stored dataset from a file, so
75- the ` Range ` header is honoured and composes with the auth cookie; no new
76- endpoint is involved. On cat2.cloud's ` kevlar-tomo.b2nd ` a corner slice costs
77- 0.031 MB instead of 2.723 MB, and a slice touching ten chunks takes 0.14 s
78- against 1.01 s. Three things add up to it: one pooled HTTP client instead of a
79- connection per request (0.162 s → 0.046 s each), fetches that overlap by
80- default as ` afetch() ` already did, and one request carrying the whole wave of
81- ranges (` multipart/byteranges ` , which no object store offers). A dataset the
82- subscriber * computes* — a lazy expression, an HDF5 leaf, a ` .b2z ` member — is
83- fetched a whole chunk at a time as before; which it is costs at most one small
84- request to find out, and is not asked again -- unless the subscriber could not
85- say, which a busy or unreachable one cannot: a 5xx or a connection that failed
86- is asked again on the next fetch rather than written off, since neither
87- downloaded anything to find out. The same holds once blocks are being read: a
88- dataset that stops being served from a file, or a subscriber too busy to serve
89- it, costs the granularity and not the fetch — whatever is still missing comes
90- as whole chunks, which every dataset can be read as. ` blosc2.ByteRangeNDSource ` is
91- the frame reader ` FsspecNDSource ` and the new ` C2NDSource ` share: subclass it
92- with a ` read_range(offset, size) ` to give any transport the same treatment.
93- Opening a remote frame through either of them now costs two requests instead
94- of four (0.237 s → 0.138 s against cat2.cloud), and one for a frame small
95- enough to arrive whole in the first read: the two reads that only measured the
96- next one are guessed at generously instead, since over a network a few hundred
97- bytes and a few kilobytes cost the same. Of those two, only the header is read
98- when the frame is opened — it is what says the frame can be read this way at
99- all — and where the chunks are waits for the first chunk anything asks about.
100- So ` blosc2.open(url, lazy=True) ` reads once, and so does a whole run over a
101- ` cache_storage= ` that already holds the slice wanted: it fetches nothing, and
102- now asks for no index either. ` FsspecNDSource ` also asks the filesystem for
103- the object's metadata, which is where its ` stamp ` comes from, so its floor is
104- that call plus the header read; ` C2Array ` gets geometry and stamp together
105- from ` api/info ` , and its floor is that one request. A persisted cache keeps
106- what the source read about * where* things are — the frame's chunk offsets, and
107- the block offsets of the chunks it holds only part of — so a later run over it
108- starts from those instead of reading them again. A warm fetch of blocks missing
109- from a chunk already half held goes from 4 requests to 2 against a subscriber,
110- and drops the offsets read and one layout read per chunk touched against an
111- object store. Only for a source that can name the bytes it read: positions in a
112- frame are worth nothing against a frame that was replaced, so an unstamped
113- source keeps none of this and reads as before.
114- ` bench/ndarray/cat2-block-granularity.py ` measures all of it on any dataset,
115- against a real subscriber or a stand-in it starts itself.
69+ * A ` Proxy ` over a ` C2Array ` now reads only the required compressed blocks from
70+ file-backed datasets using concurrent, batched HTTP byte ranges. On
71+ cat2.cloud's ` kevlar-tomo.b2nd ` , this reduced a corner slice from 2.723 MB to
72+ 0.031 MB and a ten-chunk slice from 1.01 s to 0.14 s. Computed datasets fall
73+ back to whole-chunk reads. ` blosc2.ByteRangeNDSource ` provides the same frame
74+ reader to ` FsspecNDSource ` , ` C2NDSource ` and custom transports, with lazy,
75+ persistent caching of frame layout metadata.
11676
11777* ` DictStore.member_window(key) ` says where a leaf's frame lies inside a ` .b2z ` ,
11878 as ` (offset, nbytes) ` . A zip store keeps each external leaf uncompressed, so
@@ -126,20 +86,10 @@ XXX version-specific blurb XXX
12686 output it is, and said plainly when it is the store's own super-chunk.
12787
12888* ` blosc2.Proxy(src, urlpath=..., mode="a") ` now adopts the cache left by an
129- earlier run instead of failing on the existing file, so a proxy's cache can
130- outlive the process. The cache must come from a proxy over a source of the same
131- shape and dtype; anything else at that path raises. A cache that holds only
132- some blocks of a chunk keeps them across runs too. Sources that can name the
133- bytes they read are held to that as well, so a remote array * replaced* while
134- keeping its shape is noticed rather than served stale: ` FsspecNDSource ` uses
135- fsspec's token and ` C2Array ` the subscriber's mtime, both free with metadata
136- they already fetch. A proxy that ` blosc2.open ` rebuilds over its own cache
137- gets the same treatment without the raise -- there is no ` mode="w" ` to offer
138- it -- so a cache whose stamp no longer matches starts as though nothing had
139- been fetched and fills again from the bytes served now; opened read-only there
140- is nothing to empty, and every read falls through to the source. Caches from
141- earlier 4.11.1 development builds are not adopted (the stamp moved to a
142- ` proxy-stamp ` entry); pass ` mode="w" ` once.
89+ earlier run, including partially fetched chunks. It validates the cache's
90+ shape, dtype and source stamp, refetching stale data if the remote array was
91+ replaced and raising for incompatible caches. Caches from pre-release 4.12.0
92+ builds require one fresh open with ` mode="w" ` .
14393
14494* The source protocol moved to its own module, ` blosc2.proxy_source ` :
14595 ` ProxySource ` , ` ProxyNDSource ` , ` ByteRangeNDSource ` , ` FsspecNDSource ` and the
0 commit comments