Skip to content

Commit 66f9bfe

Browse files
FrancescAltedclaude
andcommitted
Stop overselling persisted block layouts
The benchmark column this claim rested on times a fetch whose layouts are known but whose blocks are not, and the implementation makes clear how rare that pairing is: layouts are read only for chunks with missing blocks, so a repeated slice, a new slice in the same session, and a repeated slice in a new session all skip that read already. What is actually on the table is one round trip for a new process reaching into a partly explored chunk, against a vlmeta blob rewritten whole on every update. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 1f569f0 commit 66f9bfe

2 files changed

Lines changed: 16 additions & 6 deletions

File tree

‎plans/fsspec-blocks.md‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -389,9 +389,17 @@ The measurement that gated this is done, and it says build it.
389389
exact wanted-bytes figure, so the rule is a one-liner: fetch the whole chunk
390390
when the wanted blocks exceed ~50% of `cbytes`, or when `cbytes` is below the
391391
~1 MB break-even, or when the chunk is memcpyed. Everything else goes by block.
392-
3. **Mitigation 2 (persist the layouts) is worth as much as the feature itself**:
393-
it is another 2–3x on top (the "blocks, cached" column), for `4 * nblocks`
394-
bytes per chunk in the cache's vlmeta.
392+
3. **Mitigation 2 (persist the layouts) is worth much less than the "blocks,
393+
cached" column suggests, and was not built.** That column times a fetch whose
394+
layouts are known but whose blocks are not — and once the implementation
395+
existed it became clear how rare that combination is. A fetch only reads
396+
layouts for chunks with *missing* blocks, so a repeated slice, a new slice in
397+
the same session (layouts are memoized on the source) and a repeated slice in
398+
a new session (the blocks are in the cache) all skip phase 1 already. What is
399+
left is one round trip saved on the first fetch of a new process reaching into
400+
a chunk it had only partly explored — against a vlmeta blob of `4 * nblocks`
401+
bytes per chunk, rewritten whole on every update. Revisit if many short-lived
402+
processes share one cached array; otherwise the arithmetic does not close.
395403
4. **Keep route B as the answer for the formats route A cannot reach** (sparse
396404
frames, `.b2d`, plain SChunks, `offset`), and prototype the GIL behaviour
397405
before committing to it. Do not build both at once.

‎src/blosc2/proxy.py‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1158,9 +1158,11 @@ def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY):
11581158
else 1
11591159
)
11601160
# ponytail: layouts are memoized for the life of the source, so a second
1161-
# slice pays no header read. Persisting them in the cache would extend
1162-
# that across runs (worth a further 2-3x, measured); do it when someone
1163-
# reopens the same array often enough to notice.
1161+
# slice pays no header read. Persisting them in the cache would save one
1162+
# round trip more, but only for a new process reaching into a chunk it
1163+
# had partly explored before -- every other repeat already skips the read,
1164+
# since a fetch asks for layouts only where blocks are missing. Do it if
1165+
# many short-lived processes ever share one cached array.
11641166
self._layouts = {}
11651167

11661168
@property

0 commit comments

Comments
 (0)