Repository navigation

Block-granular reads for Caterva2, and what they needed - #702

Merged

FrancescAlted merged 32 commits into

mainfrom

cat2-block-granularity

Aug 20, 2026

Commits on Aug 17, 2026
- Add the plan for block-granular Caterva2 reads
  
  FrancescAlted
  and
  claude
  committed
  c15bda7
- Pool the HTTP client C2Array requests go through
  
  FrancescAlted
  and
  claude
  committed
  7bb0eff
- Lift the frame reading out of FsspecNDSource into ByteRangeNDSource
  
  FrancescAlted
  and
  claude
  committed
  422ba4b
- Read a C2Array's blocks over HTTP ranges
  
  FrancescAlted
  and
  claude
  committed
  d541eb3
- Ask a Caterva2 subscriber for a whole wave of ranges at once
  
  FrancescAlted
  and
  claude
  committed
  b021104
- Record what the block-granularity work landed as, and what it found
  
  FrancescAlted
  and
  claude
  committed
  1b2dd44
- Keep C2NDSource's max_ranges with the rest of its class body
  FrancescAlted
  committed
  b868c53
Commits on Aug 18, 2026
- Measure the Caterva2 block path the way the fsspec one is measured
  
  FrancescAlted
  and
  claude
  committed
  e8d9ba1
- Say when a dataset has no chunks to fetch, rather than tracebacking
  
  FrancescAlted
  and
  claude
  committed
  f9f755c
- The api/chunk gap is fixed; note what a container leaf still lacks
  
  FrancescAlted
  and
  claude
  committed
  4fe79af
- Open a frame in two requests instead of four
  
  FrancescAlted
  and
  claude
  committed
  944ca80
- Notice a remote array that was replaced under a proxy's cache
  
  FrancescAlted
  and
  claude
  committed
  b2918c8
- Record the whole of the block-granularity work in the plan
  
  FrancescAlted
  and
  claude
  committed
  7b9aa62
- Plan byte ranges for container leaves
  
  FrancescAlted
  and
  claude
  committed
  b6b7303
- Say where a leaf's frame lies inside a .b2z
  
  FrancescAlted
  and
  claude
  committed
  e0cc7da
- Pin that a container leaf reads its blocks like any other array
  
  FrancescAlted
  and
  claude
  committed
  460b060
- Record what container-leaf ranges landed as
  
  FrancescAlted
  and
  claude
  committed
  830f370
- Cross off container-leaf ranges in the block-granularity plan
  
  FrancescAlted
  and
  claude
  committed
  acf2d24
- Shim to reproduce itertools.batched for python 3.11
  FrancescAlted
  committed
  438d2fc
Commits on Aug 19, 2026
- Skip the block tests on wasm, where a socket cannot listen
  
  FrancescAlted
  and
  claude
  committed
  3c24a11
- Read a frame's chunk offsets when a chunk is first asked for
  
  FrancescAlted
  and
  claude
  committed
  aeb5a1b
- Keep where the chunks and blocks are in the cache
  
  FrancescAlted
  and
  claude
  committed
  9937188
- Count the reads an S3 block fetch makes now, and pin the kept index
  
  FrancescAlted
  and
  claude
  committed
  95850a8
- Stop the subscriber stand-in idling half a second per test
  
  FrancescAlted
  and
  claude
  committed
  eb7fd15
- Check the cache's stamp, and the bytes a range answer actually carries
  
  FrancescAlted
  and
  claude
  committed
  31a84fc
Commits on Aug 20, 2026
- Empty a cache whose remote was replaced, rather than serve it
  
  FrancescAlted
  and
  claude
  committed
  4a1b927
- Give what a proxy reads from a module of its own
  
  FrancescAlted
  and
  claude
  committed
  0753fa7
- Leave a cache a proxy fits exactly as it found it
  
  FrancescAlted
  and
  claude
  committed
  7254129
- Read a .b2z leaf only where the zip kept it whole
  
  FrancescAlted
  and
  claude
  committed
  410f89f
- Keep the layouts of a cache whose offsets do not fit, and keep them once
  
  FrancescAlted
  and
  claude
  committed
  92b02b3
- Let the source batch the layout reads it already batches
  
  FrancescAlted
  and
  claude
  committed
  8fa4b8f
- Read whole chunks when a subscriber stops answering ranges
  
  FrancescAlted
  and
  claude
  committed
  f81016b

Navigation Menu

Repository navigation

Block-granular reads for Caterva2, and what they needed - #702

Block-granular reads for Caterva2, and what they needed#702
FrancescAlted merged 32 commits into
mainfrom
cat2-block-granularity

Commits on Aug 17, 2026

Add the plan for block-granular Caterva2 reads

Pool the HTTP client C2Array requests go through

Lift the frame reading out of FsspecNDSource into ByteRangeNDSource

Read a C2Array's blocks over HTTP ranges

Ask a Caterva2 subscriber for a whole wave of ranges at once

Record what the block-granularity work landed as, and what it found

Keep C2NDSource's max_ranges with the rest of its class body

Commits on Aug 18, 2026

Measure the Caterva2 block path the way the fsspec one is measured

Say when a dataset has no chunks to fetch, rather than tracebacking

The api/chunk gap is fixed; note what a container leaf still lacks

Open a frame in two requests instead of four

Notice a remote array that was replaced under a proxy's cache

Record the whole of the block-granularity work in the plan

Plan byte ranges for container leaves

Say where a leaf's frame lies inside a .b2z

Pin that a container leaf reads its blocks like any other array

Record what container-leaf ranges landed as

Cross off container-leaf ranges in the block-granularity plan

Shim to reproduce itertools.batched for python 3.11

Commits on Aug 19, 2026

Skip the block tests on wasm, where a socket cannot listen

Read a frame's chunk offsets when a chunk is first asked for

Keep where the chunks and blocks are in the cache

Count the reads an S3 block fetch makes now, and pin the kept index

Stop the subscriber stand-in idling half a second per test

Check the cache's stamp, and the bytes a range answer actually carries

Commits on Aug 20, 2026

Empty a cache whose remote was replaced, rather than serve it

Give what a proxy reads from a module of its own

Leave a cache a proxy fits exactly as it found it

Read a .b2z leaf only where the zip kept it whole

Keep the layouts of a cache whose offsets do not fit, and keep them once

Let the source batch the layout reads it already batches

Read whole chunks when a subscriber stops answering ranges