Repository navigation

Fsspec with block granularity - #701

Merged

FrancescAlted merged 46 commits into

mainfrom

fsspec-blocks

Aug 17, 2026

Commits on Aug 16, 2026
- Add a plan for reading Blosc2 containers through fsspec URLs
  
  FrancescAlted
  and
  claude
  committed
  9dc41b8
- Read and write Blosc2 containers through fsspec URLs
  
  FrancescAlted
  and
  claude
  committed
  1ee2f32
- Open remote containers through a local fsspec cache
  
  FrancescAlted
  and
  claude
  committed
  2a5dd2c
- Read remote frames chunk by chunk with lazy=True
  
  FrancescAlted
  and
  claude
  committed
  afbdcea
- Name the http test after what it checks
  
  FrancescAlted
  and
  claude
  committed
  f5b4ba2
- Let a Proxy pick up the cache left by an earlier run
  
  FrancescAlted
  and
  claude
  committed
  9ba346e
- Keep the install page about installing
  
  FrancescAlted
  and
  claude
  committed
  c5961c5
- Trim the fsspec docs to one home per fact
  
  FrancescAlted
  and
  claude
  committed
  ea7543b
- Let lazy= and cache_storage= compose
  
  FrancescAlted
  and
  claude
  committed
  7e6d6e6
- Stamp the chunk cache with fsspec's ukey, not guessed metadata
  
  FrancescAlted
  and
  claude
  committed
  01ca574
- Save a container to an fsspec URL in one PUT
  
  FrancescAlted
  and
  claude
  committed
  950d3bf
- Add an fsspec read/write example
  
  FrancescAlted
  and
  claude
  committed
  2da443f
- Point the fsspec docs at the runnable example
  
  FrancescAlted
  and
  claude
  committed
  f305665
- Fetch a remote chunk in one range read, statelessly
  
  FrancescAlted
  and
  claude
  committed
  215b6b7
- Overlap chunk fetches in Proxy.fetch with a thread pool
  
  FrancescAlted
  and
  claude
  committed
  77f4304
- Default a lazy fsspec proxy to 8 concurrent fetches
  
  FrancescAlted
  and
  claude
  committed
  5beb7ef
- Add a concurrent-fetch example for lazy fsspec arrays
  
  FrancescAlted
  and
  claude
  committed
  be00ee8
- Link the concurrency example from the docs
  
  FrancescAlted
  and
  claude
  committed
  8e4364c
- Bring the plan up to date with what phase 3 became
  
  FrancescAlted
  and
  claude
  committed
  c7f916d
- Install fsspec in the test group so CI actually runs its tests
  
  FrancescAlted
  and
  claude
  committed
  1fd52c2
- Add a benchmark for the concurrency default against a real endpoint
  
  FrancescAlted
  and
  claude
  committed
  2cc1f12
- Document the moto[server] recipe in the benchmark
  
  FrancescAlted
  and
  claude
  committed
  090e149
- Fix aget_chunk against real async filesystems
  
  FrancescAlted
  and
  claude
  committed
  01dfef5
- Test the fsspec support against a real S3 endpoint
  
  FrancescAlted
  and
  claude
  committed
  42fac93
- Address the PR review: file:// URLs, cache identity, cache geometry
  
  FrancescAlted
  and
  claude
  committed
  d2fe249
- Fix six defects found reviewing the frame reader
  
  FrancescAlted
  and
  claude
  committed
  ea09cb8
- Fix what two reviews found in the fsspec support
  
  FrancescAlted
  and
  claude
  committed
  5aec265
- Check the source identity when a Proxy adopts a cache
  
  FrancescAlted
  and
  claude
  committed
  bf87015
- Track fetched chunks explicitly in the Proxy cache
  
  FrancescAlted
  and
  claude
  committed
  ff47bae
- Fix file:// drive URLs on POSIX and .b2d detection with a query
  FrancescAlted
  committed
  2b3a112
- Trim the fsspec proxy plumbing
  FrancescAlted
  committed
  82a971a
- Snapshot the sidecar caches before iterating them
  FrancescAlted
  committed
  c6285e1
- Fetch blocks, not whole chunks, from a lazy fsspec array
  
  FrancescAlted
  and
  claude
  committed
  473511f
- Say what lazy=True fetches now that it is not whole chunks
  
  FrancescAlted
  and
  claude
  committed
  449d459
- Stop truncating lazy chunks down to their header
  
  FrancescAlted
  and
  claude
  committed
  aacd9cc
- Analyse and measure block-granular fsspec downloads
  
  FrancescAlted
  and
  claude
  committed
  6ff74e2
- Benchmark block fetching end to end, against a moto S3 server
  
  FrancescAlted
  and
  claude
  committed
  4e1db6c
- Record what route A became
  
  FrancescAlted
  and
  claude
  committed
  d434ad2
Commits on Aug 17, 2026
- Note the rewrite-per-fetch ceiling where it lives
  
  FrancescAlted
  and
  claude
  committed
  1f569f0
- Stop overselling persisted block layouts
  
  FrancescAlted
  and
  claude
  committed
  66f9bfe
- Stop reading a chunk back to find out what is already in it
  
  FrancescAlted
  and
  claude
  committed
  cbe577e
- Record what the route B prototype settled
  
  FrancescAlted
  and
  claude
  committed
  91d5e75
- Finish waiting for modal screens instead of pausing once
  
  FrancescAlted
  and
  claude
  committed
  161b008
- Address the review: keep the proxy's bookkeeping its own
  
  FrancescAlted
  and
  claude
  committed
  372dc28
- Address the review: chunks that cannot be spliced, and evictions that outlive a proxy
  
  FrancescAlted
  and
  claude
  committed
  dbf6cdf
- Do not read block offsets out of a chunk that has none
  
  FrancescAlted
  and
  claude
  committed
  484f502

Commits

Commits on Aug 16, 2026

Commits on Aug 17, 2026