[Not for merging] Experiment with different obstore reading cache methods

[maxrjones]

Using claude to try out various caching mechanisms with the ObstoreReader

[TomNicholas]

This seems cool - would be great to know how it performs.

[maxrjones]

This seems cool - would be great to know how it performs.

from the scripts code in this PR

[TomNicholas]

Okay so IIUC we should expect parsing a HDF file to extract virtual references to be basically ~4-5x faster using obstore_hybrid than it is when just naively downloading the whole file, which was the workaround I've used previously. That's great.

(I feel like fetching data with Icechunk should be faster than 9.85s, but that's a separate problem for another time.)

[maxrjones]

Okay so IIUC we should expect parsing a HDF file to extract virtual references to be basically ~4-5x faster using obstore_hybrid than it is when just naively downloading the whole file, which was the workaround I've used previously. That's great.

Yeah it'll depend on the file structure and such, but you should be able to find a reasonable choice based on a single file and apply it to all.

(I feel like fetching data with Icechunk should be faster than 9.85s, but that's a separate problem for another time.)

Yeah the slowness of loading a spatial subset is weird. There could be a bug