Split `ReadableStore` into sync and async versions?

[kylebarron]

It seems weird that you'd have a single protocol that supports both synchronous and asynchronous request functions. The "colored functions" between sync/async is annoying, so it'll still be useful to separate out useful underlying logic, e.g. for caching, between sync and async implementations.

But I think it's cleanest when downstream applications can understand whether to call sync or async methods based on what the input provides. When the input provides both, you don't know whether the underlying implementation is async, and whether the sync API is a blocking layer on top. Or whether the underlying implementation is sync, and the async API is an abstraction with a threadpool.

Having separate base protocols would mean that your AiohttpStore doesn't have to implement blocking synchronous functions.

(You could have a composable layer like AsyncToSync that translates an async store into a sync one using asyncio blocking calls, but that would be a lot more clear to end users what's happening)

[geospatial-jeff]

The best way to approach this in my opinion is to ensure that we carry await syntax through the entire call chain until the top-level synchronous call.

This is done well in AiohttpStore where, for example AiohttpStore.get_range is a simple wrapper over Aiohttp.get_range_async. The protocols built on top of the aiohttp store should implement the same pattern, for example:

1. Create a new BlockStorereader.read_async method. This should call ReadableStore.get_range_async.
2. Update the implementation of BlockStorereader.read to call BlockStorereader.read_async.

In this way you can always carry await syntax as far as possible through the call chain before switching to synchronous programming.

I'll add that keeping this library entirely await and leaving the caller to decide if, when, and how to call it synchronously is another attractive option here. This would help avoid issues like #50. This is the approach taken by libraries like aiohttp.

[kylebarron]

This would help avoid issues like #50.

💯 any time you're using async code in Jupyter, you should be using await at the top level so you can make use of jupyter's running event loop

[maxrjones]

thanks for this feedback. please feel encouraged to share more thoughts about the library ❤️

It sounds like there's two proposals here:

1. split up each ReadableStore into a AsyncReadableStore and a SyncReadableStore
2. keep async and sync methods on each ReadableStore, and have sync call the async method

I'd prefer (2) because that structure should be more familiar to Zarr users and I'd like obspec-utils to feel familiar to them.

This would help avoid issues like #50.

💯 any time you're using async code in Jupyter, you should be using await at the top level so you can make use of jupyter's running event loop

I added an MVCE in #50 (comment). For the use case in which I found the issue, scipy is using the sync reader methods. So it's going to be necessary for AiohttpStore to use Jupyter's event loop via something like #59 rather than expecting the user to call the async functions.

[kylebarron]

You note in a comment that these protocols are private, right? In that case there's less of a public need to keep these in a single protocol. At a minimum, it's much easier to combine them as needed with

class SyncAndAsyncStore(ReadableStore, AsyncReadableStore, Protocol): ...

While it's impossible to separate them. That's why I feel it's much better to keep protocols as decentralized as possible (and why obspec's protocols are all individual exports)

Wouldn't it better match Zarr-Python's Store abc if it's fully async?

[maxrjones]

so, it would look like:

protocols

class SyncReadableStore(Get, GetRange, GetRanges, Head, Protocol): ...

class AsyncReadableStore(GetAsync, GetRangeAsync, GetRangesAsync, HeadAsync, Protocol): ...

stores

class AiohttpStore(AsyncReadableStore): ...

should there be a SyncAiohttpStore or just a HttpStore(ReadableStore) that doesn't use aiohttp?

wrappers

class AsyncCachingStore(AsyncReadableStore): ...

class SyncCachingStore(SyncReadableStore): ...

class CachingStore(AsyncReadableStore, SyncCachingStore): ...
...

Does sub-classing in CachingStore go against https://developmentseed.org/obspec/latest/blog/2025/06/25/introducing-obspec-a-python-protocol-for-interfacing-with-object-storage/#protocols-duck-typing-not-subclassing? should it instead used shared functions, or not exist at all?