Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify runtime abstractions #2352

Open
AdamGS opened this issue Feb 13, 2025 · 2 comments
Open

Unify runtime abstractions #2352

AdamGS opened this issue Feb 13, 2025 · 2 comments

Comments

@AdamGS
Copy link
Contributor

AdamGS commented Feb 13, 2025

There are multiple places where we want to specialize for different async runtimes, or where we make assumptions about the availability of a specific runtime (like in Datafusion, where we assume we're running in a tokio one). The IoDispatcher covers some of these cases, but its a bit too low level which makes us hack around it.
I think the general things we want to cover are:

  1. Spawn any work (both futures and streams)
  2. Disk and network IO (probably includes centralizing our current IO traits)
  3. Consistent behavior for features

Other crates do similar things, Examples include:

@AdamGS AdamGS changed the title Unify runtime abstracting Unify runtime abstractions Feb 13, 2025
@Xuanwo
Copy link

Xuanwo commented Feb 26, 2025

By the way, are you interested in adding OpenDAL support to vortex-io? I'm willing to contribute.

@AdamGS
Copy link
Contributor Author

AdamGS commented Feb 26, 2025

Yeah that would be great! I think our IO traits are still changing and having more examples will be good + opendal seems pretty stable so I don't expect it to be hard to maintain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants