Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update azure_* crates to 0.20 #7545

Closed
arpad-m opened this issue Apr 29, 2024 · 4 comments · Fixed by #10036
Closed

Update azure_* crates to 0.20 #7545

arpad-m opened this issue Apr 29, 2024 · 4 comments · Fixed by #10036
Labels
c/storage Component: storage

Comments

@arpad-m
Copy link
Member

arpad-m commented Apr 29, 2024

PR #7539 has done the 0.19 update but for the 0.20 update we are blocked by Azure/azure-sdk-for-rust#1652 , because 0.20 is pulling in async-io which we don't want in our dependency graph.
@arpad-m arpad-m added the c/storage Component: storage label Apr 29, 2024
@skyzh
Copy link
Member

skyzh commented Apr 29, 2024

A quick look at that pull request, the async-process crate is only used for AzureCliCredential. If we do not retrieve credentials from azure cli, but in other ways, then this code path won't be called.

Optimally, they should have a feature flag on opt-in azure cli authentication.

@skyzh
Copy link
Member

skyzh commented Apr 29, 2024

might be related: https://github.com/apache/opendal, not sure if they have s3 time travel support but would be good for unifying everything without worrying about vendor-dependent things.

@arpad-m
Copy link
Member Author

arpad-m commented Apr 29, 2024

@skyzh that's interesting. could you file a separate issue about evaluating adoption of opendal? It's orthogonal to this issue.

@skyzh skyzh mentioned this issue Apr 29, 2024
Open
@arpad-m arpad-m mentioned this issue Apr 30, 2024
Merged
5 tasks
@arpad-m
Copy link
Member Author

arpad-m commented Dec 4, 2024

There is a 0.21 version now, and we still can't update, because of Azure/azure-sdk-for-rust#1652 .

@arpad-m arpad-m mentioned this issue Dec 6, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Dec 9, 2024
@arpad-m
Use updated patched azure SDK crates (#10036)
e74e7aa 
For a while already, we've been unable to update the Azure SDK crates
due to Azure adopting use of a non-tokio async runtime, see #7545.

The effort to upstream the fix got stalled, and I think it's better to
switch to a patched version of the SDK that is up to date.

Now we have a fork of the SDK under the neondatabase github org, to
which I have applied Conrad's rebased patches to:
https://github.com/neondatabase/azure-sdk-for-rust/tree/neon .

The existence of a fork will also help with shipping bulk delete support
before it's upstreamed (#7931).

Also, in related news, the Azure SDK has gotten a rift in development,
where the main branch pertains to a future, to-be-officially-blessed
release of the SDK, and the older versions, which we are currently
using, are on the `legacy` branch. Upstream doesn't really want patches
for the `legacy` branch any more, they want to focus on the `main`
efforts. However, even then, the `legacy` branch is still newer than
what we are having right now, so let's switch to `legacy` for now.

Depending on how long it takes, we can switch to the official version of
the SDK once it's released or switch to the upstream `main` branch if
there is changes we want before that.

As a nice side effect of this PR, we now use reqwest 0.12 everywhere,
dropping the dependency on version 0.11.

Fixes #7545
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
c/storage Component: storage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use updated patched azure SDK crates neondatabase/neon
2 participants
@skyzh @arpad-m