pageserver: adopt opendal?

[skyzh]

discussed in #7545, keeping track of vendor sdks and working around them is hard.

it should work as a drop-in replacement of the s3/azure sdks, but need to investigate whether they support time travel queries to facilitate our disaster recovery tools.

[arpad-m]

As one data point, they seem to require account keys for Azure blob storage, or maybe it's just not in their docs for Azure blob storage. Ideally we don't want that but authenticate from the environment.

For S3, more authentication methods are supported, link. Still, I don't think they support authentication via profiles, which makes life easier. And also (quoting page linked just before):

OpenDAL will not refresh the temporary security credentials, please keep in mind to refresh those credentials in time.

[arpad-m]

I think @koivunej 's proposal was to ditch SDK crates entirely and directly invoke the http calls, so go in the opposite direction from the one proposed in this issue.

[jcsp]

Looks like S3 versioning support is still a work in progress: apache/opendal#3943

In general I'm a bit cautious about adopting third party crates rather than vendor SDKs: like Arpad says, authentication can be an awkward area -- ideally we should be using libraries that support all the cloud vendor's native auth methods, so that we have flexibilty in how we deploy.

[koivunej]

To update my stance, I think the s3 SDK is looking very good, as we've "recently" observed with its handling of "slow down" and apart from the defaults, it has been working unsurprisingly. I wouldn't want to ditch it.

[Xuanwo]

For S3, more authentication methods are supported, link. Still, I don't think they support authentication via profiles, which makes life easier. And also (quoting page linked just before):

OpenDAL will not refresh the temporary security credentials, please keep in mind to refresh those credentials in time.

Hi, opendal maintainers here. Seems there are some confusion about this comment.

OpenDAL does support most authentication methods that aws provide, including:

• Env
• Profile
• IDMSv2
• Web Identity (a.k.a Assume Role With Web Identity)
• Assume Role

Our goal is to support native authentication methods from various storage services, enabling users to seamlessly access all available storage options.

Accroding to this comment:

OpenDAL will not refresh the temporary security credentials, please keep in mind to refresh those credentials in time.

When users statically set security_credentials through our service builder, opendal cannot and will not refresh the credentials.

[Xuanwo]

As one data point, they seem to require account keys for Azure blob storage, or maybe it's just not in their docs for Azure blob storage.

Apologies for the confusion.

For azblob services, neither the account name nor the account key is necessary. OpenDAL will automatically attempt to load these from the environment, configuration, and web identity if they are not provided by the user. I will take some time to clarify this in our documentation.

[Xuanwo]

Hi, I'm going to start working on this to demonstrate what it will look like if there are no strong objections.

I plan to add GCS support or replace Azure Storage first (which I believe should have less impact than s3). Which option do you prefer?

cc @jcsp @koivunej @skyzh @arpad-m

[jcsp]

@Xuanwo I'm happy to see a prototype, but please be aware that we haven't committed to this as a direction yet -- we'd need to see what the prototype looks like & if there are any caveats.

[Xuanwo]

Hi, @jcsp @koivunej @skyzh @arpad-m,

I have created a PoC for your evaluation: #10181

It's not a fully functional PR, but I believe it serves as a good starting point.