ADLS Blobfuse2 SPN Authentication Issues to Specific ADLS, auth-resource doesn't seem to be working

[red324]

Hello,

I am currently using blobfuse2 version 2.0.3 in order to mount to an Azure Data Lake storage container.

To mount, I am using the command:

sudo blobfuse2 mount ~/mycontainer --tmp-path=/mnt/tmp --config-file=./config.yml

My config file (config.yml) is as follows:

allow-other: true

logging:
  type: base
  level: log_debug
  file-path: ~/blobfuse.log

components:
  - libfuse
  - azstorage

libfuse:
  default-permission: 0777
  attribute-expiration-sec: 120
  entry-expiration-sec: 120
  negative-entry-expiration-sec: 240

azstorage:
  type: adls
  account-name: <account_name>
  mode: spn
  container: raw
  # service principal authentication
  tenantid: <tenantid>
  clientid: <clientId>
  clientsecret: <clientSecret>
  auth-resource: https://storage.azure.com

The mounting seems to succeed, but when I try to get to a directory in the file system and list the contents, I receive the error:

ls: reading directory '.': Input/output error

Looking at the blobfuse logs, I see:

Earlier in the logs, I can see the tokens that are being created. The token for the blob service uri (https://xxx.blob.core.windows.net/) works when using postman to make calls to the directory.

The token generated using the .dfs.core.windows.net does NOT work and produces this error. When I generate a token with: https://storage.azure.com as the resource, I am also able to successfully make the API call.

My service principal has Reader RBAC access to the ADLS Container, and has R-X ACL access to container and RWX access on the directory I am trying to ls in. It is not possible to grant further RBAC access due to organizational security processes, so we use ACLs for authorization. I am able to ls on the container, but not on the directory underneath the container, I am also able to cd into the directory from the container.

I have tested mounting this same storage container using blobfuse version 1 and it does work as expected. I checked the token it is using and the resource appears to be: https://storage.azure.com.

So I tried to set the auth-resource in the config file for blobfuse2 to equal https://storage.azure.com so that the correct token will be generated and used, but I am still receiving access issues shown above.

I am unsure of how to 100% confirm the token that is being used for the GET call, but it would seem to me that regardless of what I set as the auth-resource, it is using the https://xxx.dfs.core.windows.net endpoint. I will also mention, that when I tried to specify a subdirectory in the config file, I got authorization errors even just mounting the container:

Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage]

To reiterate: Blobfuse V1 is working for mounting this container (and I am able to cd/ls for the directory), Postman also works with making these API calls to the directory if I use either the blob.core.windows.net endpoint as the resource, same with the https://storage.azure.com endpoint. But blobfuse2 is not working for working with the ADLS Directory.

I have also set up a demo ADLS account and tried to match the same settings... blobfuse2 DID work for mounting and cd/ls/mkdir in the directory in that ADLS. I noticed that the Token retrieved for that ADLS account using the https://xxx.dfs.core.windows.net as the resource contains groups, whereas when I generate a token for the resource that I am getting auth errors on, the token does not contain groups when using the https://xxx.dfs.core.windows.net endpoint... but it does when hitting the blob.core.windows endpoint and the https://storage.azure.com endpoint as the resource for token generation....

Can anyone steer me in the right direction here as to what might be happening and how to use blobfuse2 for this specific ADLS?

[vibhansa-msft]

components:
  - libfuse
  - azstorage

You need to correct this configuration to

components:
  - libfuse
  - file_cache
  - attr_cache
  - azstorage

For the auth issue kindly validate your SPN has "storage blob data contributor" role assigned to storage account.

[red324]

Updated configuration file as follows:

allow-other: true
 
logging:
  type: base
  level: log_debug
  file-path: ~/blobfuse.log

components:
  - libfuse
  - file_cache
  - attr_cache
  - azstorage
 
libfuse:
  default-permission: 0777
  attribute-expiration-sec: 120
  entry-expiration-sec: 120
  negative-entry-expiration-sec: 240
 
azstorage:
  type: adls
  account-name: <account_name>
  mode: spn
  container: <container_name>
  # service principal authentication
  tenantid: <tenant_id>
  clientid: <client_id>
  clientsecret: <client_secret>
  auth-resource: https://storage.azure.com

I am still receiving this error:
ls: reading directory '.': Input/output error

X-Ms-Error-Code: [AuthorizationPermissionMismatch]

Note: The Authorization is redacted so I am just guessing as to which token is being used based on the results

As I mentioned in my original post, we are unable to grant the storage blob data contributor on the storage account because we use ACLs exclusively to partition access. Storage blob data contributor grants read access to everything, which violates the way the Data Lake Access partitioning is set up.

But like I said, I am able to make this same API call using a different audience and the same Service Principal. Audience = https://storage.azure.com is generating a token with proper access. Audience = https://account_name.dfs.core.windows.net produces the Authorization mismatch.

Even from taking the tokens generated in the blobfuse log file, I can authenticate properly with the .blob.core.windows.net token:

but not the .dfs.core.windows.net token:

According to the way ACLs vs. RBAC work in the Data Lake, it would appear that I should not need the storage blob data contributor role since I have r-x access at container and rwx at the directory level: https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-access-control-model#how-permissions-are-evaluated

Additionally, the proper token auth resources appear to be either https://storage.azure.com or https://<account>.blob.core.windows.net as stated here: https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory#microsoft-authentication-library-msal and here: https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-azure-active-directory#use-oauth-access-tokens-for-authentication

Blobfuse V1 also works with the same container. I think there is a bug in V2 where the token is being generated with the ``https://account_name.dfs.core.windows.net` uri as the audience/resource (and/or also not taking into account the auth-resource in the config file)

[vibhansa-msft]

Kindly use a bug report instead of discussion if you are facing some trouble. It's not possible to assign an engineer to question or track it for certain release.

[red324]

Submitted issue here: #1156