Mounted directory contents are not accessible in blobfuse2 2.4.x

[SumanNakka]

Recently, we've encountered an issue where the contents of a mounted directory on a server are not accessible. When attempting to list the sub-folder contents, the directory can be listed once, but if the command is run again, it throws the error:

ls: cannot open directory '.': Input/output error.

Upon troubleshooting, we identified that the Blobfuse version was recently upgraded from 2.3.0 to 2.4.0 through Azure Update Manager. After rolling back the version to 2.3.0, the issue was resolved, and the directory became accessible again.

Could someone please help us understand why this issue occurs with blobfuse2 version 2.4.0 and not with version 2.3.0?

[vibhansa-msft]

Kindly share debug logs with 2.4.0 where directory listing fails, along with your mount command and config.

[SumanNakka]

Thanks for the response, here are the requested,

blobfuse2 mount /root/nalchbp-dev-sftpv3/ --config-file=/root/blobfuse2/blobfuse2.yaml

blobfuse2-dev.yaml.zip

blobfuse2-dev-debug-logs.zip

[syeleti-msft]

I tried to reproduce the issue, but I'm not able to do get the behavior that you are seeing. I would appreciate if you shared the steps to reproduce the behavior.

Also is it possible to verify storage account is not hns enabled. If it is, then you should change your config file likewise.

[SumanNakka]

Hi, sure, here are the steps to reproduce,

1. After mounting the directory on the VM using blobfuse, navigate inside the directory.
2. Navigate to a sub-folder if there is any.
3. List out the contents by ls -lrt or ls -la, and the contents will pop-up.
4. If we try to list them out again and then we see the error.

Also, regarding the HNS, I don't see anything related to it. So, I'm assuming it was not enabled when the storage account was created.

[syeleti-msft]

I am not able to reproduce,

fantom@syeleti-vm:~/mycontainer$ blobfuse2 --version
blobfuse2 version 2.4.0
fantom@syeleti-vm:~/mycontainer$ cd c2
fantom@syeleti-vm:~/mycontainer/c2$ ls -lrt nalc_test
total 0
-rwxr-xr-x 1 fantom fantom    0 Mar  7 05:18 nalc_test
drwxrwxr-x 2 fantom fantom 4096 Mar  7 05:18 uploads
drwxrwxr-x 2 fantom fantom 4096 Mar  7 05:18 1095buploads
fantom@syeleti-vm:~/mycontainer/c2$
fantom@syeleti-vm:~/mycontainer/c2$ ls -lrt nalc_test
total 0
-rwxr-xr-x 1 fantom fantom    0 Mar  7 05:18 nalc_test
drwxrwxr-x 2 fantom fantom 4096 Mar  7 05:18 uploads
drwxrwxr-x 2 fantom fantom 4096 Mar  7 05:18 1095buploads
fantom@syeleti-vm:~/mycontainer/c2$
fantom@syeleti-vm:~/mycontainer/c2$ ls -lrt nalc_test
total 0
-rwxr-xr-x 1 fantom fantom    0 Mar  7 05:18 nalc_test
drwxrwxr-x 2 fantom fantom 4096 Mar  7 05:18 uploads
drwxrwxr-x 2 fantom fantom 4096 Mar  7 05:18 1095buploads

The following are some things you can do to give us more context.

1. Please send the debug logs for 2.3.0 and 2.4.x
2. Please go to the Azure portal -> navigate to your container -> search for the blob nalc_test and share the result. Just wanted to know whether the directory blob is created or not.

3. Modify stream to block_cache in your config.

components:
  - libfuse
  - block_cache
  - attr_cache
  - azstorage