Static provisioning example errors: /bin/sh: line 0: echo: write error: Input/output error

[lucidprogrammer]

/kind bug

NOTE: If this is a filesystem related bug, please take a look at the Mountpoint repo to submit a bug report

What happened?
s3-app logs /bin/sh: line 0: echo: write error: Input/output error
I am able to see the files already in the s3 bucket in the /data path. But I cannot write.
What you expected to happen?
Read and write should work
How to reproduce it (as minimally and precisely as possible)?
EKS Cluster, Create the necessary IAM policy, role and add the Addon. Apply the example static provisioning example
Anything else we need to know?:
nodes are arm64
mount command shows this entry correctly
mountpoint-s3 on /data type fuse (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions)
No errors in the driver pod.
Environment

• Kubernetes version (use kubectl version): 1.31
• Driver version: v1.9.0-eksbuild.1

[muddyfish]

Hi, thanks for creating this issue.

Can you say what IAM permissions you assigned to the policy?
If you've only given read access, I wouldn't be surprised if this is what happens.

Otherwise, can you include the mountpoint logs using this runbook: https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/docs/LOGGING.md#mountpoint-logs

[lucidprogrammer]

Thanks. That was clear. What happened what I gave the specific role to the csi driver addon which has write and read to the specific bucket. I was under the impression, that was only needed. Once I added the same policy to the node, it works perfectly, tks.

[muddyfish]

You should be able to give permissions to the role of the CSI Driver addon - could you send logs using the runbook previously linked?

[dannycjones]

Thanks. That was clear. What happened what I gave the specific role to the csi driver addon which has write and read to the specific bucket. I was under the impression, that was only needed. Once I added the same policy to the node, it works perfectly, tks.

I'm wondering if the underlying node already had a policy attached which granted read access, and then you are adding the new policy with read&write afterwards which gets picked up. If Mountpoint is unable to find the IAM credentials associated with the driver's service account, it'll currently attempt to use the credentials vended by IMDS.

This makes me think it could be a misconfiguration of the IAM Role, specifically the trust policy. Can you double check how you configured it? Please do share here (redacted if necessary).

Reviewing Mountpoint logs would still be a good next step. In particular, if you could run with --debug-crt flag (so the spec for the volume should include - debug-crt under mountOptions list. This should tell us what the credentials provider is doing.