Now standard AWS symlink from /dev/xvdaa to /dev/nvme1n1 break the driver for minikube

[stevemadere]

/kind bug

What happened?
EBS volumes get created and attached just fine but then are unavailable to kubernetes running under minikube.

AWS EC2 instances these days often don't have a /dev/xvdaa device for attaching EBS volumes but instead have a /dev/nvme1n1 device which gets attached and a symlink is then created: /dev/xvdaa -> /dev/nvme1n1

Under this circumstance, the MountVolume.MountDevice fails:

Events:
Type Reason Age From Message

---

Normal Scheduled 14m default-scheduler Successfully assigned default/postgres-deployment-7b6556544c-8ztdk to minikube
Normal SuccessfulAttachVolume 14m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-378c1c8f-500c-4bee-a16d-2ab9457a5511"
Warning FailedMount 2m31s (x14 over 14m) kubelet MountVolume.MountDevice failed for volume "pvc-378c1c8f-500c-4bee-a16d-2ab9457a5511" : rpc error: code = Internal desc = Failed to find device path /dev/xvdaa. no device path for device "/dev/xvdaa" volume "vol-0f1bd87aca480b924" found
Warning FailedMount 4s (x8 over 68s) kubelet MountVolume.MountDevice failed for volume "pvc-378c1c8f-500c-4bee-a16d-2ab9457a5511" : rpc error: code = Internal desc = Failed to find device path /dev/xvdaa. no device path for device "/dev/xvdaa" volume "vol-0f1bd87aca480b924" found

While investigating, I found that the minikube container itself does not seem to be aware of /dev/xvdaa:

[ec2-user@ip-172-31-31-77 ]$ minikube ssh
docker@minikube:$ ls -l /dev/xv* /dev/nvm*
ls: cannot access '/dev/xv*': No such file or directory
crw------- 1 root root 250, 0 Sep 20 20:30 /dev/nvme0
brw-rw---- 1 root disk 259, 0 Sep 20 20:30 /dev/nvme0n1
brw-rw---- 1 root disk 259, 1 Sep 20 20:30 /dev/nvme0n1p1
brw-rw---- 1 root disk 259, 2 Sep 20 20:30 /dev/nvme0n1p128
crw------- 1 root root 250, 1 Sep 20 20:30 /dev/nvme1
brw-rw---- 1 root disk 259, 3 Sep 20 20:30 /dev/nvme1n1
docker@minikube:~$

But the volume is definitely attached and AWS claims it's attached to /dev/xvdaa:

[

ec2-user@ip-172-31-31-77 ~]$ aws ec2 describe-volumes --volume-id vol-0f1bd87aca480b924 --region us-west-2

{
    "Volumes": [
        {
            "AvailabilityZone": "us-west-2b",
            "Attachments": [
                {
                    "AttachTime": "2024-09-20T20:17:13.000Z",
                    "InstanceId": "i-0a6641c5d48bab3e8",
                    "VolumeId": "vol-0f1bd87aca480b924",
                    "State": "attached",
                    "DeleteOnTermination": false,
                    "Device": "/dev/xvdaa"
                }
            ],
            "Tags": [
                {
                    "Value": "pvc-378c1c8f-500c-4bee-a16d-2ab9457a5511",
                    "Key": "kubernetes.io/created-for/pv/name"
                },
                {
                    "Value": "pvc-378c1c8f-500c-4bee-a16d-2ab9457a5511",
                    "Key": "CSIVolumeName"
                },
                {
                    "Value": "default",
                    "Key": "kubernetes.io/created-for/pvc/namespace"
                },
                {
                    "Value": "true",
                    "Key": "ebs.csi.aws.com/cluster"
                },
                {
                    "Value": "pg-data-pvc",
                    "Key": "kubernetes.io/created-for/pvc/name"
                }
            ],
            "Encrypted": false,
            "VolumeType": "gp3",
            "VolumeId": "vol-0f1bd87aca480b924",
            "State": "in-use",
            "Iops": 3000,
            "SnapshotId": "",
            "CreateTime": "2024-09-20T20:17:10.322Z",
            "MultiAttachEnabled": false,
            "Size": 8
        }
    ]
}

What you expected to happen?

After the volume is attached, the device can be mounted even if the attachment Device name is a symlink.

How to reproduce it (as minimally and precisely as possible)?

helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver
helm install aws-ebs-csi-driver aws-ebs-csi-driver/aws-ebs-csi-driver --namespace kube-system

pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pg-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
  storageClassName: ebs-sc

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version):
• Driver version:

[torredil]

Hey @stevemadere, thanks for reporting this : )

We will treat this as a feature request. Currently, the driver is not officially supported or previously tested by our team in minikube environments. It seems that there is a big opportunity to improve the resiliency of FindDevicePath here.

/kind feature

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[AndrewSirenko]

/remove-lifecycle rotten