kubelet Unable to attach or mount volumes

[cnzf1]

how should i do ,if I want to use static pvc with backend minio in kubernetes v1.22?

Now , I meet an error:
Warning FailedMount 2m6s (x6 over 13m) kubelet Unable to attach or mount volumes: unmounted volumes=[webroot], unattached volumes=[webroot kube-api-access-ttwx2]: timed out waiting for the condition
Warning FailedAttachVolume 52s (x7 over 13m) attachdetach-controller AttachVolume.Attach failed for volume "mypv" : Attach timeout for volume data

# pvc.yaml:
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mypv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: ch.ctrox.csi-s3-driver 
    volumeHandle: data
    volumeAttributes:
      endpoint: "http://127.0.0.1:9000"
      bucket: "data"
      accessKey: "5CJABSyn48fLAucMEUmT"
      secretKey: "4tWFfPpKKK2gmEpcig5bkDUlvRZJqm6D0fjBP7Iu"
---
# pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-s3
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: ""
  volumeName: mypv

[manju-rn]

@cnzf1
The problem is with the attacher - in the attacher.yaml - update the image tag to new one
image: quay.io/k8scsi/csi-attacher:v3.1.0

[msaustral]

Hi we have same issue,
attacher, provisioner, csi-s3, secret and storage class are on namespace kube-system

PVC is on another namespace

on the attacher we have this error:

I0313 23:17:16.924292       1 controller.go:198] Started VA processing "csi-2132bfd798226240de63190967823fe261df57e34bda9e0c694806cf93742e19"
I0313 23:17:16.924323       1 trivial_handler.go:53] Trivial sync[csi-2132bfd798226240de63190967823fe261df57e34bda9e0c694806cf93742e19] started
I0313 23:17:16.924328       1 util.go:37] Marking as attached "csi-2132bfd798226240de63190967823fe261df57e34bda9e0c694806cf93742e19"
W0313 23:17:16.927907       1 trivial_handler.go:57] Error saving VolumeAttachment csi-2132bfd798226240de63190967823fe261df57e34bda9e0c694806cf93742e19 as attached: volumeattachments.storage.k8s.io "csi-2132bfd798226240de63190967823fe261df57e34bda9e0c694806cf93742e19" is forbidden: User "system:serviceaccount:kube-system:csi-attacher-sa" cannot patch resource "volumeattachments/status" in API group "storage.k8s.io" at the cluster scope

on the pod we get this error:

AttachVolume.Attach failed for volume "pvc-9b8dba18-c486-43fd-8240-8f1f9dba8438" : timed out waiting for external-attacher of ch.ctrox.csi.s3-driver CSI driver to attach volume backup-k8n/pvc-9b8dba18-c486-43fd-8240-8f1f9dba8438

we delete the attacher, change

image: quay.io/k8scsi/csi-attacher:v3.1.0
image: quay.io/k8scsi/csi-node-driver-registrar:v2.1.0
image: quay.io/k8scsi/csi-provisioner:v2.1.2

and re-create all

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: csi-s3-secret
stringData:
  accessKeyID: xxxxxxxxxxxxxx
  secretAccessKey: xxxxxxxxxxxxxxxxxx
  endpoint: https://sfo3.digitaloceanspaces.com
  region: ""
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: csi-s3
provisioner: ch.ctrox.csi.s3-driver
parameters:
  # specify which mounter to use
  # can be set to rclone, s3fs, goofys or s3backer
  mounter: s3fs
  bucket: backup-k8n
  csi.storage.k8s.io/provisioner-secret-name: csi-s3-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  csi.storage.k8s.io/controller-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
  csi.storage.k8s.io/node-stage-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-stage-secret-namespace: kube-system
  csi.storage.k8s.io/node-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-s3-pvc
  namespace: xxx-prod
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 256Gi
  storageClassName: csi-s3

the PVC is Bound and created on the S3 platform, the only issue is to attached it to the container on creation

any idea?

[msaustral]

Hi, we solve the issue by changing the ClusterRole of the attacher from

- apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update", "patch"]

to

- apiGroups: ["storage.k8s.io"]
    resources: ["*"]
    verbs: ["get", "list", "watch", "update", "patch"]

[Alestrix]

See Issue #80, the solution is described here: #80 (comment)
Worked also for me (I deployed csi-s3, including the secret, in a dedicated namespace and it all works after updating the references)