[efs-controller] Encrypted EFS filesystem never reaches synced status

[gnadaban]

Describe the bug
I'm trying to create a FileSystem resource that is encrypted with a custom KMS key with accompanying MountTargets and AccessPoint, but there are various issues:

• The FileSystem CR is stuck in creating state while in AWS console the resource appears ready and available
• If using the fileSystemRef for MountTarget and AccessPoint resources they are never created as the controller does not consider the target FileSystem resource ready ("the referenced resource is not synced yet.")

Steps to reproduce

1. Attempt to create a FileSystem resource
2. Attempt to create MountTarget and AccessPoint resources with fileSystemRef pointing at FileSystem resource
3. FileSystem resource state is stuck in "creating"

Expected outcome

1. The FileSystem resource should become "ready" after it is successfully created and its status in AWS console is "Available".
2. The MountTarget and AccessPoint resources should be created as soon as the FileSystem resource is created and has an ARN

Environment

• Kubernetes version: v1.30.4-eks-a737599
• Using EKS (yes/no), if so version? yes
• AWS service targeted (S3, RDS, etc.) EFS

[Yunbo-Lu]

I encountered the similar issue.

When I deploy the FileSytem manfiest like below, the STATE status is stuck at creating and SYNCED status is stuck at False:

apiVersion: efs.services.k8s.aws/v1alpha1
kind: FileSystem
metadata:
  name: my-filesystem
  namespace: ack-system
spec:
  encrypted: true
  throughputMode: elastic
  performanceMode: generalPurpose
  lifecyclePolicies:
    - transitionToIA: AFTER_30_DAYS
  tags:
    - key: Name
      value: MyEFSFileSystem
    - key: Environment
      value: Production

When I deploy the MountTarget manfiest like below, the SYNCED status is True but the STATE status is stuck at creating:

---
apiVersion: efs.services.k8s.aws/v1alpha1
kind: MountTarget
metadata:
  name: mount-target-az1
  namespace: ack-system
spec:
  fileSystemRef:
    from:
      name: my-filesystem
      namespace: ack-system
  subnetID: subnet-02c1239a3f791d57f
  securityGroups:
    - sg-0d74b83661232689d

In both cases, even after I leave them for a day, the status is still not correctly updated.

[gnadaban]

I'm not sure I'd call this level of completeness "Generally Available" worthy.

[gnadaban]

It would seem that this only happens when encrypted: true is configured.

[gnadaban]

Other fields like policy or lifecyclePolicies have a similar effect: when configured, the resource never gets initialized.
Also, if initially the policy field is set, and is later deleted, the resource never gets synced again due to a missing field error.

{"level":"error","ts":"2024-10-03T20:44:44.492Z","msg":"Reconciler error","controller":"filesystem","controllerGroup":"efs.services.k8s.aws","controllerKind":"FileSystem","FileSystem":{"name":"dummyfs","namespace":"dummy"},"namespace":"dummy","name":"dummyfs","reconcileID":"58c2ae6b-48a5-462d-aa9c-1837b2060f1c","error":"InvalidParameter: 1 validation error(s) found.\n- missing required field, PutFileSystemPolicyInput.Policy.\n","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222"}

[ack-bot]

Issues go stale after 180d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 60d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale

[gnadaban]

/remove-lifecycle stale

[cellux]

We are hitting the same issue with the following manifest:

apiVersion: efs.services.k8s.aws/v1alpha1
kind: FileSystem
metadata:
  name: foo
  namespace: bar
spec:
  availabilityZoneName: eu-west-1a
  backup: true
  backupPolicy:
    status: ENABLED
  encrypted: true
  lifecyclePolicies:
  - transitionToIA: AFTER_30_DAYS
  - transitionToPrimaryStorageClass: AFTER_1_ACCESS
  policy: |
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "elasticfilesystem:ClientMount"
          ],
          "Principal": {
            "AWS": "arn:aws:iam::00000000000:role/foo"
          }
        }
      ]
    }
  tags:
  - key: Name
    value: foobar
  throughputMode: elastic

AWS console shows that the filesystem is ready, the ACK resource on the Kubernetes side is stuck in creating:

status:
  ackResourceMetadata:
    arn: arn:aws:elasticfilesystem:eu-west-1:000000000000:file-system/fs-0
    ownerAccountID: "000000000000"
    region: eu-west-1
  availabilityZoneID: euw1-az3
  conditions:
  - lastTransitionTime: "2025-04-10T14:36:28Z"
    status: "False"
    type: ACK.ResourceSynced
  creationTime: "2025-04-10T14:36:28Z"
  fileSystemID: fs-0
  lifeCycleState: creating
  name: foo
  numberOfMountTargets: 0
  ownerID: "000000000000"
  sizeInBytes:
    value: 0
    valueInArchive: 0
    valueInIA: 0
    valueInStandard: 0

Kubernetes version: v1.31.5-eks-5d632ec
Helm chart version: 3.1.8

[michaelhtm]

Hello @cellux, is this issue solved for you?

[cellux]

Hello @cellux, is this issue solved for you?

Yes, both Filesystem and MountTarget seem to work properly since v1.0.15.

Thanks for your efforts.