RWX not recognized

[jensjohansen]

Installing in AWS EKS

helm install wireguard netmaker/netmaker -n networking --set wireguard.kernel=true --set baseDomain=dev-wireguard.amz-link-labs.net --set replicas=2 --set ingress.enabled=true --set ingress.className
=nginx-external --set ingress.tls.issuerName=letsencrypt-production --set dns.enabled=false  --set postgresql-ha.postgresql.replicaCount=2 --set dns.enabled=true --set dns.RWX.storageClassName=efs-sc --set dns.clusterI
P=10.100.100.100

Error: INSTALLATION FAILED: execution error at (netmaker/templates/mq.yaml:103:23): A valid .Values.RWXStorageClassName entry required! Specify an available RWX storage class.

I have tried using both EBS and EFS (AWS' nfs implementation).

What does netmaker SPECIFICALLY require, or, how is it determining that my storage class is not RWX or RWO when it is the OV/PVC that sets the access mode?

[jensjohansen]

Reading through mq.yaml, it looks like the problem is that the installation instructions are wrong.

There is an additional --set required for RWXStorageClassName (in addition to dns.RWX.storageClassName. The following worked:

helm install wireguard netmaker/netmaker -n networking --set wireguard.kernel=true --set baseDomain=dev-wireguard.amz-link-labs.net --set replicas=2 --set ingress.enabled=true --set ingress.className
=nginx-external --set ingress.tls.issuerName=letsencrypt-production --set dns.enabled=false  --set postgresql-ha.postgresql.replicaCount=2 --set dns.enabled=true --set RWXStorageClassName=efs-sc --set dns.clusterIP=10.
100.100.100 --set dns.RWX.storageClassName=efs-sc

[jensjohansen]

For AWS deployments, you need to set up EFS

Create an IAM policy to grant EFS permissions with the name AmazonEKS_EFS_CSI_Driver_Policy

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Action": [
       "elasticfilesystem:DescribeAccessPoints",
       "elasticfilesystem:DescribeFileSystems"
     ],
     "Resource": "*"
   },
   {
     "Effect": "Allow",
     "Action": [
       "elasticfilesystem:CreateAccessPoint"
     ],
     "Resource": "*",
     "Condition": {
       "StringLike": {
         "aws:RequestTag/efs.csi.aws.com/cluster": "true"
       }
     }
   },
   {
     "Effect": "Allow",
     "Action": "elasticfilesystem:DeleteAccessPoint",
     "Resource": "*",
     "Condition": {
       "StringEquals": {
         "aws:ResourceTag/efs.csi.aws.com/cluster": "true"
       }
     }
   }
 ]
}

 

Create a service account with this policy for EFS access

eksctl create iamserviceaccount --cluster <cluster-name> --namespace kube-system --name efs-csi-controller-sa --attach-policy-arn arn:aws:I am::<your-aws-account>:policy/AmazonEKS_EFS_CSI_Driver_Policy --override-existing-serviceaccounts --approve --region us-east-1

Install the EFS driver

helm repo add aws-efs-csi-driver https://kubernetes-sigs.github.io/aws-efs-csi-driver/
helm repo update
helm upgrade -i aws-efs-csi-driver aws-efs-csi-driver/aws-efs-csi-driver --namespace kube-system --set image.repository=602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/aws-efs-csi-driver --set controller.serviceAccount.create=false --set controller.serviceAccount.name=efs-csi-controller-sa

You should see a response like this:

Release "aws-efs-csi-driver" does not exist. Installing it now.
NAME: aws-efs-csi-driver
LAST DEPLOYED: Fri May 20 15:21:28 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that aws-efs-csi-driver has started, run:
kubectl get pod -n kube-system -l "app.kubernetes.io/name=aws-efs-csi-driver,app.kubernetes.io/instance=aws-efs-csi-driver"

Retrieve the EKS cluster’s VPC:

vpc_id=$(aws eks describe-cluster --name linklabs-DEV --query "cluster.resourcesVpcConfig.vpcId" --region us-east-1 --output text)
echo $vpc_id

(or just look it up in the console, e.g. something like vpc-0e9920e88fb6788fe

Look up the EKS Cluster’s CIDR:

cidr_range=$(aws ec2 describe-vpcs  --vpc-ids $vpc_id  --query "Vpcs[].CidrBlock" --region us-east-2 --output text)
echo $cidr_range

(or just look it up in the console, e.g. EKS clusters usually default to is 10.100.0.0/16

Look up the VPC’s security group:

security_group_id=$(aws ec2 create-security-group --group-name MyEfsSecurityGroup --description "My EFS security group" --vpc-id $vpc_id --region us-east-2 --output text)
echo $security_group_id

Or just look it up in the console: you'll find the security group for your EKS cluster looks like sg-0afdd009a4f2ae974

Create an inbound rule to allow EFS traffic with the EKS using the values you "found" above:

aws ec2 authorize-security-group-ingress  --group-id sg-0afdd009a4f2ae974 --protocol tcp  --port 2049 --region us-east-1 --cidr 10.100.0.0/16

Create an EFS files system for the cluster

aws efs create-file-system --region us-east-1 --performance-mode generalPurpose --query 'FileSystemId' --output text

Get the list of subnets for the EKS cluster from the management console or with the command:

aws ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-0e9920e88fb6788ef" --query 'Subnets[*].{SubnetId: SubnetId,AvailabilityZone: AvailabilityZone,CidrBlock: CidrBlock}' --region us-east-1 --output table

For each PRIVATE Subnet in the cluster

aws efs create-mount-target --file-system-id fs-084d82c065d8a54ee --region us-east-1 --security-groups sg-0afdd009a4f2ae974 --subnet-id subnet-0ec610be563fd00f3

Create a storage class for EFS:

apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-084d82c065d8a54ee
  directoryPerms: "700"
  gidRangeStart: "1000" # optional
  gidRangeEnd: "2000" # optional
  basePath: "/dynamic_provisioning" # optional

This process sets up a shareable filesystem that you can reference through the storageClassName efs-sc