aws-efs-csi-driver: Missing permission in the example IAM policy file

this is a major issue…as in, following the default AWS instructions leads to EFS CSI driver not working. This should be fixed yesterday.

Based on the issue description and the comments above ( https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/489#issuecomment-892682895 and https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/489#issuecomment-869449047 )

I found another solution is to make a new policy and service account, and be sure to attach that to the node when installing/updating the helm chart. Otherwise, my AccessDenied was from the Node role, and not the pod role.

Then in the helm command, override BOTH the controller and node service accounts:

helm upgrade -i aws-efs-csi-driver aws-efs-csi-driver/aws-efs-csi-driver \
    --namespace kube-system \
    --set image.repository=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-efs-csi-driver \
    --set controller.serviceAccount.create=false \
    --set controller.serviceAccount.name=efs-csi-controller-sa \
    --set node.serviceAccount.create=false \
    --set node.serviceAccount.name=efs-csi-node-sa

Inline node policy (not worth an attachment)

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "elasticfilesystem:DescribeMountTargets",
              "ec2:DescribeAvailabilityZones"
          ],
          "Resource": "*"
      }
  ]
}

Hi, we are facing this issue as well. The node daemonset also doesn’t have service account attached to it. And the IAM permissions mentioned above seem to be required by the daemonset pods.

Any update on when this issue will be fixed?

In my case, creating a policy that only contains DescribeMountTargets and DescribeAvailabilityZones, and attaching that newly created policy to the node group role was sufficient to fix the issue.

Nodes do need to have ''elasticfilesystem:DescribeMountTargets" and “ec2:DescribeAvailabilityZones”

I also had to add ec2:DescribeAvailabilityZones

I also ran into this issue, it made the installation process much more difficult to work through these issues until I eventually found this bug.

@BloodyIron I just wanted to leave a note that your last post saved me countless hours of troubleshooting. I was running into the same issue and it indeed has something to with the EFS being in Single vs Multiple AZ mode. I had the exact same config in the same VPCs with the same AZs and the single AZ EFS would not work. Switching to multi AZ EFS solved the problem.

Fixed in the merge just now #647, please close this?

In my case, I met the error when I set wrong file system id in values.yaml. I recommend you to check the file system id you assigned again.

What do you mean by “file system id” in which values.yaml?

I meant this. For me, I didn’t have to add additional statement below.

        {
            "Effect": "Allow",
            "Action": [
                "elasticfilesystem:DescribeAccessPoints",
                "elasticfilesystem:DescribeFileSystems",
                "elasticfilesystem:DescribeMountTargets"
            ],
            "Resource": "*"
        },

I just set correct fileSystemId in values.yaml and issue was solved. By the way, error message down below is quite misleading in my case because it looks like I have to add additional permission.

Output: Failed to resolve "[fileSystemId].efs.us-east-1.amazonaws.com". The file system mount target ip address cannot be found, please pass mount target ip address via mount options. 
User: arn:aws:sts::888888888888:assumed-role/LordOfTheVolumes/i-0dc4f12f9fdb8dcd3 is not authorized to perform: elasticfilesystem:DescribeMountTargets on the specified resource

In my case, creating a policy that only contains DescribeMountTargets and DescribeAvailabilityZones, and attaching that newly created policy to the node group role was sufficient to fix the issue.

@fptiangco you saved my day, thanks a lot for sharing this. i have attached the iam policy (AmazonEKS_EFS_CSI_Driver_Policy) to my eks node group role. that did the trick @

@AndrewFarley This has been fixed in the master branch, but none of the subsequent releases (v1.3.8, helm-chart-aws-efs-csi-driver-2.2.6) include this fix.

(Sorry, I know this probably shouldn’t be directed at you, but I wasn’t sure who to @)

Correct me if I am wrong. EFS CSI driver is based on efs-utils to perform EFS mounting action. I reviewed the code and my understanding is that it will use elasticfilesystem:DescribeMountTargets to retrieve private IP (unless have mounttargetip option in EFS config) as fallback if the DNS name of EFS resource failed to resolve [1]. (Error and relevant exception message is here [2])

Perhaps the root cause was relating to EFS mount target is not in the same Availability Zone as the client instance (that is the instance running your pods), which are both mentioning in document [3] and [4]:

DNS resolution for file system DNS names requires that the Amazon EFS file system has a mount target in the same Availability Zone as the client instance.

Otherwise I guess it might be relating to DNS resolution on Kubernetes (e.g. CoreDNS), intermittent resolution error, or Pod has own DNS config cause the EFS mount point DNS resolution result cannot correctly propagate to the EFS CSI driver client.

[1] https://github.com/aws/efs-utils/blob/a63e2170f7d1cdaa836cf5e79ef0eccf408f2b92/src/mount_efs/__init__.py#L2146-L2186 [2] https://github.com/aws/efs-utils/blob/a63e2170f7d1cdaa836cf5e79ef0eccf408f2b92/src/mount_efs/__init__.py#L2313-L2327 [3] Troubleshooting Mount Issues - File System Mount Using DNS Name Fails - https://docs.aws.amazon.com/efs/latest/ug/troubleshooting-efs-mounting.html#mount-fails-dns-name [4] Mounting on Amazon EC2 with a DNS name - https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-cmd-dns-name.html