aws-ebs-csi-driver: Unable to use EKS IAM role

/kind bug

What happened? When using IAM role and policy with a helm chart I get the below error:

1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-test", UID:"090eba72-a253-424c-9753-f4a65ed2f820", APIVersion:"v1", ResourceVersion:"115104014", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc-fast": rpc error: code = Internal desc = Could not create volume "pvc-090eba72-a253-424c-9753-f4a65ed2f820": could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message:

Same Policy works when using a kubernetes secret with the aws key and secret.

What you expected to happen? The role should be used successfully when attempting to perform the operation.

How to reproduce it (as minimally and precisely as possible)? Attempt to create and use the IAM role auth policy using the helm chart and terraform module: terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc

Anything else we need to know?:

Environment

Kubernetes version (use kubectl version): v1.20.7-eks-d88609
Driver version: helm chart: 2.1.0

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 2
Comments: 19 (4 by maintainers)

Most upvoted comments

Anybody struggling with this, I recommend you restart the CSI controllers.

kubectl delete pods -n kube-system -l=app=ebs-csi-controller

If you annotate the serviceAccount, it doesn’t utilize the assigned role until rebooted.

+32

mathisve on Feb 17, 2022

Is this a bug, intended or any other sort of problem? I have been struggling with this for weeks

Anybody struggling with this, I recommend you restart the CSI controllers.

kubectl delete pods -n kube-system -l=app=ebs-csi-controller

If you annotate the serviceAccount, it doesn’t utilize the assigned role until rebooted.

cande1gut on Oct 17, 2022

@mathisve you saved my time so much Thank you… !!

kyeongjun-dev on Dec 12, 2022

I had to delete the "${OIDC_PROVIDER}:aud": "sts.amazonaws.com" part from the condition in the IAM role policy to make it work, when using EKS 1.23 with AWS managed EBS CSI addon. Otherwise the ServiceAccount didn’t assumed the IAM role and caused UnauthorizedOperation: You are not authorized to perform this operation.

shturcite on Aug 25, 2022

Hi,

I see the same problem. I’m using:

EKS 1.21
Helm chart aws-ebs-csi-driver v2.1.0

Role policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateSnapshot",
                "ec2:AttachVolume",
                "ec2:DetachVolume",
                "ec2:ModifyVolume",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeInstances",
                "ec2:DescribeSnapshots",
                "ec2:DescribeTags",
                "ec2:DescribeVolumes",
                "ec2:DescribeVolumesModifications",
                "ec2:CreateVolume",
                "ec2:DeleteVolume",
                "ec2:DeleteSnapshot",
                "ec2:CreateTags",
                "ec2:DeleteTags"
            ],
            "Resource": "*"
        }
    ]
}

I also tried the example https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/example-iam-policy.json with the same result.

vtomasr5 on Aug 24, 2021