aws-load-balancer-controller: Failing to create target group with default policy

Describe the bug The controller is failing to create a target group based on the provided policy.

User: arn:aws:sts::123:assumed-role/AWSALBIngressController_3f2bb898eae5ea79ebdb9cb3514f5ec6/1655374570081000102 is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:us-west-2:123:targetgroup/foo/*

This error occurs after trying to create the target group and before reconciliation.

Based on my understanding of the policy, AddTags requires a null cluster tag in the request but also requires a non-null resource tag. I’m not sure how that can be possible during target group creation. Especially since the request has the cluster tag and presumably the target group has no tags on creation.

Steps to reproduce

Expected outcome The target group should be created with expected tags

Environment

  • AWS Load Balancer controller version 2.2.3
  • Kubernetes version v1.17.12-eks-7684af
  • Using EKS (yes/no), if so version? yes, v1.17.12-eks-7684af

Additional Context:

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 21
  • Comments: 37

Commits related to this issue

Most upvoted comments

For anyone who got this issue and suddently see it works, you might be able to check if there is any message is detected in your AWS Health Dashboard to allow you still can use deprecating policy to create target groups.

To make sure you won’t be impacted by this API change, please make sure you are applying the latest IAM Policy [1] for your controller. Your policy generally needs to include:

        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:AddTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "StringEquals": {
                    "elasticloadbalancing:CreateAction": [
                        "CreateTargetGroup",
                        "CreateLoadBalancer"
                    ]
                },
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            }
        }

If you still have any issue, feel free to open a technical support case in account basis.

Hope this helps!

[1] https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json

+13
0xlen on Jun 22, 2023

This is due to recent behavior change on the requirement of explicit elasticloadbalancing:AddTags permission for create resources, and potentially affect new AWS accounts. Please add the following additional IAM policies if you encounter this error:

{
  "Effect": "Allow",
  "Action": [
    "elasticloadbalancing:AddTags"
  ],
  "Resource": [
    "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
    "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
    "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
  ],
  "Condition": {
    "StringEquals": {
      "elasticloadbalancing:CreateAction": [
        "CreateTargetGroup",
        "CreateLoadBalancer"
      ]
    },
    "Null": {
      "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
    }
  }
}

We will update our recommended policies and documentation.

+10
kishorj on Feb 10, 2023

ELB team have temporarily rolled back their change regards IAM permissions

+9
M00nF1sh on Feb 11, 2023

We also just started seeing this issue today. Using v2.4.4 (via helm chart v1.4.5) on EKS 1.23 We installed 1 cluster and it worked as expected. Then we installed a second and third cluster and it was failing with the AccessDenied error.

        {
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:RemoveTags",
                "elasticloadbalancing:AddTags"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": [
                        "true"
                    ],
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": [
                        "false"
                    ]
                }
            }
        },

Removing the Condition block allowed the ALB Controller to function again. This is so strange because this has been working for us for literally months and this is the first time we’ve run into it.

+9
nickzelei on Feb 10, 2023

We’ve also run into that issue Today. Surprisingly that happened to only one of our clusters that was running fine for over 4 months. We are currently using EKS 1.22 and aws-lb-controller v2.4.2 (we tried also to update to v2.4.6, it didn’t help though). Only a cluster in ap-northeast-1 is affected.

Event record

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
        "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXX",
        "accountId": "XXXXXXXXXXXX",
        "accessKeyId": "XXXXXXXXXXXXXXXXXXXX",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "XXXXXXXXXXXXXXXXXXXXX",
                "arn": "arn:aws:iam::XXXXXXXXXXXX:role/role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX",
                "accountId": "XXXXXXXXXXXX",
                "userName": "role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX"
            },
            "webIdFederationData": {
                "federatedProvider": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.ap-northeast-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
                "attributes": {}
            },
            "attributes": {
                "creationDate": "2023-02-08T02:58:05Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2023-02-08T03:40:24Z",
    "eventSource": "elasticloadbalancing.amazonaws.com",
    "eventName": "CreateTargetGroup",
    "awsRegion": "ap-northeast-1",
    "sourceIPAddress": "XXX.XXX.XXX.XXX",
    "userAgent": "elbv2.k8s.aws/v2.4.2 aws-sdk-go/1.42.27 (go1.17.10; linux; amd64)",
    "errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXX is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:targetgroup/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/* because no identity-based policy allows the elasticloadbalancing:AddTags action",
    "requestParameters": null,
    "responseElements": null,
    "requestID": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "eventID": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXXXXX",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.2",
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "elasticloadbalancing.ap-northeast-1.amazonaws.com"
    }
}

For now we have set up elasticloadbalancing:AddTags without restrictions.

+5
marcinswigon on Feb 8, 2023

The same problems occurs on Cluster 1.24 eks.4 alb 2.4.6.

It works if only aws:RequestTag/elbv2.k8s.aws/cluster line is removed, but this seems like a temporary solution.

Hopefully there will be a quick update. 🥲

+4
seungmun on Feb 10, 2023

The above policy change does not work for me unless I also add CreateListener to the elasticloadbalancing:CreateAction list.

Versions: aws-load-balancer-controller v2.4.1, EKS 1.22

+3
askalski-tcg on Feb 10, 2023

Same for me here: EKS 1.24 and aws-load-balancer-controller 2.4.2

I’ll remove the Condition block temporary.

+2
vinixaavier on Feb 10, 2023

We’re having the same issue with a new cluster running kubernetes 1.24 and aws-load-balancer-controller 2.4.6 with the policy from https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json.

Following for advice.

+2
sjones512 on Feb 10, 2023

My EKS version is 1.20 (not fresh, but upgraded from 1.15 -> 1.16 -> 1.17 -> 1.18 -> 1.19 -> 1.20) with ALB Ingress Controller v 1.1.5.

When upgrading it to:

  • AWS Load Balancer Controller v2.3.1 No error with the default IAM policy of the alb controller. So, I will keep this version for my environment.

  • AWS Load Balancer Controller v2.4.2 I had to apply @croachrose’s workaround to get rid of the error.

In order to fix this, I temporarily removed

            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }

Below didn’t work either.

            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            }
+2
chanpark on Jun 24, 2022

@berry2012 It’s almost like AWS changed the implementation of CreateTargetGroup to invoke AddTags if tags are part of the request. In that case, aws:ResourceTag/elbv2.k8s.aws/cluster is null and aws:RequestTag/elbv2.k8s.aws/cluster is not null.

+2
patrickmscott on Jun 20, 2022

The same problem is stil exists, eks 1.27 & AWS lb controller 2.6.2 Removing “conditions” works.

+1
tirelibirefe on Dec 1, 2023

Same issue as described above, only one of ap-southeast-1 clusters affected, happed for the first time today.

aws-load-balancer-controller v2.4.4 EKS 1.24, Platform version eks.3

Policy attached to the role

{
    "Statement": [
        {
            "Action": "iam:CreateServiceLinkedRole",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                }
            },
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:DescribeSSLPolicies",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeListenerCertificates",
                "ec2:GetCoipPoolUsage",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeTags",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeInstances",
                "ec2:DescribeCoipPools",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeAddresses",
                "ec2:DescribeAccountAttributes"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "wafv2:GetWebACLForResource",
                "wafv2:GetWebACL",
                "wafv2:DisassociateWebACL",
                "wafv2:AssociateWebACL",
                "waf-regional:GetWebACLForResource",
                "waf-regional:GetWebACL",
                "waf-regional:DisassociateWebACL",
                "waf-regional:AssociateWebACL",
                "shield:GetSubscriptionState",
                "shield:DescribeProtection",
                "shield:DeleteProtection",
                "shield:CreateProtection",
                "iam:ListServerCertificates",
                "iam:GetServerCertificate",
                "cognito-idp:DescribeUserPoolClient",
                "acm:ListCertificates",
                "acm:DescribeCertificate"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "ec2:RevokeSecurityGroupIngress",
                "ec2:CreateSecurityGroup",
                "ec2:AuthorizeSecurityGroupIngress"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": "ec2:CreateTags",
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                },
                "StringEquals": {
                    "ec2:CreateAction": "CreateSecurityGroup"
                }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Sid": ""
        },
        {
            "Action": [
                "ec2:DeleteTags",
                "ec2:CreateTags"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:ec2:*:*:security-group/*",
            "Sid": ""
        },
        {
            "Action": [
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DeleteSecurityGroup",
                "ec2:AuthorizeSecurityGroupIngress"
            ],
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:CreateTargetGroup",
                "elasticloadbalancing:CreateLoadBalancer"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:DeleteRule",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:CreateRule",
                "elasticloadbalancing:CreateListener"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:RemoveTags",
                "elasticloadbalancing:AddTags"
            ],
            "Condition": {
                "Null": {
                    "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ],
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:RemoveTags",
                "elasticloadbalancing:AddTags"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
                "arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
            ],
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:SetSubnets",
                "elasticloadbalancing:SetSecurityGroups",
                "elasticloadbalancing:SetIpAddressType",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:DeleteTargetGroup",
                "elasticloadbalancing:DeleteLoadBalancer"
            ],
            "Condition": {
                "Null": {
                    "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
                }
            },
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:DeregisterTargets"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
            "Sid": ""
        },
        {
            "Action": [
                "elasticloadbalancing:SetWebAcl",
                "elasticloadbalancing:RemoveListenerCertificates",
                "elasticloadbalancing:ModifyRule",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:AddListenerCertificates"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        }
    ],
    "Version": "2012-10-17"
}

event with error

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "XXXXXXXXXXXX",
        "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/XXX-aws-load-balancer-controller/fXXXXXXXXXXXXXXXXXXX",
        "accountId": "XXXXXXXXXXXX",
        "accessKeyId": "XXXXXXXXXXXXXXXXXXXXX",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "XXXXXXXXXXXXXXXXXXXXXX",
                "arn": "arn:aws:iam::XXXXXXXXXXXX:role/XXX-aws-load-balancer-controller",
                "accountId": "XXXXXXXXXXXX",
                "userName": "XXX-aws-load-balancer-controller"
            },
            "webIdFederationData": {
                "federatedProvider": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.ap-southeast-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
                "attributes": {}
            },
            "attributes": {
                "creationDate": "2023-02-09T14:00:28Z",
                "mfaAuthenticated": "false"
            }
        }
    },
    "eventTime": "2023-02-09T14:39:00Z",
    "eventSource": "elasticloadbalancing.amazonaws.com",
    "eventName": "CreateTargetGroup",
    "awsRegion": "ap-southeast-1",
    "sourceIPAddress": "XXX.XXX.XXX.XXX",
    "userAgent": "elbv2.k8s.aws/v2.4.4 aws-sdk-go/1.42.27 (go1.18.6; linux; amd64)",
    "errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/XXX-aws-load-balancer-controller/XXXXXXXXXXXXXXXXXXX is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:ap-southeast-1:XXXXXXXXXXXX:targetgroup/XXXXXXXXXXXXXXXXXXXX/* because no identity-based policy allows the elasticloadbalancing:AddTags action",
    "requestParameters": null,
    "responseElements": null,
    "requestID": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
    "eventID": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "XXXXXXXXXXXX",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.2",
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "elasticloadbalancing.ap-southeast-1.amazonaws.com"
    }
}
+1
nepalevov on Feb 9, 2023

@berry2012 Any idea what is different about our environment? Is it the controller version? For what it’s worth, this was working just fine and we changed nothing. It just stopped working suddenly.

+1
patrickmscott on Jun 20, 2022
Read more comments on GitHub
← aws-load-balancer-controller: Failed TargetGroup creation. There are multiple interfaces attached to instance
aws-load-balancer-controller: Getting 502 when running a deployment rolling update →