aws-load-balancer-controller: Internal error occurred: failed calling webhook "webhook.cert-manager.io"

I am currently following the guide at https://kubernetes-sigs.github.io/aws-load-balancer-controller/guide/walkthrough/echo_server/ to test out the new controller and try to work through setting up the same environment with only terraform and no eksctl.

I am not taking any actions beyond those instructed I believe, yet I get this error.

Error from server (InternalError): error when creating "v2_0_0_full.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: no endpoints available for service "cert-manager-webhook"
Error from server (InternalError): error when creating "v2_0_0_full.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s: no endpoints available for service "cert-manager-webhook"

These are the commands that were run:

eksctl create cluster --name eksctl-test --fargate
eksctl utils associate-iam-oidc-provider --cluster eksctl-test --approve
eksctl create iamserviceaccount --cluster=eksctl-test --namespace=kube-system --name=aws-load-balancer-controller --attach-policy-arn=arn:aws:iam::<acctNum>:policy/aws-load-balancer-controller-policy --override-existing-serviceaccounts --approve
aws eks update-kubeconfig --name eksctl-test
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.0.2/cert-manager.yaml
wget https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/main/docs/install/v2_0_0_full.yaml
# Here I edit to the correct cluster name.
kubectl apply -f v2_0_0_full.yaml

The outcome of this is that the container gets stuck creating indefinitely.

➜  godoc-eks (master) ✗ kubectl get po -n kube-system
NAME                                            READY   STATUS              RESTARTS   AGE
aws-load-balancer-controller-859b7f65d6-hsjwr   0/1     ContainerCreating   0          20m
coredns-58c89c64-8g45t                          1/1     Running             0          32m
coredns-58c89c64-x8skp                          1/1     Running             0          32m

The policy being attached is:

Click to expand 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "iam:CreateServiceLinkedRole",
        "elasticloadbalancing:DescribeTargetHealth",
        "elasticloadbalancing:DescribeTargetGroups",
        "elasticloadbalancing:DescribeTargetGroupAttributes",
        "elasticloadbalancing:DescribeTags",
        "elasticloadbalancing:DescribeSSLPolicies",
        "elasticloadbalancing:DescribeRules",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeLoadBalancerAttributes",
        "elasticloadbalancing:DescribeListeners",
        "elasticloadbalancing:DescribeListenerCertificates",
        "ec2:DescribeVpcs",
        "ec2:DescribeTags",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeInternetGateways",
        "ec2:DescribeInstances",
        "ec2:DescribeAddresses",
        "ec2:DescribeAccountAttributes"
      ],
      "Resource": "*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "wafv2:GetWebACLForResource",
        "wafv2:GetWebACL",
        "wafv2:DisassociateWebACL",
        "wafv2:AssociateWebACL",
        "waf-regional:GetWebACLForResource",
        "waf-regional:GetWebACL",
        "waf-regional:DisassociateWebACL",
        "waf-regional:AssociateWebACL",
        "shield:GetSubscriptionState",
        "shield:DescribeProtection",
        "shield:DeleteProtection",
        "shield:CreateProtection",
        "iam:ListServerCertificates",
        "iam:GetServerCertificate",
        "cognito-idp:DescribeUserPoolClient",
        "acm:ListCertificates",
        "acm:DescribeCertificate"
      ],
      "Resource": "*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "ec2:RevokeSecurityGroupIngress",
        "ec2:AuthorizeSecurityGroupIngress"
      ],
      "Resource": "*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": "ec2:CreateTags",
      "Resource": "arn:aws:ec2:*:*:security-group/*",
      "Condition": {
        "Null": {
          "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
        },
        "StringEquals": {
          "ec2:CreateAction": "CreateSecurityGroup"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "ec2:DeleteTags",
        "ec2:CreateTags"
      ],
      "Resource": "arn:aws:ec2:*:*:security-group/*",
      "Condition": {
        "Null": {
          "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
          "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "ec2:RevokeSecurityGroupIngress",
        "ec2:DeleteSecurityGroup",
        "ec2:AuthorizeSecurityGroupIngress"
      ],
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "elasticloadbalancing:CreateTargetGroup",
        "elasticloadbalancing:CreateLoadBalancer"
      ],
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/elbv2.k8s.aws/cluster": "false"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "elasticloadbalancing:DeleteRule",
        "elasticloadbalancing:DeleteListener",
        "elasticloadbalancing:CreateRule",
        "elasticloadbalancing:CreateListener"
      ],
      "Resource": "*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "elasticloadbalancing:RemoveTags",
        "elasticloadbalancing:AddTags"
      ],
      "Resource": [
        "arn:aws:elasticloadbalancing:*:*:targetgroup/*",
        "arn:aws:elasticloadbalancing:*:*:loadbalancer/*"
      ],
      "Condition": {
        "Null": {
          "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
          "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "elasticloadbalancing:SetSubnets",
        "elasticloadbalancing:SetSecurityGroups",
        "elasticloadbalancing:SetIpAddressType",
        "elasticloadbalancing:RegisterTargets",
        "elasticloadbalancing:ModifyTargetGroupAttributes",
        "elasticloadbalancing:ModifyTargetGroup",
        "elasticloadbalancing:ModifyLoadBalancerAttributes",
        "elasticloadbalancing:DeregisterTargets",
        "elasticloadbalancing:DeleteTargetGroup",
        "elasticloadbalancing:DeleteLoadBalancer"
      ],
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
        }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "elasticloadbalancing:SetWebAcl",
        "elasticloadbalancing:RemoveListenerCertificates",
        "elasticloadbalancing:ModifyRule",
        "elasticloadbalancing:ModifyListener",
        "elasticloadbalancing:AddListenerCertificates"
      ],
      "Resource": "*"
    }
  ]
}

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 2
  • Comments: 22

Most upvoted comments

If you are trying to set up on EKS Fargate it currently doesn’t support the cert-manager setup, use the Helm chart to install ALB Controller without cert-manager dependency

Ref: https://aws.amazon.com/premiumsupport/knowledge-center/eks-alb-ingress-controller-fargate/

+1
dijeesh on Mar 9, 2021

This article might be helpful: https://medium.com/@yspreen/get-started-with-fargate-on-aws-https-ingress-991a09020cc0

On Mon, Nov 30, 2020 at 6:09 AM Ali notifications@github.com wrote:

Actually, I launched my eks cluster on fargate. Forgive my level of novice on this topic, but I thought fargate prevented concern for size and scaling.

Did you add cert-manager namespace to the FargateProfile ?

test % eksctl get fargateprofile --cluster test -o yaml

  • name: fp-default podExecutionRoleARN: arn:aws:iam::xxxxxxxxxxxx:role/eksctl-test-cluster-FargatePodExecutionRole- selectors:
    • namespace: cert-manager
    • namespace: default
    • namespace: kube-system subnets:
    • subnet-077968433d7f377c4
    • subnet-0e77c4136a989305c
    • subnet-00cfda3f451dedf7f

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/1563#issuecomment-735548410, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADAL3Z7D7D7EFYIX5SHSJWLSSMSINANCNFSM4S7JZL2A .

+1
yspreen on Nov 30, 2020

Could you verify whether cert-manager is functional on your cluster? You can try the helm package if you don’t want to use the cert-manager.

+1
kishorj on Oct 26, 2020
Read more comments on GitHub
← aws-load-balancer-controller: Installation of AWS load balancer failed to deploy successfully
aws-load-balancer-controller: Internal load balancer throws curl: (52) Empty reply from server →