ingress-nginx: aws-load-balancer-name annotations doesn't work

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): v1.1.0

Kubernetes version (use kubectl version): v1.21.2-eks-0389ca3

Environment:

  • Cloud provider or hardware configuration: EKS

  • OS (e.g. from /etc/os-release): amazon-eks-node-1.21-v20211109

  • Kernel (e.g. uname -a): not relevant

  • Install tools:

    • Helm Chart

  • Basic cluster related info:

    • kubectl version: v1.21.2-eks-0389ca3

  • How was the ingress-nginx-controller installed:

    • If helm was used then please show output of helm ls -A | grep -i ingress
ingress-nginx                 	[REDACTED]	1       	2021-12-09 10:31:03.476645858 +0000 UTC	deployed	ingress-nginx-4.0.13                	1.1.0
  • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
USER-SUPPLIED VALUES:
controller:
  admissionWebhooks:
    patch:
      priorityClassName: system-cluster-critical
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app.kubernetes.io/name
              operator: In
              values:
              - ingress-nginx
            - key: app.kubernetes.io/instance
              operator: In
              values:
              - ingress-nginx
            - key: app.kubernetes.io/component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
        weight: 100
  autoscaling:
    enabled: true
    maxReplicas: 10
    minReplicas: 2
    targetCPUUtilizationPercentage: 90
    targetMemoryUtilizationPercentage: 90
  config:
    client-header-buffer-size: 32k
    forwarded-for-header: X-Real-IP
    hsts-max-age: "31536000"
    hsts-preload: "true"
    log-format-escape-json: "true"
    log-format-upstream: '[REDACTED]'
    proxy-add-original-uri-header: "true"
    proxy-real-ip-cidr: [REDACTED]
    server-tokens: "false"
    use-forwarded-headers: "true"
    use-gzip: "true"
    use-proxy-protocol: "true"
    whitelist-source-range: |-
      [REDACTED]
  extraArgs:
    default-backend-service: [REDACTED]
  metrics:
    enabled: true
    service:
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
  minAvailable: 1
  nodeSelector:
    node-selector-key: system
  priorityClassName: system-cluster-critical
  replicaCount: 2
  resources:
    limits:
      cpu: 500m
      memory: 1Gi
    requests:
      cpu: 10m
      memory: 69.6Mi
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: [REDACTED]
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
      service.beta.kubernetes.io/aws-load-balancer-eip-allocations: [REDACTED]
      service.beta.kubernetes.io/aws-load-balancer-name: [REDACTED]
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
      service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
      service.beta.kubernetes.io/aws-load-balancer-type: nlb
    externalTrafficPolicy: Local
  tolerations:
  - key: CriticalAddonsOnly
    operator: Exists
podSecurityPolicy:
  enabled: true
  • Current State of the controller:
    • kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/version=1.1.0
              helm.sh/chart=ingress-nginx-4.0.13
Annotations:  helm.fluxcd.io/antecedent: [REDACTED]:helmrelease/ingress-nginx
              meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: [REDACTED]
Controller:   k8s.io/ingress-nginx
Events:       <none>
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
Name:                 ingress-nginx-controller-688465b86c-84sp6
Namespace:            [REDACTED]
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 [REDACTED]
Start Time:           Thu, 09 Dec 2021 11:31:26 +0100
Labels:               app.kubernetes.io/component=controller
                      app.kubernetes.io/instance=ingress-nginx
                      app.kubernetes.io/name=ingress-nginx
                      pod-template-hash=688465b86c
Annotations:          kubernetes.io/psp: ingress-nginx
Status:               Running
IP:                   [REDACTED]
IPs:
  IP:           [REDACTED]
Controlled By:  ReplicaSet/ingress-nginx-controller-688465b86c
Containers:
  controller:
    Container ID:  docker://509b8c5f97d717a236e7e31b728237472765c053fcf55fe7589548e2dc193d17
    Image:         k8s.gcr.io/ingress-nginx/controller:v1.1.0@sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a
    Image ID:      docker-pullable://k8s.gcr.io/ingress-nginx/controller@sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a
    Ports:         80/TCP, 443/TCP, 10254/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --default-backend-service=[REDACTED]
    State:          Running
      Started:      Thu, 09 Dec 2021 11:31:27 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  1Gi
    Requests:
      cpu:      10m
      memory:   72980889600m
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-688465b86c-84sp6 (v1:metadata.name)
      POD_NAMESPACE:  [REDACTED] (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mp9q9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-mp9q9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             node-selector-key=system
Tolerations:                 CriticalAddonsOnly op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From                      Message
  ----    ------     ----  ----                      -------
  Normal  Scheduled  17m   default-scheduler         Successfully assigned [REDACTED]/ingress-nginx-controller-688465b86c-84sp6 to [REDACTED].eu-west-1.compute.internal
  Normal  Pulled     17m   kubelet                   Container image "k8s.gcr.io/ingress-nginx/controller:v1.1.0@sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a" already present on machine
  Normal  Created    17m   kubelet                   Created container controller
  Normal  Started    17m   kubelet                   Started container controller
  Normal  RELOAD     16m   nginx-ingress-controller  NGINX reload triggered due to a change in configuration
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>
Name:                     ingress-nginx-controller
Namespace:                [REDACTED]
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/version=1.1.0
                          helm.sh/chart=ingress-nginx-4.0.13
Annotations:              helm.fluxcd.io/antecedent: [REDACTED]:helmrelease/ingress-nginx
                          meta.helm.sh/release-name: ingress-nginx
                          meta.helm.sh/release-namespace: [REDACTED]
                          service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: [REDACTED]
                          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
                          service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: 3600
                          service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
                          service.beta.kubernetes.io/aws-load-balancer-eip-allocations: [REDACTED]
                          service.beta.kubernetes.io/aws-load-balancer-name: [REDACTED]
                          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
                          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: *
                          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
                          service.beta.kubernetes.io/aws-load-balancer-type: nlb
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       [REDACTED]
IPs:                      [REDACTED]
LoadBalancer Ingress:     a1e09897b8[PARTIALLY REDACTED]0a632f5df9.elb.eu-west-1.amazonaws.com
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31805/TCP
Endpoints:                [REDACTED]
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31896/TCP
Endpoints:                [REDACTED]
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     31718
Events:
  Type    Reason                Age                From                Message
  ----    ------                ----               ----                -------
  Normal  EnsuringLoadBalancer  18m (x2 over 19m)  service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   18m (x2 over 19m)  service-controller  Ensured load balancer

What happened:

The Load Balancer was not created with the custom name that I choose using service.beta.kubernetes.io/aws-load-balancer-name annotation.

What you expected to happen:

I expect that LB is created with the custom name of my choice.

How to reproduce it:

Using the annotation while deploying the ingress controller.

Anything else we need to know:

The Proxy Protocol v2 was NOT enabled (Ref #7905) so we presume a more general problem with annotations.

/kind bug

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 13
  • Comments: 18 (6 by maintainers)

Most upvoted comments

I’m still running into this issue using service.beta.kubernetes.io/aws-load-balancer-name on a Nginx with a AWS load balancer ingress nlb, no matter what we try the LB name ends up being a seemingly random string.

+4
gijsdpg on Mar 8, 2022

Hi @longwuyuan, thank you for your answer. Sorry, I couldn’t extract that information from this thread. I’m currently in contact with AWS support to see what can be done to address this issue. They argue this is an issue on the Nginx side (they even refer to the discussion in the Github issue), while I suspect it could be a missing parameter in the AWS API. Is there something AWS could do to address this issue?

thank you for your time.

+2
gijsdpg on Mar 9, 2022

@aliusmiles I found all the available annotations directly inside the K8s code: (ServiceAnnotationLoadBalancerType code just to make an example).

I found unbelievable the fact that is no documentation about the internal AWS providers. I understand that is “legacy” but it is still inside K8s.

+1
pierluigilenoci on Dec 10, 2021

@pierluigilenoci that will only get you lb created by in-tree controller. to use aws-lb-controller you need to:

  1. have it installed into cluster
  2. aws-load-balancer-type set to either nlb-ip or external. ref: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/guide/service/annotations/#lb-type

Just in case, by in-tree I mean lb creation mechanism that’s baked right into kubernetes code. it has limited functionality, for example it uses classic loadbalancer by default. and I’m quite sure it does not support names for loadbalancers. To have more options you need to use aws-load-balancer-controller

+1
tbondarchuk on Dec 9, 2021
Read more comments on GitHub
← ingress-nginx: auth-tls-pass-certificate-to-upstream does not work with https
ingress-nginx: Basic auth: Blowfish/Bcrypt password hashing doesn't work →