aws-load-balancer-controller: alb.ingress.kubernetes.io/tags not updating properly

If you update alb.ingress.kubernetes.io/tags, tags are not created, deleted, or updated on ELB. For it to update you need to change alb.ingress.kubernetes.io/tags again, and it will show the previous tags. Then you change it again and the tags from the second application will appear, but not the ones from the last application.

	Tags on Kubernetes YAML	Tags on AWS ELB
Initial Apply	A=a	A=a
2nd Apply	B=b	A=a
3rd Apply	C=c	B=b
4th Apply		C=c
5th Apply	D=d
6th Apply	E=e	D=d
7th Apply	E=e,F=f	E=e
8th Apply	F=f	E=e,F=f
9th Apply	G=g	F=f
10th Apply	G=g	G=g

The AWS tags are always one state behind from the ones specified on Kubernetes YAML.

It is even stranger because AddTags and DeleteTags events are logged on CloudTrail and on AWS search bar I can see the tags and its values, but if I filter no ELBs are listed. The same happens on AWS CLI and Boto3 if you describe the ELBs, its tags do not appear or appear outdated.

I’ve waited more than 10min to verify if there was some delay on tag application but nothing happened. The tags just update if I change Ingress YAML again.

The changes on YAML doesn’t necessarily need to be on alb.ingress.kubernetes.io/tags, any change the forces Kubernetes to reconfigure Ingress will make the tags from the previous configuration appear.

• Initial Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: A=a
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 2nd Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: B=b
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 3rd Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: C=c
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 4th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 5th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: D=d
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 6th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: E=e
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 7th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: E=e,F=f
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 8th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: F=f
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 9th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: G=g
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80

• 10th Apply

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/tags: G=g
    alb.ingress.kubernetes.io/target-type: ip
    H: h
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: nginx
              servicePort: 80