external-dns: The reliance on deprecated annotation causes Kubernetes validation issues

What happened:

Kubernetes has in-built validation to prevent users from adding the annotation kubernetes.io/ingress.class alongside the newer spec.ingressClassName.

Although external-dns was recently updated for K8s 1.22 support, it still relies on this annotation. As a result, it is not possible to create a new ingress in one go. One must apply the ingress yaml and then apply a patch to include the annotation. This has a knock-on effect to breaking declarative ‘gitOps’ tools such as argoCD.

More information on the validation here: https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/pkg/apis/networking/validation/validation.go?L237#tab=references

What you expected to happen: External-dns needs to rely on the newer spec.ingressClassName and remove the reliance on the deprecated kubernetes.io/ingress.class annotation.

How to reproduce it (as minimally and precisely as possible): Run the following:

cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  ingressClassName: nginx
  rules:
  - host: "my.example.domain"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myexample-service
            port:
              number: 80
EOF

Resulting error: The Ingress "example" is invalid: annotations.kubernetes.io/ingress.class: Invalid value: "nginx": can not be set when the class field is also set

Anything else we need to know?:

Environment:

External-DNS version (use external-dns --version): k8s.gcr.io/external-dns/external-dns:v0.10.0
DNS provider: Route53
Others:

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 13
Comments: 21 (2 by maintainers)

Most upvoted comments

This also breaks gitlab auto-dev-ops.

I’m a bit surprised to find this issue open. The issue is almost a month old and external-dns is broken on every GCP cluster running the most recent release. Shouldn’t this be an extremely high priority issue for this software?

I mean, this must affect a significant percentage of all existing k8s clusters.

I assumed that packages under K8S SIGs had a maintenance team, but I can’t actually find anything.

alexanderkjeldaas on Nov 14, 2021

Just ran into this myself while prepping for Kubernetes 1.22 upgrade.

The backward compatible workaround would be a hack reading the ingressClass field from the Ingress, generating a synthetic kubernetes.io/ingress.class: nginx annotation, and then filtering on that, but not only is that ugly, I am not sure it is even realistically feasible.

Given that the upgrade path requires non-automated changes to the Ingress resource itself, I think it is fine to require a non-automated change to external-dns, too. I propose a new filter --ingress-class-filter which filters on the Ingress.spec.ingressClassName value.

Not a substitute, but it would also be good if the project publish an official annotation for this, something like

external-dns.alpha.kubernetes.io/ingress-class

Nuru on Jun 1, 2022

I believe the solution is implemented in this unapproved PR https://github.com/kubernetes-sigs/external-dns/pull/2054.

gw0 on Mar 7, 2023

Totally agree with this. Please note that you can still use the --annotation-filter to filter other annotations defined by yourself on your ingress objects.

In my case I run two nginx instances and two external-dns instances, one for public network and the other one for private network.

For example, to workaround this issue I annotated my ingress resources with something like kubernetes.io/external-dns-class: "nginx-public" and in my “public” external-dns instance I configured the following with --annotation-filter=kubernetes.io/external-dns-class in (nginx-public). I then used the same approach for the “private” instances.

You can use any annotation you like, just be sure to match the filter in the external-dns configuration.

By using this method you workaround the warning/error and your gitops pipeline will work without issues.

Hope it helps!

calexandre on Oct 21, 2021