external-dns: hetzner provider: AAAA records not set

What happened:

The hcloud-controller-manager added support for IPv6 in 1aea7ddac1e2a6ffd358feb147db9e5a9d71a02d. Note: This is not yet part of a release.

When using any version of the hcloud-ccm with this commit included, an ingress will report an IPv4 and an IPv6 address:

NAME    CLASS    HOSTS               ADDRESS                              PORTS     AGE
minio   <none>   minio.example.com   167.xxx.x.xx,2a01:xxx:xxxxx:xxx::1   80, 443   19m

external-dns then starts to:

not set any AAAA records
update all A records on every run (I think this is due to two records being desired, but only one being present)

What you expected to happen:

external-dns to add an A record for the IPv4 address and an AAAA record for the IPv6 records.

How to reproduce it (as minimally and precisely as possible):

Run the hcloud-ccm with e.g. the image hetznercloud/hcloud-cloud-controller-manager@sha256:b78fd1715be087791f7bd324f5de1aed4fc22209abc773a774e988e2f96dc500 and external-dns 0.7.6.

Have an ingress with both IPv4 and IPv6 addresses (e.g. by using Hetzner Cloud Loadbalancers) for a zone that is updated by external-dns.

Anything else we need to know?:

It would be great if we could get this fixed before hcloud-ccm gets a new release so that everyone will have a working external-dns immediately using hetzner-ccm.

If I can help out in any way by debugging this further or testing changes, let me know. I tried to understand the code and submit a PR myself, but failed to get all of it.

Tagging @21h as maintainer of the hetzner provider.

Environment:

External-DNS version (use external-dns --version): 0.7.6
DNS provider: hetzner
Others:
Using hetzner cloud controller manager with image hetznercloud/hcloud-cloud-controller-manager@sha256:b78fd1715be087791f7bd324f5de1aed4fc22209abc773a774e988e2f96dc500
Deployed with the bitnami/external-dns helm chart

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 4
Comments: 17 (6 by maintainers)

Most upvoted comments

/remove-lifecycle rotten

This issue won’t be resolved magically by having a bot close it.

morremeyer on Jan 15, 2022

TL;DR: external-dns does not support IPv6 in a generic way. AWS seems to be the only provider with dualstack support in external-dns. There is a rather new issue https://github.com/kubernetes-sigs/external-dns/issues/2051 which hopefully changes that (?).

After upgrading my hcloud cloud controller manager to 1.9.1 which includes the change from https://github.com/hetznercloud/hcloud-cloud-controller-manager/issues/121 I was facing the same issue. After a deep dive into the code of external-dns I think I have a rough understanding of the issue.

external-dns consists of different parts, for this issue the ingress source, the aws provider and the hcloud provider are of interest. The source fetches all ingress resources and figures out which DNS records should exist, those records are represented as endpoints. Then those are passed to the provider which is responsible to apply those to your actual DNS API.

So how does dualstack work for AWS in external-dns? In the ingress source setDualstackLabel checks the ingress for the specific AWS annotation for dualstack loadbalancers/ingress and marks the endpoint as dualstack: https://github.com/kubernetes-sigs/external-dns/blob/5806e3474f2e13254498bd2af34302a4e283ae39/source/ingress.go#L234 Then the AWS provider checks all endpoints for this and creates additional AAAA records if necessary: https://github.com/kubernetes-sigs/external-dns/blob/5806e3474f2e13254498bd2af34302a4e283ae39/provider/aws/aws.go#L561

For the hetzner provider there is no special handling of IPv6 so far, neither on the source side nor on the provider side. So the IP addresses of the ingress are extracted here https://github.com/kubernetes-sigs/external-dns/blob/5806e3474f2e13254498bd2af34302a4e283ae39/source/ingress.go#L295, together with the hosts of the ingress rules the IP addresses are passed on to https://github.com/kubernetes-sigs/external-dns/blob/5806e3474f2e13254498bd2af34302a4e283ae39/source/source.go#L196 which uses suitableType to figure out if it is an IP or a CNAME: https://github.com/kubernetes-sigs/external-dns/blob/5806e3474f2e13254498bd2af34302a4e283ae39/source/source.go#L188. ParseIP from go (https://golang.org/pkg/net/#ParseIP) correctly identifies both our IP addresses as IP addresses and external-dns concludes that both have to be IPv4 addresses and continues to tell the provider to create two A records. The hetzner provider then tries to create/update an A record with an IPv6 address which silently fails and is retried during every reconcile loop.

Another thing I noticed while debugging this: It looks like the hetzner provider does not respect the --dry-run flag of external-dns and always applies the changes. But that’s probably another issue we should open 😉

ekeih on May 1, 2021