coredns: Enabling autopath in Kubernetes causes IPv6 issues

Hi all,

I enabled CoreDNS on my kops-managed Kubernetes 1.10.5 cluster last week and had it running in parallel to kube-dns to evaluate performance. Since my applications do a lot of external dns-requests, I was especially interested in enabling the autopath feature, to optimize all the search domain queries a bit.

However, I noticed that as soon as I enabled pods verified and autopath @kubernetes in my config, I would start getting ENETUNREACH errors from some applications trying to connect to IPv6 addresses. My cluster/VPC is not set up to be able to connect outwards on IPv6, so it makes sense to see these errors if the application is trying to reach an IPv6 address.

Trying to debug manually, I first of all noticed that for a query like dig google.com ANY @<pod-IP> I would get different kind of results from different pods. Some returned only an A record, some returned A and NS, some returned A, NS and AAAA and others returned A, NS, AAAA, SOA and TXT. I guess it somehow depends on what is in the cache, but I expected something a bit more consistent? Kube-DNS seems to behave similarly though. However, I noticed from time to time I would see a short moment where only an AAAA record was returned, which would cause my application to connect on IPv6 (IPv4 is always preferred, unless there are not v4 records). I have not been able to replicate this without the pods verified and autopath enabled. Only using pods verified without autopath also seems to behave as expected.

Happy to debug further, but am a bit lost myself on what could cause this.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 2
Comments: 21 (10 by maintainers)

Commits related to this issue

disabled AAAA (ipv6) in CoreDNS @see https://github.com/coredns/coredns/issues/2039#issuecomment-416588977 — committed to utopia-planitia/kubernetes by david-sauer-turbinekreuzberg 5 years ago

Most upvoted comments

As a hack, you can block all AAAA queries with the template plugin. (I didn’t test syntax below, but i think it should work)

    .:53 {
        errors
        health
        template ANY AAAA . {
          rcode NXDOMAIN
        }
        kubernetes cluster.local. in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        reload
    }

What I don’t fully understand is why enabling autopath for my kubernetes plugin affects the DNS resolution of external names

This is because with autopath, coredns does the domain search on behalf of the client. That includes the final . domain. There isn’t a way for coredns to tell the client to do that final query on it’s own - the client still thinks its on the first domain in the search list, so it would pick back up where it left off, by querying the 2nd domain in the list.

chrisohaver on Aug 28, 2018

Are there pcaps?

On Thu, 10 Jan 2019, 17:03 Carlos Rivera <notifications@github.com wrote:

@bcorijn https://github.com/bcorijn, @rocktavious https://github.com/rocktavious, the issue showed up again today 😦 I think it was a fluke that we didn’t see it yesterday after bumping node

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/coredns/coredns/issues/2039#issuecomment-453172922, or mute the thread https://github.com/notifications/unsubscribe-auth/AAVkW0wP8r4zWZ7CGcoB33-sB6QM6ipHks5vB3JYgaJpZM4WDrRn .

miekg on Jan 11, 2019