coredns: plugin/autopath, plugin/cache: Reproducing bugs around autopath

What happened:

CoreDNS gets confused about namespace/service ownership of PodIPs
The problem does not appear to resolve with time
Primary issue: A race condition between two requests can cause an authoritative NXDOMAIN on an IPv6 request and a NOERROR on the IPv4 request for an autopath’d request - This combo causes the stub resolver to stop searching search domains and return to the caller that the name does not exist (rather than proceeding through search domains).
Secondary issue: The cache, in cases beyond where autopath is involved, can cache records in a way that cause CoreDNS to violate the guidance given in RFC 4074 (https://tools.ietf.org/html/rfc4074) as there is no coordination between any of the records for the same name in the cache. For example, when adding a NoError response to the cache, any other NxDomain records for the same name should be removed (for example, across A/AAAA records).
Possible additional issues:
- Reverse resolution for a PodIP will probably return the namespace/pod name
- Autopath + Pods/Verified has danger that is more drastic that the documentation notes imply. Autopath has a race condition (because of the way Pods/Verified works) where it will always (as in: cannot stop it from happening, not that it happens on every request) result in a stub resolver thinking that a name does not exist. The percentage of times that this can happen depends on how frequently pods start and stop in a given cluster.

What you expected to happen:

See above.

How to reproduce it (as minimally and precisely as possible):

I can provide the reproduction instructions after confirmation that there is interest in fixing these bugs.

As a preview, the issue is that Kubernetes will reuse PodIPs immediately in different namespaces. When a PodIP is reused it has been observed that CoreDNS does not end up with a correct internal state about what namespace/service that PodIP belongs to. I did not look further to figure out if the bug is in Kubernetes itself, in the Kubernetes client API, or in the usage of the Kubernetes Client API.

Anything else we need to know?:

[TBC]

Environment:

the version of CoreDNS: master
Corefile:
logs, if applicable: there are no log messages in the app that would catch this condition - we should consider adding verbose logging messages that can be enabled to troubleshoot this kind of problem without having to modify and recompile the app
OS (e.g: cat /etc/os-release):
Others:

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 4
Comments: 46 (29 by maintainers)

Most upvoted comments

Current workaround there is a block of:
.:53 {
 ...
 template ANY AAAA . {
   rcode NXDOMAIN
   fallthrough
 }
 ...
}

Ok. That fails with newer alpine images though: https://git.musl-libc.org/cgit/musl/commit/?id=5cf1ac2443ad0dba263559a3fe043d929e0e5c4c (since musl-1.2.1)

(From bug report: https://gitlab.alpinelinux.org/alpine/aports/-/issues/11879 )

Replacing NXDOMAIN with NOERROR (=NODATA) may work, or get back to a buggy state of things.

wdoekes on Feb 3, 2021

We ran into the same issue with AAAA returning NOERROR while A returned NXDOMAIN after upgrading a CoreDNS from 1.2.6 to 1.6.6 on another cluster.

~Current workaround there is a block of:~

.:53 {
  ...
  template ANY AAAA . {
    rcode NXDOMAIN
    fallthrough
  }
  ...
}

(edit: see below; NXDOMAIN is no good, but NOERROR possibly is)

Obviously this workaround is worse than using ossobv/nss-dns4only as this breaks quad-A resolving, even when explicitly requested. But on the upside, it works with Alpine images.

wdoekes on Feb 3, 2021

Thanks, I have DNS query client tool i use for testing that I can modify for this, and logging mods are straight forward.

For rough comparison purposes, do you recall the makeup of the cluster such as how many nodes, and the compute resources of the nodes and master?

chrisohaver on Mar 31, 2020