kubernetes: ndots breaks DNS resolving

I bet your pod image is based on alpine.

You can customize pod dns similar to the example here - https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-config You can set ndots to 1.

From earlier comments, this seems like a musl implementation issue rather than kubernetes dns? https://bugs.alpinelinux.org/issues/9017 says the dnsclient stops trying other searchpaths if a previous one returns unexpected error. Looks like this unexpected error is returned from whatever server is the upstream for the node. Maybe cloudFlare, something else. Setting ndots to a small value like 1, will ensure searchpath expansion does not kick in for hostnames with atleast 2 labels.

Based on this, i am closing the issue, please reopen if needed.

/close

I am also experiencing this same issue. Setting spec.dnsConfig.options to {name: ndots, value: "1"} does fix it. However, I don’t want to do this for every container, is there any way that I can change the global default or do something else to resolve this.

Any help is appreciated

You can customize pod dns similar to the example here - https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-config

Correct me if I’m wrong, but this I would have to do manually for each pod. I would like to configure this globally and make it the default: I’m experiencing random DNS failures with my GitLab-Runner, which starts multiple new temporary pods for running the individual jobs: They seem to inherit the default ndots:5 and fail to resolve the external DNS entry. I’m still investigating the underlying problem, but it mostly happens when the DNS cache is cold, e.g. I’m starting jobs after a period if inactivity. If I restart those failing jobs manually, the 2nd time the succeed and are able to resolve the external DNS entry. https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4129

sorry to comment on a closed topic, but was a global ndots option ever introduced?

For anyone that came here looking for solution for very similar issue where eg wget or other apps reported bad domain but nslookup worked fine also ndots:1 resolves issue but the same container works on other cluster. The issue was in extra . at the end of search in resolv.conf: search default.svc.cluster.local svc.cluster.local cluster.local . that dot came from search . on the host, and removing that line resolved dns issues

Running into the same issue, been doing the dnsConfig as an override for applications that experience this. Would love a longterm solution.

Running Kubes 1.11 with CoreDNS on bare-metal

EDIT: Also, I realized by containers aren’t running alpine image? E.g. golang:1.10.3

i’ve got the same issue with some container today :

Linux test-77fd4f49f7-m5b75 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 Linux

this is my resolve.conf file :

nameserver 10.43.0.10
search gitlab.svc.cluster.local svc.cluster.local cluster.local darkink.internal.
options ndots:5

if i remove (this is not very possible because some container are build from gitlab and others) this line : options ndots:5 my dns is working correctly again. The issue is only on some host (not all) and if i add a . at the end of the host everything is resolving corretly

Can someone help ?

@davidkarlsen could you try this image davidzqwang/alpine-dns:3.7 on your cluster, I made following changes to musl, build a new image based on alpine:3.7, and I test on my cluster, it seems work.

diff --git a/src/network/lookup_name.c b/src/network/lookup_name.c
index 209c20f..abb7da5 100644
--- a/src/network/lookup_name.c
+++ b/src/network/lookup_name.c
@@ -202,7 +202,7 @@ static int name_from_dns_search(struct address buf[static MAXADDRS], char canon[
                        memcpy(canon+l+1, p, z-p);
                        canon[z-p+1+l] = 0;
                        int cnt = name_from_dns(buf, canon, canon, family, &conf);
-                       if (cnt) return cnt;
+                       if (cnt > 0 || cnt == EAI_AGAIN) return cnt;
                }
        }

@tesharp, well, even with blank I get the same issue. Strange thing is that it works for some time and suddenly breaks…

@davidkarlsen, you can override the ndots option on the ClusterFirst policy by setting spec.dnsConfig.options to {name: ndots, value: "1"}

This feature was first introduced in 1.9

ref: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service#pod-s-dns-config

spec: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/network/pod-resolv-conf.md

for me it doesn’t work even with ndots:1