docker-alpine: Not resolving using search domain

I’m not sure why this issue is closed as I’ve tested with the alpine:latest aka alpine:3.4, and it still has this problem with not honoring search ... in /etc/resolv.conf. I’ve also tried some of the images and fixes that claim to have resolved the problem. None of them did for me.

This is a major problem for containers deployed into a k8s cluster, where k8s service is a critical part of the end-to-end user experience. For example, the most downloaded image from docker hub is nginx, and it’s based on alpine, and it’s broken when deployed into k8s.

Just because no one has mentioned it yet… DNS search/domain resolution is really important in a kubernetes cluster. Kubernetes services are used for just about every integration point and they all resolve through DNS. Kubernetes injects the right information into /etc/resolv.conf to do this.

The fact that I can’t connect to myservice.mynamespace from within an alpine container prevents me from doing anything useful. For example, I have a service called etcd.system that resolves to etcd host for http puts and gets. A very common thing is to write to etcd via curl for application configuration, etc. Alpine is the smallest base image that easily lets you download and use curl, but because my kubernetes service, etcd.system, isn’t resolvable I am dead in the water.

I now have to use a 100 meg image to curl, when the 6 meg one would have done fine.

Just food for thought.

Very sad to see this broken in both 3.4 and the edge on k8s.

• It works without search:

/ # cat /etc/resolv.conf
#search ivan.svc.cluster.local svc.cluster.local cluster.local in.foo.bar.
nameserver 10.32.0.10
options ndots:5
/ # ping -c 1 google.com
PING google.com (216.58.208.142): 56 data bytes
64 bytes from 216.58.208.142: seq=0 ttl=36 time=0.324 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.324/0.324/0.324 ms

• Doesn’t work with search:

/ # cat /etc/resolv.conf
search ivan.svc.cluster.local svc.cluster.local cluster.local in.foo.bar.
nameserver 10.32.0.10
options ndots:5
/ # ping -c 1 google.com
ping: bad address 'google.com'

alpine does not support multiple nameservers as well. Only one

This issue still exists with current alpine:3.4, alpine:3.5 and alpine:edge.

/ # nslookup kubernetes.default
nslookup: can't resolve '(null)': Name does not resolve

Name:      kubernetes.default
Address 1: 10.254.0.1 kubernetes.default.svc.cluster.local
/ # nslookup google.com
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'google.com': Name does not resolve
/ # cat /etc/resolv.conf 
search testspace.svc.cluster.local svc.cluster.local cluster.local testcluster.example.com
nameserver 10.254.0.10
options ndots:5
/ #

The problem remained for me: I tried using alpine:3.4 and alpine:edge …

/etc/resolv.conf:

search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.254.0.10
nameserver 185.48.116.10
nameserver 185.48.118.6
options ndots:5

nslookup kubernetes.default

nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'kubernetes.default': Name does not resolve

nslookup kubernetes

nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'kubernetes': Name does not resolve

Still have an issue with alpine 3.4 / kube-dns:1.9 Hope that kube-dns:2.x.x would fix this.

Sure, there is Alpine-kubernetes: A base image that comes with a special sauce for giving SEARCH domain powers to Docker Alpine. See here: https://github.com/janeczku/docker-alpine-kubernetes.

I’ve stumbled onto this same issue, and I’ve solved it removing the a search directive, which was triggering a CloudFlare DNS (buggy) resolution.

I’ve found out about the solution with this blog post: https://blog.maio.me/alpine-kubernetes-dns/

Try this:

echo "options single-request-reopen" >> /etc/resolv.conf

@ansel1 @andyshinn just fyi - This is already included in 1.2.0 alpha line - https://github.com/kubernetes/kubernetes/releases/tag/v1.2.0-alpha.5 - SkyDNS is the only NS for Pods with DNSPolicy=ClusterFirst (#15645, @ArtfulCoder)

Alternatively we have a container called Resolvable that helps: https://github.com/gliderlabs/resolvable

same issue with alpine 3.5 and kubernetes 1.5.x 😦 Sometimes it works, and sometimes does not.

bash-4.3# nslookup zookeeper.kafka
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'zookeeper.kafka': Name does not resolve
bash-4.3# nslookup zookeeper.kafka
nslookup: can't resolve '(null)': Name does not resolve

Name:      zookeeper.kafka
Address 1: 10.3.0.142
bash-4.3#

Even based on its own documentation this would appear to be a bug in musl libc:

musl’s resolver previously did not support the “domain” and “search” keywords in resolv.conf. This feature was added in version 1.1.13, but its behavior differs slightly from glibc’s: queries with fewer dots than the ndots configuration variable are processed with search first then tried literally (just like glibc), but those with at least as many dots as ndots are only tried in the global namespace (never falling back to search, which glibc would do if the name is not found in the global DNS namespace). This difference comes from a consistency requirement not to return different results subject to transient failures or to global DNS namespace changes outside of one’s control (addition of new TLDs).

While I can confirm the second part (queries greater than ndots never fall-back to using search), the first part (queries smaller than ndots fall-back to using an absolute query) isn’t what I observe.

Using dig on an Ubuntu container and attempting to resolve the nonsensical query google.com.default.svc.cluster.local (simulates the type of initial query for a short domain that would be occurring) returns a QUESTION SECTION and an AUTHORITY SECTION, but no ANSWER SECTION. This should cause musl libc to attempt to resolve the absolute query (google.com) instead, yet it doesn’t seem to based on the final result of the query.

Here’s the (tiny) commit where support for search and domain was added to musl libc, and here’s the name_from_dns function that that diff relies on. I think this dns_parse_callback function maybe the thing that determines whether we consider we’ve received a result or not, yet the code indicates this would only occur if we receive either an A, AAAA or CNAME record, yet in our case there’s no ANSWER SECTION whatsoever.

… 😕

I set up a dnsmasq container. In resolv.dnsmasq.conf, I added

nameserver 8.8.8.8
search example.com

In alpine container I set the dns pointing to that dnsmasq container.

The problem is when I do ping base in alpine container, the log of dnsmasq shows that only base is sent to 8.8.8.8, but base.example.com is never queried.

I think it could use a full dnsmasq config for the custom+upstream DNS scenario so people can figure out how to run it on the host, then a pointer to a good generic dnsmasq Docker container that would be easy to apply that configuration for people that want to run it in a container, and lastly, once available, a pointer to our magic resolve container.

On Tue, Mar 24, 2015 at 3:36 PM, Aris Pikeas notifications@github.com wrote:

A brief description of dnsmasq and how to use it - on the host, as an additional container, etc. In other words, specific instructions on use, rather than “maybe try this workaround”.

— Reply to this email directly or view it on GitHub https://github.com/gliderlabs/docker-alpine/issues/8#issuecomment-85683529 .

Jeff Lindsay http://progrium.com