go: net: dns lookup sometimes fails with "no such host" when there are too many open files

What did you do?

My network-intensive code would sometimes fail resolving domain names, returning “no such host” using the default resolver (pure go).The domains that fail resolving are valid domains that are successfully resolved using dig and the same dns server address.

I noticed that increasing the number of open files limit fix the issue.

I managed to reproduce the failure with: https://play.golang.org/p/MEZUS8h-o5

go build test.go
ulimit -n 20 # Set open file limit to something low
$ ./test 
2017/01/10 11:01:11 lookup munic.io on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
$ ./test 
2017/01/10 11:01:11 lookup munic.io on 127.0.1.1:53: dial udp 127.0.1.1:53: socket: too many open files
$ ./test 
2017/01/10 12:01:12 lookup munic.io on 127.0.1.1:53: no such host

What did you expect to see?

Always “too many open files”

What did you see instead?

Sometimes Lookup returns “no such host” when it shouldn’t.

Does this issue reproduce with the latest release (go1.7.4)?

I first noticed the problem with 1.7.3. I reproduced the problem on 1.8beta2.

System details

go version go1.8beta2 linux/amd64
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/costanzj/projects/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build335463609=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
GOROOT/bin/go version: go version go1.8beta2 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.8beta2 X:framepointer
uname -sr: Linux 4.4.0-38-generic
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.1 LTS
Release:	16.04
Codename:	xenial
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Ubuntu GLIBC 2.23-0ubuntu5) stable release version 2.23, by Roland McGrath et al.
gdb --version: GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1

About this issue

Original URL
State: open
Created 7 years ago
Reactions: 1
Comments: 24 (14 by maintainers)

Most upvoted comments

I’m running into this as well on go 1.12.5. I’d like to help if there’s a decision made on how to approach on solving this.

simar7 on Jun 30, 2019

5 years later, this is still an issue

matti on Aug 23, 2022

I can believe that it’s a glibc problem, although I just saw this same problem happen on a Mac. On the Mac we’re also invoking C code to run the lookup (because DNS traffic is blocked for ordinary Go code), so maybe the Mac library has the same bug.

rsc on Jan 31, 2023

I experienced the same problem. And, I did tried to compile code with GODEBUG=netdns=go CGO_ENABLED=0 but problem persists. I did adjusted ulimit for number of files too (e.g. my limit was set to 1048576) and yet the same issue appears. And, I’ve seen this issue in Linux, macOS, and in k8s environments. What seems to be working is literally reduce number of concurrent calls and use a queue mechanism in the application.

vkuznet on Jun 6, 2021

@batara666 If you didn’t get a too many open files error, then that is a different problem.

ianlancetaylor on Jan 31, 2023