go: net: DNS resolves slowly

@nussjustin i used httptrace to trace the request, the code like this:

package main

import (
	"fmt"
	"io/ioutil"
	"net"
	"net/http"
	"net/http/httptrace"
	"strings"
	"time"
)

func main() {
	
	dialer := &net.Dialer{Timeout: 7 * time.Second}
	// For security's sake, sorry, I can't provide detailed domain names
        url := ""
       // json content
	payload := strings.NewReader("")

	req, _ := http.NewRequest("POST", url, payload)

	req.Header.Add("content-type", "application/json")
	req.Header.Add("cache-control", "no-cache")
	
	trace := &httptrace.ClientTrace{
		DNSStart: func(dnsInfo httptrace.DNSStartInfo) {
			fmt.Printf("dns start: %v\n", time.Now())
		},
		DNSDone: func(dnsDoneInfo httptrace.DNSDoneInfo) {
			fmt.Printf("dns end: %v\n", time.Now())
		},
	}
	req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))

	transport := &http.Transport{

		DialContext:         (dialer).DialContext,
		Dial:                (dialer).Dial,
		TLSHandshakeTimeout: 2 * time.Second,
	}
	
	client := &http.Client{
		Transport: transport,
		Timeout:   time.Duration(10000) * time.Millisecond,
	}


	now := time.Now()

	res, err := client.Do(req)

	cost := time.Since(now)
	if err != nil {
		fmt.Println(err)
		return
	}

	defer res.Body.Close()
	body, _ := ioutil.ReadAll(res.Body)

	fmt.Println(string(body))
	fmt.Printf("client cost: %v\n", cost.Seconds()*1e3)


}

and get the result :

dns start: 2017-09-26 15:45:28.148847712 +0800 CST
dns end: 2017-09-26 15:45:33.15817369 +0800 CST
client cost: 5114.887975000001

I believe I’ve found a way to reproduce the issue using a public domain name. The /etc/resolv.conf file on the Linux machine looked like this when the 10+ second DNS lookup times were happening with Go 1.11 (and not with Go 1.10) with subdomains ending in faa.gov:

domain amc.faa.gov
search amc.faa.gov act.faa.gov ad.faa.gov faa.gov jccbi.gov 
nameserver 172.25.84.100
nameserver 172.25.116.100
nameserver 172.19.50.27

The last nameserver 172.19.50.27 entry was an old entry that no longer resolved to an actual machine. Public domains like www.google.com or www.yahoo.com were resolving within tens of milliseconds even with this configuration. However, if I add, for example, yahoo.com to the search line, now Go 1.11 takes 10+ seconds to resolve sports.yahoo.com (trying all nameserver entries); whereas, Go 1.10 resolves within tens of milliseconds with this configuration (only using the first nameserver entry). Also, host and dig both resolve sports.yahoo.com with a few milliseconds using this configuration. It appears this configuration causes Go-based DNS lookup on 1.11 to try all nameservers listed in the /etc/resolv.conf; whereas, Go-based DNS lookup on 1.10 and standard command line tools only use the first nameserver entry.

In my service, the net.Resolver stalling 5s+ occurred once every 150 queries. This service is built in go1.11 .

The AAAA reponse alway this: