go: net/http/httptrace: panic on GotConn

What version of Go are you using (go version)?

$ go version
go version devel +03f63654c4 Thu Sep 12 21:02:57 2019 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/cuonglm/Library/Caches/go-build"
GOENV="/Users/cuonglm/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/cuonglm/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/cuonglm/sources/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/cuonglm/sources/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/y4/hs76ltbn7sb66lw_6934kq4m0000gn/T/go-build718378058=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

package main

import (
	"context"
	"log"
	"net/http"
	"net/http/httptrace"
	"sync"
)

func gotConn(info httptrace.GotConnInfo) {
	_ = info.Conn.RemoteAddr
}

func doGet() {
	ctx := httptrace.WithClientTrace(context.Background(), &httptrace.ClientTrace{
		GotConn: gotConn,
	})

	req, _ := http.NewRequest(http.MethodGet, "https://google.com", nil)
	req = req.WithContext(ctx)

	if _, err := http.DefaultClient.Do(req); err != nil {
		log.Println(err)
	}
}

func main() {
	var wg sync.WaitGroup
	n := 1000
	wg.Add(n)

	for i := 0; i < n; i++ {
		go func() {
			defer wg.Done()
			doGet()
		}()
	}

	wg.Wait()
}

Thanks @tmthrgd for simpler and not racy program.

What did you expect to see?

Run successfully.

What did you see instead?

Panic

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12ad645]

goroutine 72 [running]:
main.gotConn(0x0, 0x0, 0x1a500101, 0xc0ce52)
	/Users/cuonglm/sources/goissues/34282/trace.go:12 +0x5
net/http.(*Transport).getConn(0x15cc420, 0xc0025eec60, 0x0, 0x135530c, 0x5, 0xc0029f1650, 0xe, 0x0, 0x0, 0x0, ...)
	/Users/cuonglm/sdk/go1.13/src/net/http/transport.go:1210 +0x394
net/http.(*Transport).roundTrip(0x15cc420, 0xc00027e900, 0x0, 0xc00020ab20, 0x100e118)
	/Users/cuonglm/sdk/go1.13/src/net/http/transport.go:522 +0x603
net/http.(*Transport).RoundTrip(0x15cc420, 0xc00027e900, 0x15cc420, 0x0, 0x0)
	/Users/cuonglm/sdk/go1.13/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc00027e900, 0x13be5a0, 0x15cc420, 0x0, 0x0, 0x0, 0xc000282050, 0x203000, 0x1, 0x0)
	/Users/cuonglm/sdk/go1.13/src/net/http/client.go:250 +0x43a
net/http.(*Client).send(0x15d1200, 0xc00027e900, 0x0, 0x0, 0x0, 0xc000282050, 0x0, 0x1, 0x13479a0)
	/Users/cuonglm/sdk/go1.13/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0x15d1200, 0xc00027e900, 0x0, 0x0, 0x0)
	/Users/cuonglm/sdk/go1.13/src/net/http/client.go:641 +0x3ce
net/http.(*Client).Do(...)
	/Users/cuonglm/sdk/go1.13/src/net/http/client.go:509
main.main.func1(0xc00001c340, 0x13c3a40, 0xc0000ceab0)
	/Users/cuonglm/sources/goissues/34282/trace.go:31 +0x18c
created by main.main
	/Users/cuonglm/sources/goissues/34282/trace.go:25 +0xeb
exit status 2

The program works in go1.12, panic on go1.13 as well as tip.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 17 (15 by maintainers)

Most upvoted comments

In general, it’s better not to edit the original message. Now people reading the thread from the start will get confused by our initial discussion about code that doesn’t exist any more. I’ll mark our messages as outdated to hide them.

+2
ALTree on Sep 13, 2019

Here is a simpler reproducer that shows the same crash (Edit: simpler still):

package main

import (
	"context"
	"log"
	"net/http"
	"net/http/httptrace"
	"sync"
)

func gotConn(info httptrace.GotConnInfo) {
	_ = info.Conn.RemoteAddr // info.Conn == nil here
}

func doGet() {
	ctx := httptrace.WithClientTrace(context.Background(), &httptrace.ClientTrace{
		GotConn: gotConn,
	})

	req, _ := http.NewRequest(http.MethodGet, "https://google.com", nil)
	req = req.WithContext(ctx)

	if _, err := http.DefaultClient.Do(req); err != nil {
		log.Println(err)
	}
}

func main() {
	var wg sync.WaitGroup
	n := 1000
	wg.Add(n)

	for i := 0; i < n; i++ {
		go func() {
			defer wg.Done()
			doGet()
		}()
	}

	wg.Wait()
}
+2
tmthrgd on Sep 13, 2019

@gopherbot please consider this for backport 1.13

+1
cuonglm on Sep 13, 2019

CL 195237 fixes the panic, but is missing a test case.

@cuonglm If you’re looking for a temporary workaround, either:

http.DefaultTransport.(*http.Transport).DisableKeepAlives = true

or

http.DefaultTransport.(*http.Transport).MaxIdleConnsPerHost = -1

will stop the problematic persistConn from being put in the idle pool

+1
tmthrgd on Sep 13, 2019

git bisect points to 43b9fcf6fe9feb0ec67a7ddb01e5b542542b47c8

+1
cuonglm on Sep 13, 2019

@ALTree The example of @tmthrgd above is not racy and produce the panic.

I updated the issue with that program, thanks @tmthrgd .

+1
cuonglm on Sep 13, 2019
Read more comments on GitHub
← go: net/http/httptest: Server blocked in Close after 5 seconds, waiting for connections
go: net/http/httputil: Proxy terminates HTTP/2 stream before reading response body. →