go: net/http/httputil: ReverseProxy read error during body copy use of closed network connection

What version of Go are you using (go version)?

$ go version
go version go1.14.2 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Administrator\AppData\Local\go-build
set GOENV=C:\Users\Administrator\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=D:\wxgo
set GOPRIVATE=
set GOPROXY=https://goproxy.io,direct
set GOROOT=D:\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=D:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessag
e-length=0 -fdebug-prefix-map=C:\Users\ADMINI~1\AppData\Local\Temp\go-build74128
8550=/tmp/go-build -gno-record-gcc-switches

What did you do?

  • I created a proxy and a restfulapi server
  • I used postman made many post requests with body request to server through the proxy(proxy and server run in the same vm,server runed used docker,They use docker0 to connect),and server response body is larger than 4K
  • all requests use http/1.1
  • OS:Centos7.4

What did you expect to see?

post request sucessed and there is only one persistent connection

What did you see instead?

the following log appears in proxy .

httputil: ReverseProxy read error during body copy: read tcp 172.17.0.1:39810->172.17.0.2:8082: use of closed network connection

many TIME_WAIT conntecions appears . Everything was ok in any of the the following scenarios:

  • Post request without body
  • GET request with same response
  • Response body less than 4K
  • Do not use Keep-Alive
  • Server response not write “Content-Length” in header
  • Use NGINX instead of my proxy
  • Server runed in a vm
PROXY Output
package main

import ( “flag” “net/http” “net/http/httputil” “net/url” “time” )

var burl = flag.String(“u”, “”, “backend address”)

func buildProxy(target *url.URL) *httputil.ReverseProxy { flushInterval := time.Duration(100 * time.Millisecond) passHost := true passHostHeader := &passHost targetQuery := target.RawQuery proxy := &httputil.ReverseProxy{ Director: func(req *http.Request) { req.URL.Path = target.Path req.URL.Host = target.Host req.URL.Scheme = target.Scheme

		req.URL.RawPath = target.RawPath
		if targetQuery == "" || req.URL.RawQuery == "" {
			req.URL.RawQuery = targetQuery + req.URL.RawQuery
		} else {
			req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery
		}
		req.RequestURI = ""
		req.Proto = "HTTP/1.1"
		req.ProtoMajor = 1
		req.ProtoMinor = 1
		if passHostHeader != nil && !*passHostHeader {
			req.Host = req.URL.Host
		}
	},
	Transport:     http.DefaultTransport,
	FlushInterval: flushInterval,
}
return proxy

}

func handler(p *httputil.ReverseProxy) func(http.ResponseWriter, *http.Request) { return func(w http.ResponseWriter, r *http.Request) {

	p.ServeHTTP(w, r)
}

} func main() { flag.Parse() remote, err := url.ParseRequestURI(“http://” + *burl) if err != nil { panic(err) }

proxy := buildProxy(remote)
http.HandleFunc("/", handler(proxy))
err = http.ListenAndServe(":8081", nil)
if err != nil {
	panic(err)
}

}

SERVER Output
package main

import ( “fmt” “strconv” “github.com/gin-gonic/gin” )

type myJson struct { Name string json:"name" }

func createBigString(name string) string {

var bs string
for i := 1; i <= 500; i++ {
	bs += name + "abcdefghij"

}

return bs

} func main() { gin.SetMode(gin.ReleaseMode) router := gin.Default()

router.POST("/test/post", func(c *gin.Context) {
	var json myJson
	c.ShouldBindJSON(&json)
	myName := createBigString(json.Name)
	lenBody := len(myName) + 11
	fmt.Println(lenBody)
	c.Header("Content-Length", strconv.Itoa(lenBody))
	c.JSON(200, gin.H{"name": myName})
})
router.Run(":8082")

}

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 9
  • Comments: 22 (6 by maintainers)

Most upvoted comments

Sorry for the lengthy time in getting to this.

Thanks to @markusthoemmes for the reproduction case and @stephens2424 for the analysis, both were very helpful in understanding what’s going on.

The basic problem here is that the HTTP/1 server implementation does not support reading from the inbound request body after beginning to write the outbound response. On writing the response, any unread portion of the inbound body is consumed (up to a limit), to avoid deadlocking clients that expect to write the complete request before reading any of the response.

#15527 is a long-standing feature request to disable this behavior, allowing concurrent reads from the request and writes to the response.

https://github.com/golang/go/issues/15527#issuecomment-401168128 has some context on why the server implementation behaves the way it does.

The issue here is that ReverseProxy starts writing the proxied response before it finishes reading from the proxied request.

In the reproduction case above, this manifests as a very racy failure because the server behind the proxy consumes the full request before it starts responding. I believe that what’s happening is ReverseProxy’s server is closing the request body (which is both the inbound and outbound body) in between the transport’s two reads here (the first reading the full body, and the second expecting to just read an io.EOF):

https://github.com/golang/go/blob/16cec4e7a0ed4b1e5cf67a07a1bf24bdf2f6b04c/src/net/http/transfer.go#L370-L375

If the upstream server starts responding before it fully consumes the proxied request, the failure manifests consistently when the request is sufficiently large. A 1MiB request proxied to a server that does io.Copy(w, r.Body) is a good reproduction case. (Note that in this case the proposed fix in https://github.com/golang/go/issues/40747#issuecomment-926233470 does not work, because the body has not been read to EOF when the response headers are written.)

There are several ways we can address this.

We can change ReverseProxy to wait for the transport to finish consuming an HTTP/1 request body before it begins proxying the response. This will make ReverseProxy adhere to the requirements spelled out in the http.ResponseWriter documentation. I think this should be a safe change to make, since any cases where a user might want to concurrently send the proxied request body while receiving the response are already broken. (This does work for HTTP/2, so we’d want to take care not to break that case.) I think this is a good first step.

In the longer run, however, there has been a long-standing feature request (#15527) to permit HTTP/1 server handlers to concurrently read a request and write a response. If we supported this, then ReverseProxy could just behave as it does now. There are good reasons why this probably can’t be the default server behavior, but we can support it on an opt-in basis. I’ve filed #57786 with a proposed API for doing so.

Hi any news on this one (any plans for a fix)? /cc @dmitshur @stephens2424.

FWIW, my reproducer above sadly still fails on Golang 1.16.3, in case anybody was wondering.