go: net/http/httputil: ReverseProxy read error during body copy use of closed network connection
What version of Go are you using (go version
)?
$ go version go version go1.14.2 windows/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env
)?
go env
Output
$ go env set GO111MODULE= set GOARCH=amd64 set GOBIN= set GOCACHE=C:\Users\Administrator\AppData\Local\go-build set GOENV=C:\Users\Administrator\AppData\Roaming\go\env set GOEXE=.exe set GOFLAGS= set GOHOSTARCH=amd64 set GOHOSTOS=windows set GOINSECURE= set GONOPROXY= set GONOSUMDB= set GOOS=windows set GOPATH=D:\wxgo set GOPRIVATE= set GOPROXY=https://goproxy.io,direct set GOROOT=D:\Go set GOSUMDB=sum.golang.org set GOTMPDIR= set GOTOOLDIR=D:\Go\pkg\tool\windows_amd64 set GCCGO=gccgo set AR=ar set CC=gcc set CXX=g++ set CGO_ENABLED=1 set GOMOD= set CGO_CFLAGS=-g -O2 set CGO_CPPFLAGS= set CGO_CXXFLAGS=-g -O2 set CGO_FFLAGS=-g -O2 set CGO_LDFLAGS=-g -O2 set PKG_CONFIG=pkg-config set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessag e-length=0 -fdebug-prefix-map=C:\Users\ADMINI~1\AppData\Local\Temp\go-build74128 8550=/tmp/go-build -gno-record-gcc-switches
What did you do?
- I created a proxy and a restfulapi server
- I used postman made many post requests with body request to server through the proxy(proxy and server run in the same vm,server runed used docker,They use docker0 to connect),and server response body is larger than 4K
- all requests use http/1.1
- OS:Centos7.4
What did you expect to see?
post request sucessed and there is only one persistent connection
What did you see instead?
the following log appears in proxy .
httputil: ReverseProxy read error during body copy: read tcp 172.17.0.1:39810->172.17.0.2:8082: use of closed network connection
many TIME_WAIT conntecions appears . Everything was ok in any of the the following scenarios:
- Post request without body
- GET request with same response
- Response body less than 4K
- Do not use Keep-Alive
- Server response not write “Content-Length” in header
- Use NGINX instead of my proxy
- Server runed in a vm
PROXY
Output
package mainimport ( “flag” “net/http” “net/http/httputil” “net/url” “time” )
var burl = flag.String(“u”, “”, “backend address”)
func buildProxy(target *url.URL) *httputil.ReverseProxy { flushInterval := time.Duration(100 * time.Millisecond) passHost := true passHostHeader := &passHost targetQuery := target.RawQuery proxy := &httputil.ReverseProxy{ Director: func(req *http.Request) { req.URL.Path = target.Path req.URL.Host = target.Host req.URL.Scheme = target.Scheme
req.URL.RawPath = target.RawPath if targetQuery == "" || req.URL.RawQuery == "" { req.URL.RawQuery = targetQuery + req.URL.RawQuery } else { req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery } req.RequestURI = "" req.Proto = "HTTP/1.1" req.ProtoMajor = 1 req.ProtoMinor = 1 if passHostHeader != nil && !*passHostHeader { req.Host = req.URL.Host } }, Transport: http.DefaultTransport, FlushInterval: flushInterval, } return proxy
}
func handler(p *httputil.ReverseProxy) func(http.ResponseWriter, *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
p.ServeHTTP(w, r) }
} func main() { flag.Parse() remote, err := url.ParseRequestURI(“http://” + *burl) if err != nil { panic(err) }
proxy := buildProxy(remote) http.HandleFunc("/", handler(proxy)) err = http.ListenAndServe(":8081", nil) if err != nil { panic(err) }
}
SERVER
Output
package mainimport ( “fmt” “strconv” “github.com/gin-gonic/gin” )
type myJson struct { Name string
json:"name"
}func createBigString(name string) string {
var bs string for i := 1; i <= 500; i++ { bs += name + "abcdefghij" } return bs
} func main() { gin.SetMode(gin.ReleaseMode) router := gin.Default()
router.POST("/test/post", func(c *gin.Context) { var json myJson c.ShouldBindJSON(&json) myName := createBigString(json.Name) lenBody := len(myName) + 11 fmt.Println(lenBody) c.Header("Content-Length", strconv.Itoa(lenBody)) c.JSON(200, gin.H{"name": myName}) }) router.Run(":8082")
}
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 9
- Comments: 22 (6 by maintainers)
Sorry for the lengthy time in getting to this.
Thanks to @markusthoemmes for the reproduction case and @stephens2424 for the analysis, both were very helpful in understanding what’s going on.
The basic problem here is that the HTTP/1 server implementation does not support reading from the inbound request body after beginning to write the outbound response. On writing the response, any unread portion of the inbound body is consumed (up to a limit), to avoid deadlocking clients that expect to write the complete request before reading any of the response.
#15527 is a long-standing feature request to disable this behavior, allowing concurrent reads from the request and writes to the response.
https://github.com/golang/go/issues/15527#issuecomment-401168128 has some context on why the server implementation behaves the way it does.
The issue here is that
ReverseProxy
starts writing the proxied response before it finishes reading from the proxied request.In the reproduction case above, this manifests as a very racy failure because the server behind the proxy consumes the full request before it starts responding. I believe that what’s happening is
ReverseProxy
’s server is closing the request body (which is both the inbound and outbound body) in between the transport’s two reads here (the first reading the full body, and the second expecting to just read an io.EOF):https://github.com/golang/go/blob/16cec4e7a0ed4b1e5cf67a07a1bf24bdf2f6b04c/src/net/http/transfer.go#L370-L375
If the upstream server starts responding before it fully consumes the proxied request, the failure manifests consistently when the request is sufficiently large. A 1MiB request proxied to a server that does
io.Copy(w, r.Body)
is a good reproduction case. (Note that in this case the proposed fix in https://github.com/golang/go/issues/40747#issuecomment-926233470 does not work, because the body has not been read to EOF when the response headers are written.)There are several ways we can address this.
We can change
ReverseProxy
to wait for the transport to finish consuming an HTTP/1 request body before it begins proxying the response. This will makeReverseProxy
adhere to the requirements spelled out in thehttp.ResponseWriter
documentation. I think this should be a safe change to make, since any cases where a user might want to concurrently send the proxied request body while receiving the response are already broken. (This does work for HTTP/2, so we’d want to take care not to break that case.) I think this is a good first step.In the longer run, however, there has been a long-standing feature request (#15527) to permit HTTP/1 server handlers to concurrently read a request and write a response. If we supported this, then
ReverseProxy
could just behave as it does now. There are good reasons why this probably can’t be the default server behavior, but we can support it on an opt-in basis. I’ve filed #57786 with a proposed API for doing so.Hi any news on this one (any plans for a fix)? /cc @dmitshur @stephens2424.
FWIW, my reproducer above sadly still fails on Golang 1.16.3, in case anybody was wondering.