go: x/net/http2: misbehaved streams can cause connections to exhaust flow control
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go1.11 darwin/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jnewman/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jnewman/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_t/hg9f_j4x1q743pqh00kdv6m00000gn/T/go-build803039253=/tmp/go-build -gno-record-gcc-switches -fno-common"
What did you do? / What did you expect to see? / What did you see instead?
Use httputil.ReverseProxy with an http2.Transport. I’m still trying to find a minimal reproducible example.
Basically, the http2.Transport is closing an HTTP/2 stream and then sending a number of DATA frames on the connection. The http2.Server is detecting that the stream has been half-closed by the remote and is following RFC7540 and responding with an ErrCodeStreamClosed (see http2/server.go).
However, since the http2.Server does not send a window update the remote http2.Transport does not add back the sent DATA frames to its flow control. This causes a re-used http2.Transport to eventually expend all connection flow control and halt sending.
A proposed fix that I’ve applied to my fork is to make the following patch
diff --git a/http2/server.go b/http2/server.go
index 56859d1..7a39456 100644
--- a/http2/server.go
+++ b/http2/server.go
@@ -1586,6 +1586,8 @@ func (sc *serverConn) processData(f *DataFrame) error {
// "open" or "half-closed (local)" state, the recipient MUST respond with a
// stream error (Section 5.4.2) of type STREAM_CLOSED.
if state == stateClosed {
+ sc.inflow.take(int32(f.Length))
+ sc.sendWindowUpdate(nil, int(f.Length))
return streamError(id, ErrCodeStreamClosed)
}
if st == nil || state != stateOpen || st.gotTrailerHeader || st.resetQueued {
Although, one could argue that http2.Transport/httputil.Reverse proxy should be fixed to make sure that data frames aren’t sent after a close.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 3
- Comments: 16 (6 by maintainers)
Commits related to this issue
- http2: Revert a closed stream cannot receive data This reverts CL 111676 for golang/go#25023. Reason for revert: The code change no longer issued a WindowUpdate which is required when processing dat... — committed to golang/net by fraenkel 6 years ago
- net/http: update bundled http2 Updates bundled http2 to x/net git rev ed066c81e7 for: http2: Revert a closed stream cannot receive data https://golang.org/cl/153977 Updates golang/go#28204 ... — committed to golang/go by bradfitz 5 years ago
The issue reported was fixed.
I don’t know enough to say if it’s Server or Transport, but I guess Server. It’s an accounting error in
http2serverConn.flow, in the connection-level flow control.What I think is happening is that when the HTTP handler returns without reading the body (say it returns a 404), any DATA frames in flight are counted towards the connection-level flow control by the client, but not by the server, so they get out of sync.
Once the HTTP handler returns the server closes the stream like this:
The in-flight DATA frames are considered invalid:
If I print out the flow control values (
http2flow.n) at the connection level, and use the client/server from my previous comment, when it blocks the server has 1,073,807,359 bytes but the client has 0.This is the same as @prashantv described above.
They added a clarification to the spec for this:
@bcmills @fraenkel - quick ping on this issue, the http/2 flow control stalls are still biting us a fair bit. (Sorry, I’m sure you guys probably have a lot of other stuff going on with go 1.12 release date upcoming)
Change https://golang.org/cl/153977 mentions this issue:
http2: Revert a closed stream cannot receive data