go: x/net/http2: retry requests rejected with REFUSED_STREAM

What version of Go are you using (`go version`)?

go1.8.1.typealias

What operating system and processor architecture are you using (`go env`)?

Linux amd64

What did you do?

go run refused-stream.go -bucket veener-jba-doc-test-bucket -object datastore

Where refused-stream.go is https://gist.github.com/jba/9e75c3aedeb4e8b98a323424283fae88.

(veener-jba-doc-test-bucket/datastore should be publicly readable, but I’m guessing any object would do.)

What did you expect to see?

No output.

What did you see instead?

Often (not always), it fails with

2017/07/11 16:09:29 url Error &url.Error{Op:"Get", URL:"https://storage.googleapis.com/veener-jba-doc-test-bucket/datastore", Err:http.http2StreamError{StreamID:0xdd, Code:0x7, Cause:error(nil)}}, temporary=false, timeout=false
2017/07/11 16:09:29 NewReader: Get https://storage.googleapis.com/veener-jba-doc-test-bucket/datastore: stream error: stream ID 221; REFUSED_STREAM (*url.Error)
exit status 1

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 17 (12 by maintainers)

Commits related to this issue

http2: retry requests after receiving REFUSED STREAM RoundTrip will retry a request if it receives REFUSED_STREAM. To guard against servers that use REFUSED_STREAM to encourage rate limiting, or serv... — committed to golang/net by tombergan 7 years ago
http2: block RoundTrip when the Transport hits MaxConcurrentStreams Currently if the http2.Transport hits SettingsMaxConcurrentStreams for a server, it just makes a new TCP connection and creates the... — committed to golang/net by tombergan 7 years ago
http2: retry requests after receiving REFUSED STREAM RoundTrip will retry a request if it receives REFUSED_STREAM. To guard against servers that use REFUSED_STREAM to encourage rate limiting, or serv... — committed to c3mb0/net by tombergan 7 years ago
http2: block RoundTrip when the Transport hits MaxConcurrentStreams Currently if the http2.Transport hits SettingsMaxConcurrentStreams for a server, it just makes a new TCP connection and creates the... — committed to c3mb0/net by tombergan 7 years ago

Most upvoted comments

This isn’t a GFE bug implementing HTTP/2 wrong.

This is GCS (via the GFE) telling us to slow down it seems. But I can’t reproduce. Maybe their rate limiting likes me more.

Oh, if I crank it up 10x I can get the error. With GODEBUG=http2debug=2:

2017/07/12 17:45:55 http2: Transport received HEADERS flags=END_HEADERS stream=249 len=20
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=251 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=251 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=253 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=253 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=255 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=255 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=257 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=257 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=259 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=259 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=261 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=261 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=263 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=263 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=265 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=265 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=267 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=267 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=269 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=269 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=271 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=271 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=273 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=273 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=275 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=275 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=277 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=277 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=279 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=279 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=281 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=281 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=283 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=283 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Framer 0xc424630000: read RST_STREAM stream=285 len=4 ErrCode=REFUSED_STREAM
2017/07/12 17:45:55 http2: Transport received RST_STREAM stream=285 len=4 ErrCode=REFUSED_STREAM

It’s not clear what the GFE wants us to do, though: if it didn’t want us to open 100 concurrent streams, why did it tell us to?

$ h2i storage.googleapis.com
Connecting to storage.googleapis.com:443 ...
Connected to 173.194.197.128:443
Negotiated protocol "h2"
Sending: []
[FrameHeader SETTINGS len=18]
  [MAX_CONCURRENT_STREAMS = 100]
  [INITIAL_WINDOW_SIZE = 1048576]
  [MAX_HEADER_LIST_SIZE = 16384]
[FrameHeader WINDOW_UPDATE len=4]
  Window-Increment = 983041
[FrameHeader SETTINGS flags=ACK len=0]

So, okay, maybe it doesn’t like something else, like the rate (which isn’t expressible as a SETTING), so it sends us a REFUSED_STREAM instead.

What are we supposed to do? Retry in a loop forever? With some exponential backoff I guess?

Yeah, http://httpwg.org/specs/rfc7540.html#Reliability says:

The REFUSED_STREAM error code can be included in a RST_STREAM frame to indicate that the stream is being closed prior to any processing having occurred. Any request that was sent on the reset stream can be safely retried.

So, yeah, do that.

/cc @tombergan

bradfitz on Jul 12, 2017