go: net/http: http/2 throughput is very slow compared to http/1.1

You can pass in the http.Server to modify MaxReadFrameSize:

	http2.ConfigureServer(<your http.Server obj>, &http2.Server{
		MaxReadFrameSize:             256000, // 256K read frame, https://github.com/golang/go/issues/47840
	})

Yes. You need to provide similar options to the curl command to increase the buffer size.

I’ve been watching this issue as a maintainer of Caddy.

I’m seeing from the proposal https://github.com/golang/go/issues/54850 and the merged changes that changes are only being made to the HTTP/2 transport (i.e. client).

Apparently there already exists a MaxReadFrameSize option in the HTTP/2 server, but we’re using the http.Server auto-configure right now; I don’t see any way to set MaxReadFrameSize via http.Server, unless I’m missing something obvious.

Is the plan to adjust the automatic behaviour so that it’s tuned better by default for the server as well? If not, how are we meant to tune this knob? I’d rather not have to stop using http.Server’s auto-configure so that we can fix this performance issue.

per minor release policy this should only be in for 1.20

any update to this ? is someone at Google even working on this or is there no point waiting ?

As I pointed out above, increasing the frame size can achieve 38 Gbps. The issue is that constant is used for all connections. The ‘max frame size’ is connection dependent.

yeah, I’ve should explained myself better, sorry, in addition to increase the frame size that require manual configuration, it is an user decision to maximize throughput, I just wanted to add that maybe we can improve the default throughput with that change, since it is clear that the author left open that parameter to debate and it is a 2x win (at a cost of increasing memory cost of course, but this is inside a sync.Pool that may alleviate this problem a bit)

More importantly, that constant does not exist in golang.org/x/net/http - which is the basis of the future version.

it does, just with a different name 😄

https://github.com/golang/net/blob/0fccb6fa2b5ce302a9da5afc2513d351bd175889/http2/http2.go#L256-L259

IIUIC the http2 code in golang/go is a bundle created from the x/net/http2

I don’t know if you are referring to this buffer in one of your comments, but using the profiler it shows a lot of contention on the bufWriterPool

https://github.com/golang/go/blob/0e1d553b4d98b71799b86b0ba9bc338de29b7dfe/src/net/http/h2_bundle.go#L3465-L3468

Increasing the value there to the maxFrameSize value, and using the shared code in the description, you can go from 6Gbps to 12.6 Gbps

Copying my comment from https://groups.google.com/d/msgid/golang-nuts/89926c2f-ec73-43ad-be49-a8bc76a18345n%40googlegroups.com

Http2 is a multiplexed protocol with independent streams. The Go implementation uses a common reader thread/routine to read all of the connection content, and then demuxes the streams and passes the data via pipes to the stream readers. This multithreaded nature requires the use of locks to coordinate. By managing the window size, the connection reader should never block writing to a steam buffer - but a stream reader may stall waiting for data to arrive - get descheduled - only to be quickly rescheduled when reader places more data in the buffer - which is inefficient.

Out of the box on my machine, http1 is about 37 Gbps, and http2 is about 7 Gbps on my system.

Some things that jump out:

1. The chunk size is too small. Using 1MB pushed http1 from 37 Gbs to 50 Gbps, and http2 to 8 Gbps.

2. The default buffer in io.Copy() is too small. Use io.CopyBuffer() with a larger buffer - I changed to 4MB. This pushed http1 to 55 Gbs, and http2 to 8.2. Not a big difference but needed for later.

3. The http2 receiver frame size of 16k is way too small. There is overhead on every frame - the most costly is updating the window.

I made some local mods to the net library, increasing the frame size to 256k, and the http2 performance went from 8Gbps to 38Gbps.

1. I haven’t tracked it down yet, but I don’t think the window size update code is not working as intended - it seems to be sending window updates (which are expensive due to locks) far too frequently. I think this is the area that could use the most improvement - using some heuristics there is the possibility to detect the sender rate, and adjust the refresh rate (using high/low water marks).

2. The implementation might need improvements using lock-free structures, atomic counters, and busy-waits in order to achieve maximum performance.

So 38Gbps for http2 vs 55 Gbps for http1. Better but still not great. Still, with some minor changes, the net package could allow setting of a large frame size on a per stream basis - which would enable much higher throughput. The gRPC library allows this.