go: net/http: read: connection reset by peer under high load

What version of Go are you using (`go version`)?

go1.8.3

What operating system and processor architecture are you using (`go env`)?

GOARCH=“amd64” GOOS=“linux”

What did you do?

Edit /etc/sysctl.conf and reload with sysctl -p.
Edit /etc/security/limits.conf.
Execute the following server and client (on different hosts) to simulate multiple HTTP requests: Server https://play.golang.org/p/B3UCQ_4mWm Client https://play.golang.org/p/_XQRqp8K5b

What did you expect to see?

No errors at all.

What did you see instead?

Errors like the following:

Get http://<addr>:8080: read tcp [::1]:65244->[::1]:8080: read: connection reset by peer
Get http://<addr>:8080: write tcp [::1]:62575->[::1]:8080: write: broken pipe
Get http://<addr>:8080: dial tcp [::1]:8080: getsockopt: connection reset by peer
Get http://<addr>:8080: readLoopPeekFailLocked: read tcp [::1]:51466->[::1]:8080: read: connection reset by peer

About this issue

Original URL
State: open
Created 7 years ago
Reactions: 65
Comments: 19 (2 by maintainers)

Commits related to this issue

Testing on macOS requires kernel sysctl This commit prints out an error with a breadcrumb on macOS to tweak a kernel setting (`kern.ipc.somaxconn`) to allow the test to pass. Fixes: ``` [FAILED] U... — committed to cloudfoundry/bosh-dns-release by cunnie a year ago

Most upvoted comments

Just to update. my issue was related to OSX and its defaults… having tweakes the following settings I can now push 1000 concurrent transactions sudo ulimit -n 6049 sudo sysctl -w kern.ipc.somaxconn=1024

+18

REPTILEHAUS on Feb 21, 2019

Also hitting this issue. Im running 1000 concurrent connections at it eventually hits this also on OSX with file descriptors bumped up from default (256) to 4096. See a lot of stuff about this, possibly only an issue on OSX

REPTILEHAUS on Feb 7, 2019

I also see this issue under macOS where server and client talk over localhost. I don’t know the exact cause of the problem but running netstat during high load displays very large number of connections in TIME_WAIT state. Either I exhaust file descriptors or local ports.

Or both. I keep seeing connection reset by peer until the number of TIME_WAIT connections is about 7k. If I let the test run even further, I eventually get connect: can't assign requested address error which suggests that I exhausted local ports. The number of TIME_WAIT connections is at 15k.

For my project that’s not needed and I could solve the problem by limiting number of concurrent connections. Solving the problem in my code by limiting number of goroutines seemed arbitrary and still occasionally failed. Limiting MaxConnsPerHost in http transport did the trick and problem went away completely.

creker on Dec 17, 2018

Just to update. my issue was related to OSX and its defaults… having tweakes the following settings I can now push 1000 concurrent transactions sudo ulimit -n 6049 sudo sysctl -w kern.ipc.somaxconn=1024

Works for me on Mac OS @REPTILEHAUS .

iwind on Aug 31, 2019

@gabrielcalderon it doesn’t fix the problem completely. Idle connections do not limit the number of connections Go could open. If you exhausted idle connections for any reason, Go will start opening new ones and cause TIME_WAIT problem again. What fixes the problem is MaxConnsPerHost knob. It limits the number of connections Go could open and blocks subsequent requests if it’s exhausted. You want to put it equal or higher than idle connections.

creker on Feb 10, 2019