moby: docker-proxy timeouts under heavy load

I have been able to see many times the error:

Timed out proxy starting the userland proxy

This was happening on my stress test node under heavy load; not sure what other bottlenecks can cascade-cause this specific timeout, however I found that at https://github.com/docker/docker/blob/master/daemon/networkdriver/portmapper/proxy.go#L148 there is a hardcoded 1 second timeout.

I have increased it to 5 seconds, without looking too much into how this works, and it actually relieved this type of failures.

Obviously it can’t be a magic value; any idea/strategy on this?

The only option I have thought of is to allow infinite waiting, then have some tool like JeremyJones/Apachetop that can show the status of various docker-proxy’s.

This would help eventual troubleshooting as load would not decrease (no timeout failure) when problem is manifesting.