px: [WinError10053] An established connection was aborted by the software in your host machine
Hi genotrace, I’m using px0.7.0 and getting the following (unfortunately sporadic) error.
… sry but just did a screenshot which I couldn’t updload so I rewrite the traceback manually.
... px/main.py line 289, handle_one_request
... px/server.py line 413, handle_one_request
... px/main.py line 408, do_CONNECT
... px/main.py line 359, do_curl
... px/mcurl.py line 699, select
[WinError10053] An established connection was aborted by the software in your host machine
Do you’ve got any idea for a root cause?
Thanks in advance!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (9 by maintainers)
v0.9.0 has been released.
I spent some time debugging this. Turns out this is not really an error - it simply says that either the upstream server or the requesting client closed the connection and Px simply couldn’t talk to that socket anymore. Since this is an HTTPS connection, either side can close the socket if there’s nothing more to do or time’s up. They could even inform each other with the “Connection: close” HTTP header but Px has no clue since it is all encrypted. So the log message is not really an error but a side effect.
The real issue though is that we are seeing poor performance in the client - IntelliJ or VSCode and Firefox in my case. However, this is really a configuration issue. Firefox for example has a maximum connection set to 900 which means it can open 900 parallel connections before having to reuse anything. And considering almost everything is HTTPS these days, every connection is effectively left open and reused until either the client or server decide to close it or Px idle timeout is reached.
The default number of parallel connections for Px is 2 workers x 5 threads = 10 connections. Any GUI app that makes multiple parallel connections to populate search results, web pages, images and what not will quickly tie up all those connections with HTTPS and get stuck since there’s no more left. And until Px idle timeout of 30 seconds hits, it will just sit and wait there. Page loads will fail and the app will perform poorly.
A simple solution is to up the # of threads. I tested Firefox with Px @ 2 workers x 512 threads and it worked without any issues. You still see some interesting log messages but that’s just business as usual. Considering the minimal memory impact to having more threads, the default is being raised from 5 to 32. This will give us 64 parallel connections for starters which can always be increased/decreased depending on the use case.
As part of the debug, I did fix a couple of other minor issues but the above recommendation should be the real solution for this issue.
The debugging “solution” is unfortunately not a long-term solution. I had to turn it off for NPM, because Px wasn’t able to handle the mountain of connections in Debug Mode.