WSL: TCP ephemeral ports exhausted and network broken after lots of early-closed non-blocking connections
At the suggestion of therealkenc, the title and description of the issue have been completely updated.
See more progress in #3951.
Description
After creating and closing (before established) a large number of non-blocking connections in WSL, all TCP ephemeral ports will be exhausted, then no new TCP connections from WSL or Win32 can be established. Closing related processes in WSL does not release these ports. All new TCP connections or listening will failed and must to restart the LxssManager service to recover.
Reproducible Demo
The demo is from philip-searle’s comment of #2913 on 18 Jan.
You can reliably reproduce this issue using the attached program (~80 lines of C): wsl-issue-2913-repro.c.txt
Output from strace looks normal to me and is attached as wsl-issue-2913-repro.strace.zip
The program performs these steps in a loop:
- Allocate a TCP socket for IPv4 (AF_INET) family.
- Make the socket non-blocking with O_NONBLOCK.
- Attempt to connect() the socket to 127.0.0.1:1234
- Use getsockname() to obtain the port number and output it to the console.
- Close the socket
Environment
In Ubuntu in WSL, build and run the demo with these commands:
apt update
apt install gcc wget
wget -O wsl-issue-2913-repro.c https://github.com/Microsoft/WSL/files/2769821/wsl-issue-2913-repro.c.txt
gcc -o wsl-issue-2913-repro wsl-issue-2913-repro.c
./wsl-issue-2913-repro
Expected Behavior
On a Linux VM I can run the loop several hundred thousand times and see the ports being used cycle through the entire ephemeral range multiple times.
In addition, even if the program has a bug that does not properly release the occupied port, these ports should be automatically released after the program exits.
Observed Behavior
On philip-searle’s Windows laptop it loops about 16,000 times and then EINVAL is returned from connect(). At this point the symptoms described in previous comments appear: Win32 programs such as web browsers fail to connect and the output of “netstat -anoq” in a command prompt shows many connections stuck in the “BOUND” state. The only way to get network connections working again is to restart the LxssManager service.
On YihaoPeng’s PC with Windows 1809 build 17763.379, the ports will be exhausted after 2899 rounds:
...
Socket 2893 is using port 4930
Socket 2894 is using port 4943
Socket 2895 is using port 4944
Socket 2896 is using port 4945
Socket 2897 is using port 4952
Socket 2898 is using port 4953
Invalid argument returned from connect() - BOUND socket exhaustion likely after 2899 rounds
Exiting (failed)
No ports released after the program exits. If you let the program run repeatedly (so it will immediately take up the ephemeral port released by other programs), you will find that no TCP connections in your Windows can be established. For example, your EDGE browser will not be able to load any page.
Use the following commands to run the program repeatedly:
while true; do ./wsl-issue-2913-repro; done
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 11
- Comments: 33 (1 by maintainers)
Fixed in Windows Insider Build 18890
I can reliably reproduce this issue using the attached program (~80 lines of C): wsl-issue-2913-repro.c.txt Output from strace looks normal to me and is attached as wsl-issue-2913-repro.strace.zip
The program performs these steps in a loop:
On a Linux VM I can run the loop several hundred thousand times and see the ports being used cycle through the entire ephemeral range multiple times.
On my Windows laptop it loops about 16,000 times and then EINVAL is returned from connect(). At this point the symptoms described in previous comments appear: Win32 programs such as web browsers fail to connect and the output of “netstat -anoq” in a command prompt shows many connections stuck in the “BOUND” state. The only way to get network connections working again is to restart the LxssManager service.
Versions used to reproduce this: Output from
verin command prompt:Microsoft Windows [Version 10.0.17134.472]Output fromuname -ain WSL:Linux SAG-FNH45S2 4.4.0-17134-Microsoft #471-Microsoft Fri Dec 07 20:04:00 PST 2018 x86_64 GNU/LinuxSince wsl2 will release, it is not neccssary to fix it even.
I have been seeing this issue and it’s easily reproducible with Expo.
To fix it, I do Ctrl-C everything that I have open,
killall adbto get rid of the backgroundadbprocesses, exit out of every tmux window and fully close the Ubuntu WSL terminal. Then networking will work properly again in Windows, without needing to restart the LXSS process as mentioned above.@sunilmut Can you take a look at this issue again? A simple method of reproduction is given upstairs.
I am still experiencing this issue recently. Although restarting the LxssManager service is effective, it is very inconvenient.
I have been having the same issue using Expo too,
I restart LXSS services so networking work again, but it’s very annoying because all the processes you were running on WSL are killed, so you have to run everything again.
On Sun, Mar 10, 2019 at 6:04 AM Chris Hammond notifications@github.com wrote:
find my family