WSL: TCP ephemeral ports exhausted and network broken after lots of early-closed non-blocking connections

At the suggestion of therealkenc, the title and description of the issue have been completely updated.

See more progress in #3951.

Description

After creating and closing (before established) a large number of non-blocking connections in WSL, all TCP ephemeral ports will be exhausted, then no new TCP connections from WSL or Win32 can be established. Closing related processes in WSL does not release these ports. All new TCP connections or listening will failed and must to restart the LxssManager service to recover.

Reproducible Demo

The demo is from philip-searle’s comment of #2913 on 18 Jan.

You can reliably reproduce this issue using the attached program (~80 lines of C): wsl-issue-2913-repro.c.txt

Output from strace looks normal to me and is attached as wsl-issue-2913-repro.strace.zip

The program performs these steps in a loop:

  1. Allocate a TCP socket for IPv4 (AF_INET) family.
  2. Make the socket non-blocking with O_NONBLOCK.
  3. Attempt to connect() the socket to 127.0.0.1:1234
  4. Use getsockname() to obtain the port number and output it to the console.
  5. Close the socket

Environment

In Ubuntu in WSL, build and run the demo with these commands:

apt update
apt install gcc wget
wget -O wsl-issue-2913-repro.c https://github.com/Microsoft/WSL/files/2769821/wsl-issue-2913-repro.c.txt
gcc -o wsl-issue-2913-repro wsl-issue-2913-repro.c
./wsl-issue-2913-repro

Expected Behavior

On a Linux VM I can run the loop several hundred thousand times and see the ports being used cycle through the entire ephemeral range multiple times.

In addition, even if the program has a bug that does not properly release the occupied port, these ports should be automatically released after the program exits.

Observed Behavior

On philip-searle’s Windows laptop it loops about 16,000 times and then EINVAL is returned from connect(). At this point the symptoms described in previous comments appear: Win32 programs such as web browsers fail to connect and the output of “netstat -anoq” in a command prompt shows many connections stuck in the “BOUND” state. The only way to get network connections working again is to restart the LxssManager service.

On YihaoPeng’s PC with Windows 1809 build 17763.379, the ports will be exhausted after 2899 rounds:

...
Socket 2893 is using port 4930
Socket 2894 is using port 4943
Socket 2895 is using port 4944
Socket 2896 is using port 4945
Socket 2897 is using port 4952
Socket 2898 is using port 4953
Invalid argument returned from connect() - BOUND socket exhaustion likely after 2899 rounds
Exiting (failed)

No ports released after the program exits. If you let the program run repeatedly (so it will immediately take up the ephemeral port released by other programs), you will find that no TCP connections in your Windows can be established. For example, your EDGE browser will not be able to load any page.

Use the following commands to run the program repeatedly:

while true; do ./wsl-issue-2913-repro; done

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Reactions: 11
  • Comments: 33 (1 by maintainers)

Most upvoted comments

Fixed in Windows Insider Build 18890

I can reliably reproduce this issue using the attached program (~80 lines of C): wsl-issue-2913-repro.c.txt Output from strace looks normal to me and is attached as wsl-issue-2913-repro.strace.zip

The program performs these steps in a loop:

  1. Allocate a TCP socket for IPv4 (AF_INET) family.
  2. Make the socket non-blocking with O_NONBLOCK.
  3. Attempt to connect() the socket to 127.0.0.1:1234
  4. Use getsockname() to obtain the port number and output it to the console.
  5. Close the socket

On a Linux VM I can run the loop several hundred thousand times and see the ports being used cycle through the entire ephemeral range multiple times.

On my Windows laptop it loops about 16,000 times and then EINVAL is returned from connect(). At this point the symptoms described in previous comments appear: Win32 programs such as web browsers fail to connect and the output of “netstat -anoq” in a command prompt shows many connections stuck in the “BOUND” state. The only way to get network connections working again is to restart the LxssManager service.

Versions used to reproduce this: Output from ver in command prompt: Microsoft Windows [Version 10.0.17134.472] Output from uname -a in WSL: Linux SAG-FNH45S2 4.4.0-17134-Microsoft #471-Microsoft Fri Dec 07 20:04:00 PST 2018 x86_64 GNU/Linux

Since wsl2 will release, it is not neccssary to fix it even.

I have been seeing this issue and it’s easily reproducible with Expo.

  1. Start an Expo build / publish
  2. After some time the networking both on WSL and Windows stop allowing fresh connections, so webpages already open will continue to work and you can establish fresh connections to the same server, but any external assets referenced on other pages that need to load from other URLs will fail to load

To fix it, I do Ctrl-C everything that I have open, killall adb to get rid of the background adb processes, exit out of every tmux window and fully close the Ubuntu WSL terminal. Then networking will work properly again in Windows, without needing to restart the LXSS process as mentioned above.

@sunilmut Can you take a look at this issue again? A simple method of reproduction is given upstairs.

I am still experiencing this issue recently. Although restarting the LxssManager service is effective, it is very inconvenient.

I have been having the same issue using Expo too,

I restart LXSS services so networking work again, but it’s very annoying because all the processes you were running on WSL are killed, so you have to run everything again.

On Sun, Mar 10, 2019 at 6:04 AM Chris Hammond notifications@github.com wrote:

I have been seeing this issue and it’s easily reproducible with Expo https://github.com/expo/expo-cli/.

  1. Start an Expo build / publish
  2. After some time the networking both on WSL and Windows stop allowing fresh connections, so webpages already open will continue to work and you can establish fresh connections to the same server, but any external assets referenced on other pages that need to load from other URLs will fail to load

To fix it, I do Ctrl-C everything that I have open, killall adb to get rid of the background adb processes, exit out of every tmux window and fully close the Ubuntu WSL terminal. Then networking will work properly again in Windows, without needing to restart the LXSS process as mentioned above.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Microsoft/WSL/issues/2913#issuecomment-471277845, or mute the thread https://github.com/notifications/unsubscribe-auth/AKYyWgmnC2Bmy9zdlFw3XnHsfAo-Bvw_ks5vVOa_gaJpZM4R4lz0 .

find my family