msquic: Memory leak in msquic 1.0.4

Describe the bug

Since running this new version, I’ve had to restart the server twice due to memory getting to 100%. I’ve looked into it with poolmon and one tag uses ~32GB of RAM:

 Memory:67018228K Avail:  270612K  PageFlts:  9355   InRam Krnl:47756K P:390536K
 Commit:72832896K Limit:84635012K Peak:80802964K            Pool N:37417736K P:635236K
 System pool information
 Tag  Type     Allocs            Frees            Diff       Bytes                  Per Alloc

 Qc1A Nonp   92404489 (   0)      6827 (   0) 92397662 34002339616 (          0)         368

Sorry for bad formatting, I couldn’t get this formatted correctly. Here’s the screenshot:

Since the server worked correctly before without any problems, I suspected it was due to msquic, and looking at the code it seems this tag is indeed used by msquic: https://github.com/microsoft/msquic/blob/main/src/inc/quic_platform.h#L96

I do not have a memory dump. I tried creating one with notmyfault but the server crashed/rebooted immediately without writing any dump, and I have since disabled quic in IIS again.

My setup is still the same, ASP.NET Core 6.0, IIS, WS2022 latest updates. I have also disabled legacy TLS on most sites.

Originally posted here: https://github.com/microsoft/msquic/issues/2249#issuecomment-1107534912

Affected OS

All
Windows Server 2022
Windows 11
Windows Insider Preview (specify affected build below)
Ubuntu
Debian
Other (specify below)

Additional OS information

No response

MsQuic version

release/1.0

Steps taken to reproduce bug

Runs on production server, encountered it twice.

Expected behavior

Shouldn’t leak memory.

Actual outcome

Memory getting filled to 100%.

Additional details

ASP.NET Core 6.0, IIS, WS2022

NIC: Intel® I210 Gigabit. Driver: 12.18.12.1 from 23-2-2021.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 24 (9 by maintainers)

Most upvoted comments

I have enabled QUIC on 20 IIS web servers hosting 4,000 websites and have seen no issues so far.

webwizuk on Mar 20, 2023

Thanks, all is still working well! 🎉

Qc06 is now also consistent, I’ve once seen it at 5.2MB but it quickly freed it up to 4.6MB again, so that indeed seems to be working correctly.

Nuklon on Mar 19, 2023

@nuklon & @webwizuk thanks for trying again and thank you for your patients. We’re actively working on backporting v2.1 MsQuic to WS2022, which should fix all these problems, as well as bring a host of other fixes.

The tentative date is looking to be March for this to ship publically.

nibanks on Jan 24, 2023

This is the exact same behaviour I have been getting with Windows Server 2022 running IIS since it was launched last August 2021 with the “Qc1A Nonp” memory leak that uses up all available memory.

I have over 20 web servers running Windows Server 2022 IIS 10 running as virtual servers on Windows 2019 Hyper-v and they all have this issue.

While I am using Intel NICs, Hyper-V uses a virtual switch for the virtual servers, therefore I don’t think it is a problem with Intel Network Cards, but a problem with QUIC when using it in IIS.

I am constantly rebooting servers on a daily basis due to this memory leak with QUIC, unless I disable QUIC in IIS which fixes the issue.

webwizuk on May 9, 2022