runtime: Dotnet 6.0 Websocket Deflate Segfault On Ubuntu
Description
I run a service that uses a lot of WebSockets and was very excited to enable the new WebSocket deflation feature in dotnet 6.0. I enabled it and was very happy with the performance gain from the compression of the large json based messages I have to send.
However, after deploying my new dotnet 6.0 build I started getting about 1 server segfault crash a day across all of my servers. I set up lldb, attached to the process, and was able to get a call stack of the segfault. The call stack indicates there was a segfaul in the WebSocket stack while it was trying to compress an outgoing message.
I don’t know what else would be helpful for debugging, I’m happy to attach lldb to a server process again and capture more information if needed.
(lldb) continue
Process 830510 resuming
Process 830510 stopped
* thread #51, name = '.NET ThreadPool', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
frame #0: 0x00007f01d1f4cacf libz.so.1`___lldb_unnamed_symbol10$$libz.so.1 + 1535
libz.so.1`___lldb_unnamed_symbol10$$libz.so.1:
-> 0x7f01d1f4cacf <+1535>: movq 0x38(%rbp), %r12
0x7f01d1f4cad3 <+1539>: movq %rax, 0x98(%rbx)
0x7f01d1f4cada <+1546>: movq %r12, %rdi
0x7f01d1f4cadd <+1549>: callq 0x7f01d1f57470 ; ___lldb_unnamed_symbol23$$libz.so.1
(lldb) clrstack
OS Thread Id: 0xcd8e7 (51)
Child SP IP Call Site
00007F01D37F9D10 00007f01d1f4cacf [InlinedCallFrame: 00007f01d37f9d10] Interop+zlib.Deflate(ZStream*, FlushCode)
00007F01D37F9D10 00007f04323e7f17 [InlinedCallFrame: 00007f01d37f9d10] Interop+zlib.Deflate(ZStream*, FlushCode)
00007F01D37F9CE0 00007F04323E7F17 System.Net.WebSockets.Compression.WebSocketDeflater.Deflate(ZLibStreamHandle, FlushCode) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/Compression/WebSocketDeflater.cs @ 191]
00007F01D37F9D90 00007F04323FEEA7 System.Net.WebSockets.Compression.WebSocketDeflater.UnsafeFlush(System.Span`1<Byte>, Boolean ByRef) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/Compression/WebSocketDeflater.cs @ 162]
00007F01D37F9DC0 00007F04323FECE5 System.Net.WebSockets.Compression.WebSocketDeflater.DeflatePrivate(System.ReadOnlySpan`1<Byte>, System.Span`1<Byte>, Boolean, Int32 ByRef, Int32 ByRef, Boolean ByRef) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/Compression/WebSocketDeflater.cs @ 104]
00007F01D37F9E10 00007F04323FEA76 System.Net.WebSockets.Compression.WebSocketDeflater.Deflate(System.ReadOnlySpan`1<Byte>, Boolean)
00007F01D37F9E90 00007F0432026F33 System.Net.WebSockets.ManagedWebSocket.WriteFrameToSendBuffer(MessageOpcode, Boolean, Boolean, System.ReadOnlySpan`1<Byte>) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/ManagedWebSocket.cs @ 501]
00007F01D37F9EE0 00007F043202674C System.Net.WebSockets.ManagedWebSocket+<SendFrameFallbackAsync>d__58.MoveNext() [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/ManagedWebSocket.cs @ 477]
00007F01D37F9FF0 00007F0432026438 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.WebSockets.ManagedWebSocket+<SendFrameFallbackAsync>d__58, System.Net.WebSockets]](<SendFrameFallbackAsync>d__58 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
00007F01D37FA040 00007F0432026363 System.Net.WebSockets.ManagedWebSocket.SendFrameFallbackAsync(MessageOpcode, Boolean, Boolean, System.ReadOnlyMemory`1<Byte>, System.Threading.Tasks.Task, System.Threading.CancellationToken)
00007F01D37FA0F0 00007F0432046CB4 System.Net.WebSockets.ManagedWebSocket.SendAsync(System.ReadOnlyMemory`1<Byte>, System.Net.WebSockets.WebSocketMessageType, System.Net.WebSockets.WebSocketMessageFlags, System.Threading.CancellationToken) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/ManagedWebSocket.cs @ 288]
00007F01D37FA1B0 00007F0432046B10 System.Net.WebSockets.ManagedWebSocket.SendAsync(System.ReadOnlyMemory`1<Byte>, System.Net.WebSockets.WebSocketMessageType, Boolean, System.Threading.CancellationToken) [/_/src/libraries/System.Net.WebSockets/src/System/Net/WebSockets/ManagedWebSocket.cs @ 250]
00007F01D37FA1F0 00007F0432046617 OctoStream.BetterWebSocket+<Send>d__15.MoveNext() [D:\a\1\s\OctoStream\Util\BetterWebSocket.cs @ 78]
00007F01D37FA2A0 00007F0432046368 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[OctoStream.BetterWebSocket+<Send>d__15, OctoStream]](<Send>d__15 ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs @ 38]
00007F01D37FA2F0 00007F04320462B5 OctoStream.BetterWebSocket.Send(System.ReadOnlyMemory`1<Byte>, System.Net.WebSockets.WebSocketMessageType)
Reproduction Steps
Setup a server using WebSocket and WebSocket deflation. Open and close WebSockets, send and receive messages on the WebSockets frequently.
Expected behavior
The server should not segfault.
Actual behavior
After some time of WebSockets opening, closing, sending and receiving messages, my server segfaults. My servers handles about 200-500 concurrent WebSockets that come and go at any given time. It probably processes ~50-100 messages a second across all of the WebSockets connected to the server.
Regression?
No, deflation is a new feature to dotnet 6.0.
Known Workarounds
Disable deflation for WebSockets
Configuration
No response
Other information
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 24 (13 by maintainers)
Commits related to this issue
- Fixing a possible null reference error in WebSocket deflate. (#62428) Fixes #62422 — committed to dotnet/runtime by zlatanov 3 years ago
You mean .NET 6 not 5? I’m working on it, it’ll be today. @QuinnDamerell which way do you want me to send you the binary?
@zlatanov still looking great! No crashes on any of my servers thus far!
@ManickaP I love it! Honestly, when I first wrote the system I was quite naive to the space of reverse HTTP proxies. But my system is a little unique because I use a WebSocket connection from the device in the user’s home to my service as a tunnel for the HTTP requests. I do that because it makes the setup super easy for the user with no networking config changes needed, and it’s secure. So in the end I think it worked out to build a custom solution because I can tailor the entire thing just to OctoPrint and its unique needs. But I’m interested to look through the project you linked to see what it’s all about!
A quick update on the servers, no crashes! From my usage data, I see that the WebSocket usage is just about the same as it was when I got the crashes, so I think this is evidence that the change fixed the issue. I think we should wait a little longer before declaring it a success, but it’s looking great!
Quick update, no problems thus far on the one medium traffic server I’m running the test DLL on. I just now rolled out the test DLL to the rest of my servers, including the high traffic servers where I saw the most crashes.
Looking good so far, I will keep you updated!
@ManickaP email received!
I have the dll in place. I don’t have any other way to test it beyond letting it run for a little while and see if it crashes. In the past, I saw about one crash a day across my servers. I will start it on one server today to make sure it all works as expected and then roll it out more tomorrow if it all looks good.
I will keep this tread updated if I see any more crashes!
@QuinnDamerell it should make 6.0.2 which should be out ~Feb 8th.
Shameless plug for one of our projects, but have you heard about https://github.com/microsoft/reverse-proxy 😉