runtime: [linux-arm64] System.ArgumentOutOfRangeException in AwaitableSocketAsyncEventArgs.InvokeContinuation

Description

We’ve seen a crash in our CI caused by an ArgumentOutOfRangeException thrown by System.Threading.ThreadPool+<>c.<.cctor>b__86_0(System.Object), itself invoked by System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation.

The full callstack:

System.Private.CoreLib.dll!System.ThrowHelper.ThrowArgumentOutOfRangeException(System.ExceptionArgument)+0x44
System.Private.CoreLib.dll!System.Threading.ThreadPool+<>c.<.cctor>b__86_0(System.Object)+0x48
System.Net.Sockets.dll!System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean)+0x5c
System.Net.Sockets.dll!System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.OnCompleted(System.Net.Sockets.SocketAsyncEventArgs)+0xa8
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncEventArgs.OnCompletedInternal()+0x58
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32, System.Net.Sockets.SocketFlags)+0x48
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError)+0x24
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32, Byte[], Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError)+0x3c
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncContext+OperationQueue`1[[System.__Canon, System.Private.CoreLib]].ProcessAsyncOperation(System.__Canon)+0x134
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncContext+ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()+0x40
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents)+0x98
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()+0xd0
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()+0x324
System.Private.CoreLib.dll!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()+0x164
System.Private.CoreLib.dll!System.Threading.Thread.StartCallback()+0x68

I believe this is the same issue as https://github.com/dotnet/runtime/issues/70486 and https://github.com/dotnet/runtime/issues/72365.

It’s entirely possible that the crash is caused by our instrumentation code (we use the profiler API to rewrite IL at runtime, though we don’t rewrite any of the methods in this callstack). However, since the exact same issue has been reported by somebody else, and because in both cases it happened only on ARM64, it makes me think there is actually an issue in the runtime or the BCL.

We managed to capture a memory dump. I can share it if you have a place to upload it (47MB zipped).

Reproduction Steps

The issue occurs very rarely in our CI so I don’t have a repro. This is a test application that sends HTTP requests to itself (HttpClient + HttpListener).

Expected behavior

No crashes.

Actual behavior

An ArgumentOutOfRangeException is thrown and crashes the process.

Regression?

We observed the issue on .NET 6. I don’t know if previous versions are impacted.

Known Workarounds

No response

Configuration

.NET 6.0.15 Linux Debian/ARM64. The crash was never observed on x86/x64.

Other information

No response

About this issue

Original URL
State: closed
Created a year ago
Reactions: 2
Comments: 17 (10 by maintainers)

Most upvoted comments

Fixed in 7.0.7 in PR #84641 and in 6.0.18 in PR #84432. Main (8.0) is not affected - see description in #84432.

karelz on May 27, 2023

hey @stephentoub, I was reading PR #84432, and the fix was to use the volatile keyword. I’m a dotnet/C# newbie person, and I was curious to understand the keyword impact and I found some a lot people discussing about it.

Is this fix a solution for multi-processor use cases or just when both threads run on the same processor?

Compilers and hardware are free to reorder reads/writes in ways that are unobservable for the current thread. But such reorderings can be visible to other threads running concurrently. Marking a variable as volatile tells the system to use read-acquire / store-release semantics, preventing those reorderings.

The .NET memory model is described at https://github.com/dotnet/runtime/blob/0c77cbef02e1713d2b7cd01969b92e552acbdb6b/docs/design/specs/Memory-model.md#L4.

The volatile keyword is described at https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/volatile.

stephentoub on Jun 20, 2023

I checked for the past month and found only two other occurrences of the problem (out of ~34400 runs in total).

Unfortunately I don’t think we can deploy .NET 8 just yet (we first need to update our instrumentation code to support it).

kevingosse on Apr 6, 2023