aws-lambda-dotnet: AWS Lambda .NET 6 ARM Exceptions

Describe the bug

We have an application that runs on AWS Lambda. Until recently we were running it on .NET Core 3.1 and x86 architecture. We have now ported it to .NET6 on ARM64 and this has brought unexpected problems, as we are randomly getting errors that we believe point to a problem in AWS Lambda runtime for .NET 6. On the original application (.NET Core 3.1 + x86) these errors were not present. Does anyone have any experience with the migration to ARM and .NET6 and if it is more memory intensive or could it be a bug in the Lambda runtime? Any ideas on how to solve the problem will be gratefully received. Thanks.

Exception;

START RequestId: 79052b48-798c-4626-8cbb-7f9240196df6 Version: $LATEST  
Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'state')
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _)
at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
END RequestId: 79052b48-798c-4626-8cbb-7f9240196df6
REPORT RequestId: 79052b48-798c-4626-8cbb-7f9240196df6 Duration: 1051.21 ms Billed Duration: 1052 ms Memory Size: 512 MB Max Memory Used: 368 MB
RequestId: 79052b48-798c-4626-8cbb-7f9240196df6 Error: Runtime exited with error: signal: aborted Runtime.ExitError

Expected Behavior

Lambda works without errors.

Current Behavior

Lambda fails unexpectedly.

START RequestId: 79052b48-798c-4626-8cbb-7f9240196df6 Version: $LATEST  
Unhandled exception. System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'state')
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _)
at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
END RequestId: 79052b48-798c-4626-8cbb-7f9240196df6
REPORT RequestId: 79052b48-798c-4626-8cbb-7f9240196df6 Duration: 1051.21 ms Billed Duration: 1052 ms Memory Size: 512 MB Max Memory Used: 368 MB
RequestId: 79052b48-798c-4626-8cbb-7f9240196df6 Error: Runtime exited with error: signal: aborted Runtime.ExitError

Reproduction Steps

We have no trivial code for that, but we have global TryCatch block in our Lambda handler and it does not catch anything.

public class PingServer : AController
{
    public async Task<APIGatewayProxyResponse> HandleRequest(APIGatewayProxyRequest req, ILambdaContext context)
    {
        try
        {
            using (var scope = CreateScope())
            {
                return await HandleRequest(scope, scope.ServiceProvider.GetService<CoreController.PingServer>(), req, context);
            }
        }
        catch (Exception ex)
        {
            context.Logger.LogCritical($"Error from {nameof(PingServer)}_{nameof(HandleRequest)}\n{ex.ToString()}");
            throw;
        }
    }
}

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

Amazon.Lambda.APIGatewayEvents 2.4.1 Amazon.Lambda.Core 2.1.0 Amazon.Lambda.Serialization.SystemTextJson 2.3.0 AWSXRayRecorder.Handlers.AwsSdk 2.8.3

Targeted .NET Platform

.NET 6

Operating System and version

AWS Lambda

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 6
  • Comments: 21 (4 by maintainers)

Most upvoted comments

We are experiencing the exact same issue. Our try/catch in the handler doesn’t catch anything, and it crashes with the same Runtime error.

It happens randomly, on average once a day I would say.

.NET 6, ARM64 and 1024MB of configured memory.

And we observe the same thing as @kubalunacek 👇🏻

But we get this error once or four times a day. And from our observations, it’s not so much the code as it is the number of calls at the moment. Because the other 10+ Lambdas with the same code base and less frequent calls don’t suffer from anything like this.

We have the exact same issue on arm/.net6 lambdas

Hi @ashishdhingra , what do you need for reproduction? We have out production code deployed on AWS Lambdas. I am unable to provide you the production code. But we get this error once or four times a day. And from our observations, it’s not so much the code as it is the number of calls at the moment. Because the other 10+ Lambdas with the same code base and less frequent calls don’t suffer from anything like this.