aws-lambda-dotnet: .Net7 Minimal API running under provider.al2 runtime timeout on quick fire successive requests

Describe the bug

Related to discussion https://github.com/aws/aws-lambda-dotnet/discussions/1424, but figure an Issue might garner more attention, since prior response there were stale and now have a reproducible sample.

In summary we have a .Net 7 Minimal API setup (serverless.AspNetCoreMinimalAPI) switched to .Net7 and using the amazon.lambda.tools to build the package using the docker image process via the -ucfb True option.

Our sense is that in .Net7 the Host/Middleware, or something related to its interaction with the provider.al2 runtime that may have changed with the newer .Net version is now possibly incompatible the .AddAWSLambdaHosting middleware.

Expected Behavior

Successive and continuous request should complete successfully

Current Behavior

Once deployed we find:

  • that the default route / can be continuously quick-fired without error.
  • that other requests like /calculator/add/1/2 initially succeeds but then fail after ~3-5 rapid requests with a timeout, as set by the lambda deployment (30 secs in our case).
  • If the invocations are spaced out ( at ~30 second intervals) the successive requests continue to work successfully.

Reproduction Steps

We have a GitHub workflow enabled clone-able/deployable repo here with instructions here

We have used the default template samples so that we can eliminate if any of our code is introducing a side effect, and whist we had added some of that code it’s commented out.

Possible Solution

We need to migrate to .Net7 such that we can use latest C# features.

Additional Information/Context

We note comments in the RuntimeSupport.Client.InvocationResponse there is special handling of the OutputStream for successive invocations, but are unsure of the implication in this scenario, and assume this in an optimization.

Of note:

  • The / (default) route request in the Map does not leverage any default middleware where as the other routes to map the request on the controller do.
  • Whist Net7.0 is not explicitly supported for minimal API, given the lambda.NativeAOT template, we assume basic compatibility with the provider.al2 custom runtime
  • We have set <PublishAot>false</PublishAot> in the csproj, to eliminate any concern that this my be an issue.

AWS .NET SDK and/or Package version used

Amazon.Lambda.AspNetCoreServer.Hosting - 1.6.0 AWSSDK.Extensions.NETCore.Setup - 3.7.7 Amazon.Lambda.RuntimeSupport - 1.8.8 Amazon.Lambda.Core" Version - 2.1.0 Amazon.Lambda.Serialization.SystemTextJson - 2.3.1

Targeted .NET Platform

.Net 7

Operating System and version

AmazonLinux

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 16 (4 by maintainers)

Most upvoted comments

@normj & @ashishdhingra

It’s unfortunate that you and team have not be able to recreate this issue, that we are aware. We do though have more information.

Our hypothesis is that there is something wrong with the raw AL2 image, that only manifest when using the underpinning of the AddAWSLambdaHosting that regular raw or Annotations based Lambdas do not.

We get to this because we can build a self contained executable in both .Net 7 (and .Net 8 preview 6 - just to see) and deploy them to a .Net 6 runtime lambda, and they both work as expected, with no timeouts - on basis that the executable is called bootstrap, and more importantly the basis that the self-contained this runtime is overriding the runtime in the Dotnet 6 image.

This means we have a way forwards, and can avoid the AL2 runtime entirely. It had not occurred to us that one could do this before.

However if we switch the Lambda runtime to the AL2 runtime, using the same packages in turn it runs as expected but we get exactly the same failure observations as we did before.

Of note, if we build a self-contained .Net 6 package as above and deploy that to AL2 runtime, it also fails as observed.

Our assumptions is that when .Net 8 goes LTS, the existing .Net lambda runtime will be updated to support it, and what we have identified will likely impact few, if anyone.

However as MSFT support AOT builds of the .Net runtime, that say underpins Minimal API, this may surface for those trying to run that in the current AL2 image. Perhaps well event see Amazon Linux 2023 but then - right?

Well also probably be fine in the future with running Minimal API AOT images (when supported) in the current dotnet runtime too.

Is there anything y’all know about that we may be missing in running any self-contained or AOT’d packages in the dotnet runtime that might present a future problem here?

@dguisinger Is there a sample project you can share that demonstrates this behavior?

I’ll throw together a sample project tonight