aws-lambda-dotnet: .Net7 Minimal API running under provider.al2 runtime timeout on quick fire successive requests
Describe the bug
Related to discussion https://github.com/aws/aws-lambda-dotnet/discussions/1424, but figure an Issue might garner more attention, since prior response there were stale and now have a reproducible sample.
In summary we have a .Net 7 Minimal API setup (serverless.AspNetCoreMinimalAPI) switched to .Net7 and using the amazon.lambda.tools to build the package using the docker image process via the -ucfb True option.
Our sense is that in .Net7 the Host/Middleware, or something related to its interaction with the provider.al2 runtime that may have changed with the newer .Net version is now possibly incompatible the .AddAWSLambdaHosting middleware.
Expected Behavior
Successive and continuous request should complete successfully
Current Behavior
Once deployed we find:
- that the default route
/can be continuously quick-fired without error. - that other requests like
/calculator/add/1/2initially succeeds but then fail after ~3-5 rapid requests with a timeout, as set by the lambda deployment (30 secs in our case). - If the invocations are spaced out ( at ~30 second intervals) the successive requests continue to work successfully.
Reproduction Steps
We have a GitHub workflow enabled clone-able/deployable repo here with instructions here
We have used the default template samples so that we can eliminate if any of our code is introducing a side effect, and whist we had added some of that code it’s commented out.
Possible Solution
We need to migrate to .Net7 such that we can use latest C# features.
Additional Information/Context
We note comments in the RuntimeSupport.Client.InvocationResponse there is special handling of the OutputStream for successive invocations, but are unsure of the implication in this scenario, and assume this in an optimization.
Of note:
- The
/(default) route request in the Map does not leverage any default middleware where as the other routes to map the request on the controller do. - Whist
Net7.0is not explicitly supported for minimal API, given thelambda.NativeAOTtemplate, we assume basic compatibility with theprovider.al2custom runtime - We have set
<PublishAot>false</PublishAot>in thecsproj, to eliminate any concern that this my be an issue.
AWS .NET SDK and/or Package version used
Amazon.Lambda.AspNetCoreServer.Hosting - 1.6.0 AWSSDK.Extensions.NETCore.Setup - 3.7.7 Amazon.Lambda.RuntimeSupport - 1.8.8 Amazon.Lambda.Core" Version - 2.1.0 Amazon.Lambda.Serialization.SystemTextJson - 2.3.1
Targeted .NET Platform
.Net 7
Operating System and version
AmazonLinux
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 16 (4 by maintainers)
@normj & @ashishdhingra
It’s unfortunate that you and team have not be able to recreate this issue, that we are aware. We do though have more information.
Our hypothesis is that there is something wrong with the raw AL2 image, that only manifest when using the underpinning of the
AddAWSLambdaHostingthat regular raw or Annotations based Lambdas do not.We get to this because we can build a self contained executable in both .Net 7 (and .Net 8 preview 6 - just to see) and deploy them to a .Net 6 runtime lambda, and they both work as expected, with no timeouts - on basis that the executable is called
bootstrap, and more importantly the basis that the self-contained this runtime is overriding the runtime in the Dotnet 6 image.This means we have a way forwards, and can avoid the AL2 runtime entirely. It had not occurred to us that one could do this before.
However if we switch the Lambda runtime to the AL2 runtime, using the same packages in turn it runs as expected but we get exactly the same failure observations as we did before.
Of note, if we build a self-contained .Net 6 package as above and deploy that to AL2 runtime, it also fails as observed.
Our assumptions is that when .Net 8 goes LTS, the existing .Net lambda runtime will be updated to support it, and what we have identified will likely impact few, if anyone.
However as MSFT support AOT builds of the .Net runtime, that say underpins Minimal API, this may surface for those trying to run that in the current AL2 image. Perhaps well event see Amazon Linux 2023 but then - right?
Well also probably be fine in the future with running Minimal API AOT images (when supported) in the current dotnet runtime too.
Is there anything y’all know about that we may be missing in running any self-contained or AOT’d packages in the dotnet runtime that might present a future problem here?
I’ll throw together a sample project tonight