runtime: .NET AWS Lambda function using .NET 7 RC1 fails to start when using a custom runtime on AWS Linux 2 on arm64
Description
This issue is a duplicate of https://github.com/aws/aws-lambda-dotnet/issues/1310 that I’m also opening for visibility in case some collaboration is needed between Microsoft and AWS to resolve this issue to enable .NET 7 on AWS Lambda.
I attempted to deploy a .NET Lambda function I have updated from .NET 6 to .NET 7 to AWS Lambda using a custom runtime (deployment).
However, once deployed the tests for the Lambda function showed it was failing with an error. Digging into the logs in AWS CloudWatch show that the function is failing to start.
Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)
This appears to be due to the AWS Lambda runtime not having the correct version of GLIBC (2.27) for .NET 7 to run, likely due to a change in the native requirements for .NET itself.
I’m not currently aware of a way to get a .NET application to publish a specific version of GLIBC with the compiled application - if there is one, I’m all ears to unblock early adoption.
/cc @normj
Reproduction Steps
An AWS Lambda function derived from martincostello/alexa-london-travel at commit bfc8db1850aa3b6463ffd23fc96b76a730222e61 when deployed to AWS Lambda using the provided.al2 runtime will fail to initialize.
Expected behavior
The .NET Lambda function starts successfully.
Actual behavior
The Lambda function fails to start with the following error:
Failed to load /var/task/libcoreclr.so, error: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /var/task/libcoreclr.so)
Regression?
To the end-user, yes, but the change in the GLIBC version is an intentional change.
Known Workarounds
None.
Configuration
- .NET SDK:
7.0.100-rc.1.22431.12 - Operating system: Amazon Linux 2 on Graviton/arm64 (
Linux 4.14.255-276-224.499.amzn2.aarch64 #1 SMP Tue May 3 22:29:59 UTC 2022)
Other information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 21 (13 by maintainers)
We’ve been discussing this more and want to start on a solution to this problem for .NET 8. We can investigate if there is an option to backport to .NET 7, however we’ll have to satisfy a bunch of risk concerns for that so no promises there.
The basic problem is that we want to use modern distros for our build environment, while also enabling breadth targeting. The libc version ends up being the primary challenge in doing that, as has been described in this issue. We’ve already been thinking about that, but didn’t fund the solution for .NET 7. We also ran into this same problem with https://github.com/dotnet/runtime/pull/70111#issuecomment-1152613345. We were planning to tackle it for .NET 8 but now have extra motivation because of this scenario. We believe that rootfs targeting is the best direction, which we already use in some scenarios. It would be great to get your viewpoint on that.
For .NET 7, we’re seeing this challenge with Arm64. For .NET 8, we won’t be able to rely on our CentOS 7-based x64 build any longer (EOL June 2024) and so will have this same problem with X64.
We’ve previously discussed minimum libc targeting with Canonical and Red Hat. Our goal is to target the lowest version that is needed. That kind of coordination requires significant ahead-of-time planning and we’re happy to include Amazon in that. I’ll start a new issue on this topic assuming folks are interested in working on it.
Thanks @jkotas, I knew it was a long shot especially at this late point in the .NET 7 process but still had to ask.
Linux kernel 4.14+ is another minimum requirement. It is separate from the minimum glibc version requirement.
I am sorry we are not able to downgrade the Linux Arm64 build environment for Microsoft-built .NET 7 runtime to lower the minimum required glibc version. The alternatives and workarounds include:
The main motivation was that Ubuntu 18 is the oldest mainstream supported distro that works on Arm64.
CentOS 7 that we use to build for x64 does not work well on Arm64. And Ubuntu 16 that we use to build for arm64 in .NET 6 is out of mainstream support.
The change you have linked was just a follow up cleanup.
@normj Some digging was done by @slang25 earlier when we found this issue with Lambda, and this appears to be the main motivation: https://github.com/dotnet/runtime/commit/31cd7eb907133929846e5d229d18cc8bc77bde63
Thanks @richlander for continuing the discussions. I would appreciate being included in the coordination as Amazon Linux 2 is going to kick around for awhile. I am very concerned about the future path with .NET 8 especially if X64 is also a concern and I want to get ahead of the upcoming issues as soon as possible.
I should have been more precise here. We want the distro that we use as a build environment to have enough support life left. It means one year or more. It does not make sense to ship using build environment that is nearing end of life, and then switch to a different build environment after several months. It would just create unnecessary instability in servicing.
From the supported OS versions page, I note that you compile on 2.17 for x64 (CentOS7), but 2.27 on Ubuntu.
If you could compile on 2.26 that will fix the issue.