aws-sam-cli: Unable to debug golang function using dlv from AWS documentation (`GLIBC_2.32' not found)

Description

Unable to debug golang applications using sam local invoke following the documentation provided by AWS. glibc used to build dlv doesn’t seem to be compat with the glibc in the docker image. sam build cannot build in a docker container so not sure what we are suppose to do about this.

Steps to reproduce

mkdir delve
GOARCH=amd64 GOOS=linux go build -o delve/dlv github.com/go-delve/delve/cmd/dlv
GOARCH=amd64 GOOS=linux go build -o dlv github.com/go-delve/delve/cmd/dlv
sam local invoke CrmEntityImportWorkerFunction -e unittestdata/crmentityimporter/events/110791.event --profile=tbrunomind -d 5986 --debugger-path delve/ --debug-args "-delveAPI=2" --debug"

Observed result

local invoke command is called
Collected default values for parameters: {'ENV': 'dev'}
...
...
...
Private data, just list of ENV vars and template values
..
...
19 resources found in the template
Found Serverless function with name='CrmEntityImportWorkerFunction' and CodeUri='./crm-entity-import-worker/'
....
...
....
(Private data, just a list of all functions)
...
...
...
...
Found one Lambda function with name 'CrmEntityImportWorkerFunction'
Invoking crm-entity-import-worker (go1.x)
Environment variables overrides data is standard format
Loading AWS credentials from session with profile 'mywork'
Resolving code path. Cwd=/home/tbruno/Projects/GolandProjects/lambda-crm-integration, CodeUri=./crm-entity-import-worker/
Resolved absolute path to code is /home/tbruno/Projects/GolandProjects/lambda-crm-integration/crm-entity-import-worker
Code /home/tbruno/Projects/GolandProjects/lambda-crm-integration/crm-entity-import-worker is not a zip/jar file
Image was not found.
Building image...Adding custom GO Bootstrap to support debugging
.........................
Failed to download image with name amazon/aws-sam-cli-emulation-image-go1.x:debug-1.6.2
Failed to download a new amazon/aws-sam-cli-emulation-image-go1.x:debug-1.6.2 image. Invoking with the already downloaded image.
Mounting /home/tbruno/Projects/GolandProjects/lambda-crm-integration/crm-entity-import-worker as /var/task:ro,delegated inside runtime container
Setting up SIGTERM interrupt handler
/tmp/lambci_debug_files/dlv: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /tmp/lambci_debug_files/dlv)

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: ArchLinux/Manjaro
  2. sam --version: SAM CLI, version 1.6.2

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 20 (6 by maintainers)

Most upvoted comments

This is now a much wider issue with most distributions moving to the newer 3.32 version of glibc. Confirmed on Fedora 35, Ubuntu 21.10 and latest Arch & Manjaro installs.

We really need Go apps to be built in containers that contain a compatible lambda glibc version. This issue is not limited to aws-sam-cli. Any Go binary built on one of these newer distribution builds using glibc 2.32 or later will fail to run on any docker image or linux distro that uses glibc 2.31 or earlier.

I am running into the same issue @ivan-pesenti is facing. Anyone get debugging working with vscode?

Hi all!
I was stuck at trying to debug locally an AWS lambda function with Go as runtime. Below are my machine details:

  • OS: Ubuntu 21.10
  • AWS Cli: aws-cli/2.4.21 Python/3.8.8 Linux/5.13.0-37-generic exe/x86_64.ubuntu.21 prompt/off
  • SAM version: SAM CLI, version 1.43.0
  • IDE: VSCode
  • Golang version: go version go1.18 linux/amd64

Below are the repro steps (I apologize if I’m too verbose but I’d like to give you my precise setup):

  1. mkdir myLambda
  2. cd myLambda
  3. sam init
  4. Below are the choices in the prompt
    1. 1 - AWS Quick Start Template
    2. 1 - Hello World Example
    3. N
    4. 4 - go1.x
    5. 1 - Zip
    6. myLambda
  5. cd myLambda
  6. make
  7. Put a breakpoint in main.go file into handler()
  8. Hit “AWS: Add Debug Configuration” provided by CodeLens right above the signature of handler() func
  9. Start the debugger

I got this error:

2022-03-30 12:26:16 [INFO]: Building SAM application...
2022-03-30 12:26:16 [INFO]: Running command: (not started) [/usr/local/bin/sam build --build-dir /tmp/aws-toolkit-vscode/vsctkIfSuAr/output --template /home/ivan-pesenti-dev/Projects/go/myLambda/myLambda/template.yaml]
2022-03-30 12:26:16 [INFO]: Your template contains a resource with logical ID "ServerlessRestApi", which is a reserved logical ID in AWS SAM. It could result in unexpected behaviors and is not recommended.

2022-03-30 12:26:16 [INFO]: Building codeuri: /home/ivan-pesenti-dev/Projects/go/myLambda/myLambda/hello-world runtime: go1.x metadata: {} architecture: x86_64 functions: ['HelloWorldFunction']

2022-03-30 12:26:16 [INFO]: Running GoModulesBuilder:Build

2022-03-30 12:26:16 [INFO]: 
Build Succeeded

2022-03-30 12:26:16 [INFO]: 
Built Artifacts  : ../../tmp/aws-toolkit-vscode/vsctkIfSuAr/output
Built Template   : ../../tmp/aws-toolkit-vscode/vsctkIfSuAr/output/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke -t ../../tmp/aws-toolkit-vscode/vsctkIfSuAr/output/template.yaml
[*] Test Function in the Cloud: sam sync --stack-name {stack-name} --watch
[*] Deploy: sam deploy --guided --template-file ../../tmp/aws-toolkit-vscode/vsctkIfSuAr/output/template.yaml
        

2022-03-30 12:26:17 [INFO]: Build complete.
2022-03-30 12:26:17 [INFO]: Starting SAM application locally
2022-03-30 12:26:17 [INFO]: AWS.running.command
2022-03-30 12:26:17 [INFO]: Sending request to local API: http://127.0.0.1:5865/hello?
Mounting HelloWorldFunction at http://127.0.0.1:5865/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2022-03-30 12:26:18  * Running on http://127.0.0.1:5865/ (Press CTRL+C to quit)
Invoking hello-world (go1.x)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-go1.x:rapid-1.43.0-x86_64.

Mounting /tmp/aws-toolkit-vscode/vsctkIfSuAr/output/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: 7c0b5c25-69af-4047-92b7-2893547fac13 Version: $LATEST
/tmp/lambci_debug_files/dlv: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /tmp/lambci_debug_files/dlv)
/tmp/lambci_debug_files/dlv: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/lambci_debug_files/dlv)
2022/03/30 10:26:20 exit status 1

Other attempts

I tried with this Makefile:
.PHONY: build

build:
	sam build --use-container

Same error as above

CGO_ENABLED

I compiled the program with this statement

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build

After that I replaced the one below “.aws-ssm” folder but same error got.

Anyone can spot me in the right direction? If any details are missing I’ll provide.

Thanks!