aws-sam-cli: Bug: when deploying with 1.67.0 causes missing go binary in lambda

Description:

When deploying with current latest release, Sam will successfully deploy a lambda, but the lambda has no go binary attached to it. When trying to call the lambda, I get an error: fork/exec /var/task/<binary-name>: permission denied: PathError null where <binary-name> is the name of the go binary.

Steps to reproduce:

build a working go binary (tried with both go build and sam build, doesn’t matter)
write a working simple go lambda template, which takes the go binary
install aws-sam-cli (we do that with the setup-sam github action, grabbing latest release)
run:

sam deploy \
            --no-fail-on-empty-changeset \
            --stack-name <stack-name> \
            --template-file template.yaml \
            --role-arn <role-arn>

send a query to the lambda
get error: fork/exec /var/task/<binary-name>: permission denied: PathError null, query returns 500.

Observed result:

We get error fork/exec /var/task/<binary-name>: permission denied: PathError null when calling the broken lambda. I can provide --debug output if needed, though we were not running it while we were having the issue. Cloud formation or Sam don’t complain and show all green though.

The issue doesn’t always happen, it’s a little random. Sometimes it works, sometimes it doesn’t. We had pretty much the same issue about a short year ago, but the issue stopped happening suddendly. Now it’s back.

Expected result:

The lambda deploys with the updated binary, and has access to it.

Everything seems to be working correctly in sam versions 1.53.0 and 1.66.0

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: linux, latest github workflow ubuntu versions
sam --version: 1.67.0
AWS region: eu-west-1

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (8 by maintainers)

Most upvoted comments

We’ve reached out internally to Lambda to clarify about the correct permission for Go binary. I’ll update here once we get an answer.

hawflau on Jan 11, 2023

Hi, in today’s 1.70.1 release we’ve reverted the changes that @SalladinBalwer mentioned. This should hopefully resolve the issues you were facing with Go. Let us know if you keep having them after updating to the latest version.

torresxb1 on Jan 24, 2023

I can’t answer for what that resolution claims (although it feels aimed at python and js projects), I can only report that Lambda has trouble starting up go binaries when the -x flag is missing, it is not consistent but it is an issue that happens enough to prevent deploying code to a production environment.

For now we can get by by locking the SAM version to 1.66.0 but it would be nice if we could use the latest version.

SalladinBalwer on Jan 10, 2023

@victorcq @SalladinBalwer Since 1.67.0, we’ve followed the resolution to explicitly set the permissions. Specifically,

The correct permissions for all executable files within a Lambda deployment package is 644 in Unix permissions numeric notation. For folders within a deployment package, the correct permissions setting is 755.

So it seems odd to me that the Lambda would need to “self-heal” at all, unless 644 is not the right permission for Lambda to run a go binary.

hawflau on Jan 10, 2023