aws-sam-cli: Testing Api Gateway endpoint locally returning `No response from invoke container` and `Invalid lambda response received` in Dockerised sam app
Description:
I’m dockerising sam app. When accessing api endpoint for HelloWorld locally, it responds with No response from invoke container for HelloWorldFunction and Invalid lambda response received: Lambda response must be valid json.
It works fine whithout dockerisation.
Steps to reproduce:
Dockerfile
FROM ruby:2.7.0-alpine
RUN apk add --update --no-cache \
build-base \
postgresql-dev \
postgresql-client \
python3 \
py3-pip \
util-linux \
python3-dev
RUN pip3 install aws-sam-cli
ARG USER
ARG HOME
ARG UID
RUN apk add --update \
sudo
RUN echo "Welcome home: $USER => $UID"
RUN adduser -S -D -G users -u $UID $USER
RUN addgroup -S sudo
RUN echo "%sudo ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/sudo
RUN adduser $USER sudo
RUN echo "Welcome home: $USER"
WORKDIR ${HOME}
EXPOSE 3001
ENTRYPOINT ["sh", "./entrypoint.sh"]
docker-compose.yml
version: '3.8'
services:
sam_app:
build:
context: ./sam-app
args:
- HOME
- USER
- UID
user: "${UID}:100"
command: ["$PWD"]
ports:
- "3001:3001"
volumes:
- ./sam-app:$HOME
- /var/run/docker.sock:/var/run/docker.sock
Entrypoint.sh
BASEDIR="$1"
echo "Basedir => ${BASEDIR}"
sudo sam local start-api \
--template ./template.yaml \
--host 0.0.0.0 \
--port 3001 \
--docker-volume-basedir "${BASEDIR}/sam-app/" \
--docker-network drink_default \
--debug
hello_world/app.rb
def lambda_handler(event:, context:)
{
statusCode: 200,
body: {
message: "Hello World!",
# location: response.body
}.to_json
}
end
Observed result:
am_app_1 | 2020-12-21 01:31:22,493 | Constructed String representation of Event to invoke Lambda. Event: {"body": null, "headers": {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", "Accept-Encoding": "gzip, deflate, br", "Accept-Language": "en-GB,en;q=0.9", "Connection": "keep-alive", "Host": "localhost:3001", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36", "X-Forwarded-Port": "3001", "X-Forwarded-Proto": "http"}, "httpMethod": "GET", "isBase64Encoded": false, "multiValueHeaders": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["en-GB,en;q=0.9"], "Connection": ["keep-alive"], "Host": ["localhost:3001"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"], "X-Forwarded-Port": ["3001"], "X-Forwarded-Proto": ["http"]}, "multiValueQueryStringParameters": null, "path": "/hello", "pathParameters": null, "queryStringParameters": null, "requestContext": {"accountId": "123456789012", "apiId": "1234567890", "domainName": "localhost:3001", "extendedRequestId": null, "httpMethod": "GET", "identity": {"accountId": null, "apiKey": null, "caller": null, "cognitoAuthenticationProvider": null, "cognitoAuthenticationType": null, "cognitoIdentityPoolId": null, "sourceIp": "172.22.0.1", "user": null, "userAgent": "Custom User Agent String", "userArn": null}, "path": "/hello", "protocol": "HTTP/1.1", "requestId": "dd99d1bb-cc40-42f8-a2f2-eeefcc22d111", "requestTime": "21/Dec/2020:01:30:38 +0000", "requestTimeEpoch": 1608514238, "resourceId": "123456", "resourcePath": "/hello", "stage": "Prod"}, "resource": "/hello", "stageVariables": null, "version": "1.0"}
sam_app_1 | 2020-12-21 01:31:22,493 | Found one Lambda function with name 'HelloWorldFunction'
sam_app_1 | 2020-12-21 01:31:22,494 | Invoking app.lambda_handler (ruby2.7)
sam_app_1 | 2020-12-21 01:31:22,494 | No environment variables found for function 'HelloWorldFunction'
sam_app_1 | 2020-12-21 01:31:22,494 | Environment variables overrides data is standard format
sam_app_1 | 2020-12-21 01:31:22,494 | Loading AWS credentials from session with profile 'None'
sam_app_1 | 2020-12-21 01:31:24,549 | Resolving code path. Cwd=/home/sameer/projects/drink/sam-app/, CodeUri=hello_world/
sam_app_1 | 2020-12-21 01:31:24,549 | Resolved absolute path to code is /home/sameer/projects/drink/sam-app/hello_world
sam_app_1 | 2020-12-21 01:31:24,550 | Code /home/sameer/projects/drink/sam-app/hello_world is not a zip/jar file
sam_app_1 | 2020-12-21 01:31:24,574 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-ruby2.7:rapid-1.15.0.
sam_app_1 |
sam_app_1 | 2020-12-21 01:31:24,574 | Mounting /home/sameer/projects/drink/sam-app/hello_world as /var/task:ro,delegated inside runtime container
sam_app_1 | 2020-12-21 01:31:25,386 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
sam_app_1 | 2020-12-21 01:31:26,707 | Cleaning all decompressed code dirs
sam_app_1 | 2020-12-21 01:31:26,707 | No response from invoke container for HelloWorldFunction
sam_app_1 | 2020-12-21 01:31:26,707 | Invalid lambda response received: Lambda response must be valid json
sam_app_1 | 2020-12-21 01:31:26 172.22.0.1 - - [21/Dec/2020 01:31:26] "GET /hello HTTP/1.1" 502 -
sam_app_1 | 2020-12-21 01:31:26 172.22.0.1 - - [21/Dec/2020 01:31:26] "GET /favicon.ico HTTP/1.1" 403 -
Expected result:
Json output on browser {message: “hello world”}
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
- OS: ubuntu VERSION=“20.04.1 LTS (Focal Fossa)”
sam --version: 1.15.0Homebrew: VERSION=“2.6.2”docker: VERSION=“19.03.8”
Add --debug flag to command you are running
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 22
- Comments: 41 (9 by maintainers)
Commits related to this issue
- To use this in CircleCI with a remote docker, we need an ssh tunnel This works around the same error as https://github.com/aws/aws-sam-cli/issues/2492 See https://circleci.com/docs/2.0/building-dock... — committed to ismith/aws-sam-cli by ismith 3 years ago
- To use this in CircleCI with a remote docker, we need an ssh tunnel This works around the same error as https://github.com/aws/aws-sam-cli/issues/2492 See https://circleci.com/docs/2.0/building-dock... — committed to ismith/aws-sam-cli by ismith 3 years ago
- To use this in CircleCI with a remote docker, we need an ssh tunnel This works around the same error as https://github.com/aws/aws-sam-cli/issues/2492 See https://circleci.com/docs/2.0/building-dock... — committed to ismith/aws-sam-cli by ismith 3 years ago
- To use this in CircleCI with a remote docker, we need an ssh tunnel This works around the same error as https://github.com/aws/aws-sam-cli/issues/2492 See https://circleci.com/docs/2.0/building-dock... — committed to ismith/aws-sam-cli by ismith 3 years ago
@alliesground SAM CLI is meant to be run from your machine and interact with docker, not run SAM CLI within Docker. My suggestion here is to do that instead of Dockerize.
What’s the use case on wanting to do this? Why do you need to be running docker within docker?
Another common use case for dockerizing sam: CI pipelines. Like @jasonterando said, it’s a pain to maintain packages/tools installed on build agents. Different teams need their tools in particular ways. It’s more like dependencies for your projects. The tools break over time, so you need to keep your setup predictable. That’s what docker provides. If it’s “not a good idea” to use sam within docker, then I run out of ideas to make my CI pipeline stable over long period of time. I can’t trust my scripts/code to still work 1 year from now, after countless patching/upgrading of the build agents. Not to mention that normally it’s not easy to tweak infrastructure setups within big corporates, it’s a long and risky process. Sometimes impossible (e.g. they refuse to install arbitrary software on agents, because they gave you docker already).
I fixed this like this - Essentially, make sure your SAM local and database are running on the same Docker network that it not host.
Use
docker-compose up -d, thendocker network lsto get the network name or look at the logs for the network createdDATABASE_URL=postgres://user:password@postgres:5432/databaseName- Notice the host changes fromlocalhosttopostgressam local start-api --docker-network=app-name_apiThank you all for the feedback. I have added this to our backlog and we’ll post updates here as we prioritize it.
@jfuss my use case is similar to that of @rbliss i.e to create an isolated dev environment, which includes react-app and db, all composed with docker-compose.
downgrade to 1.12.0. works
@gpomykala Your issue is logged here: https://github.com/aws/aws-sam-cli/issues/2436#issuecomment-742242804 and workarounds here: https://github.com/aws/aws-sam-cli/issues/2436#issuecomment-743472832
@rbliss @alayor @alliesground We have seen this happen when SAM CLI cannot communicate to the container. Instead of exec-ing within the container, we updated to move to publishing a port and then calling into the container of http. This could be the cause here as well. SAM CLI will communicate to the container over localhost only, so there is probably some docker network trickery going on. I highly suggest to not use SAM CLI with the container like this. It’s not really a use case we directly support or recommend.
I receive same error when trying to hit a lambda spun with ‘sam local start-lambda --docker-network host’. It used to work with older version of SAM CLI. I am not doing anything extraordinary, just a basic integration test of a lambda client hitting lambda hosted in SAM CLI
Thanks for looking into this @sapessi. Have you got any updates you might be able to share on progress?
I’d like to echo the points @jasonterando made on the use-cases for this functionality. Essentially, we’d like to be able to run the CLI via docker to allow:
@kstro21 In your case, yes these two options have the same value. In some cases, for example, we want to use it in a docker container,
--container-hostwill behost.docker.internaland--container-host-interfacewill be127.0.0.1since we want it bind to loopback interface because of security reason.--container-hostis the host of locally emulated Lambda container while--container-host-interfaceis used to specify interface it binds ports to. They two options should work independently. Hopefully this makes sense to you.Ok, I have new information on the issue.
https://github.com/aws/aws-sam-cli/blob/86f88cbd7df365960f7015c5d086b0db7aedd9d5/samcli/local/docker/container.py#L153-L158
Can you see how ‘127.0.0.1’ is hardcoded on the code? Why? If I remove the
127.0.0.1from those 2 lines, everything work as expected.@rv2673 Using
--container-hostto replace127.0.0.1is a better idea in my opinion since it won’t bind to all interfaces. It would support your use case and use case of @kstro21 . Thanks for looking into the warm container issue. We realized that issue before last release so we decided to revert the change. We would like to release a completed function as possible. Currently I’m preparing a PR to include following changes:127.0.0.1with the new option--container-hostso it can support remote Docker host.For DevOps, we like to launch containers that execute build and then unit and integration testing. It keeps us from having to clutter up our CI/CD server with external dependencies for which we have to keep track of for updates, etc. For example, I can deploy a PHP application which runs composer install and PHPUnit without having to have PHP, composer, etc. installed on the CI/CD server, and have to worry about keeping them up-to-date. In the case of SAM, it’d be nice to start off with a base image of AWS SAM CLI, add in TypeScript, do my testing and then build by deployment all from a container, without having to deal with installing dependencies on my CI/CD server.
For development purposes, it would be slick to be able to set up a Docker Compose configuration that launches the SAM-configured service with accompanying dependencies (like localstack for simulating communication with Amazon resources like S3 and Kinesis), mocked external services, etc… Using Docker Compose for this is largely OS-agnostic, without having to install everything on the developer’s machine. Alternatively I can stand up Bash, PowerShell, etc. scripts to launch and integrate things, but it’s more clunky, and in the case of installing the SAM CLI, may involve installing things like Homebrew which make changes to the developer’s Python environment.
I was able to get SAM local invoke to work inside Dev Container by replacing hardcoded
localhostherehttps://github.com/aws/aws-sam-cli/blob/develop/samcli/local/docker/container.py#L41
with
host.docker.internal. I used this guide for attaching the VS Code debugger: https://github.com/aws/aws-toolkit-vscode/blob/a380685696e39247d6ec0f8f1e9928ba2562f550/docs/debugging-python-lambda-functions.md, also replacinglocalhostwithhost.docker.internalinlaunch.json.@jfuss I’m running this on macOS, AFAIK this won’t work on linux out of the box due to https://github.com/docker/for-linux/issues/264, but a SAM CLI option to specify host would be nice.
@xazhao I do understand that the 127.0.0.1 is sensible for security, though what about having a flag to be able to change to which ip/host it binds for those who need it with a ‘remote’ docker host?
In our setup we have the following:
The docker host is remote from the perspective of the build container.
I slightly modified sam cli in our build container image to get it to work currently:
I have a working compose setup where in the
ENTRYPOINTI’m hacking the localhost IP in/etc/hostsof the container runningsam local start-lambda:@alayor I am open to ways to make this work if we can find a solution that works across OS. I am not sure if moving to the
docker.for.mac.localhost(orhost.docker.internalwhich I think might work across OS).Just mapping this out:
sam localcommand will start up a container and publish ports to localhost (not sure if this is localhost of the machine or on the bridge docker network).If in step two, this does publish a port on the machines localhost, then I would assume what you are suggesting could work. Super hacky and seems much easier to just run the CLI on your machine and not have to care about docker at all. Could be a personal preference though.
I’m also having the same problem. My use case is exactly the same as @alliesground.
I followed the instructions on this post. It makes me think this used to work before. https://medium.com/monsoon-engineering/running-aws-sam-in-a-docker-container-2491596672c2
Other than downgrading to 1.12.0, is there a workaround for newer versions when not doing anything network related? I’m keen to try to use warm containers in the newer versions to profile performance without the Java classloader penalties, but I’m only doing
sam local start-apiand getting the invoke errors as with others which makes local development virtually impossible.I would assume this would be a more widespread issue if it didn’t work for anyone, so would there be something about my environment that would cause this?
I’m using Docker Toolbox (unfortunately I can’t use Desktop due to Hyper-V issues) which means I’m stuck with Docker 19.03. As I’m on Windows, this is all running in a VirtualBox VM and I’m invoking SAM on the Windows side. I’ve not done anything to the network configuration with regards to Docker, but creating a network with
docker network create testand then doing--docker-network testdoes not help in case it’s defaulting to host networking of some sort. This is a fairly standard Maven app; there’s no Dockerfile image building if that matters.That was a pretty good tip but the issue is still unresolved:
Navigating to the url:
Navigating to the url with the proper endpoint:
This is a mess. I just want to have an environment to test my lambdas before deploying them.
For those who are wondering how to downgrade to 1.12.0 – this worked for me:
Hey, @xazhao, yes, I have SAM CLI and my project on my local computer, and the Docker engine is running on a remote host, Docker has access to my SAM files through a mounted remoted directory. But if you are in Linux, you can easily reproduce it using what @jamesorlakin is using, Docker Toolbox which will run the Docker containers inside a VirtualBox Virtual Machine.
Let me know if that helps.