aws-sam-cli: 1.13.1: Invalid lambda response received: Lambda response must be valid json

Description:

sam local start-api stopped triggering Lambda by HTTP request. Downgrading to 1.12.0 gives the expected result.

Steps to reproduce:

sam build --use-container --debug
sam local start-api --port 4004 --docker-network host --skip-pull-image --debug &
curl -X POST http://127.0.0.1:4004/api/v1/app/add

Observed result:

2020-12-02 19:41:11,763 | Constructed String representation of Event to invoke Lambda. Event: {"body": null, "headers": {"Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Connection": "keep-alive", "Content-Length": "0", "Host": "127.0.0.1:4004", "User-Agent": "python-requests/2.23.0", "X-Forwarded-Port": "4004", "X-Forwarded-Proto": "http"}, "httpMethod": "POST", "isBase64Encoded": false, "multiValueHeaders": {"Accept": ["*/*"], "Accept-Encoding": ["gzip, deflate"], "Connection": ["keep-alive"], "Content-Length": ["0"], "Host": ["127.0.0.1:4004"], "User-Agent": ["python-requests/2.23.0"], "X-Forwarded-Port": ["4004"], "X-Forwarded-Proto": ["http"]}, "multiValueQueryStringParameters": null, "path": "/api/v1/app/add", "pathParameters": {"schema_app": "app", "schema_version": "v1"}, "queryStringParameters": null, "requestContext": {"accountId": "123456789012", "apiId": "1234567890", "domainName": "127.0.0.1:4004", "extendedRequestId": null, "httpMethod": "POST", "identity": {"accountId": null, "apiKey": null, "caller": null, "cognitoAuthenticationProvider": null, "cognitoAuthenticationType": null, "cognitoIdentityPoolId": null, "sourceIp": "127.0.0.1", "user": null, "userAgent": "Custom User Agent String", "userArn": null}, "path": "/api/{schema_version}/{schema_app}/add", "protocol": "HTTP/1.1", "requestId": "b5ee986a-61f4-4954-b68b-c2af1457d70c", "requestTime": "02/Dec/2020:15:39:25 +0000", "requestTimeEpoch": 1606923565, "resourceId": "123456", "resourcePath": "/api/{schema_version}/{schema_app}/add", "stage": "staging"}, "resource": "/api/{schema_version}/{schema_app}/add", "stageVariables": null, "version": "1.0"}
2020-12-02 19:41:11,764 | Found one Lambda function with name 'Locator'
2020-12-02 19:41:11,764 | Invoking handler_locator.handle (python3.7)
2020-12-02 19:41:11,764 | Environment variables overrides data is standard format
2020-12-02 19:41:11,764 | Resolving code path. Cwd=/Users/BR0kEN/projects/SANITIZED/.aws-sam/build, CodeUri=Locator
2020-12-02 19:41:11,764 | Resolved absolute path to code is /Users/BR0kEN/projects/SANITIZED/.aws-sam/build/Locator
2020-12-02 19:41:11,764 | Code /Users/BR0kEN/projects/SANITIZED/.aws-sam/build/Locator is not a zip/jar file
2020-12-02 19:41:11,783 | Requested to skip pulling images ...

2020-12-02 19:41:11,783 | Mounting /Users/BR0kEN/projects/SANITIZED/.aws-sam/build/Locator as /var/task:ro,delegated inside runtime container
2020-12-02 19:41:12,269 | Starting a timer for 180 seconds for function 'Locator'
2020-12-02 19:41:13,118 | No response from invoke container for Locator
2020-12-02 19:41:13,119 | Invalid lambda response received: Lambda response must be valid json
2020-12-02 19:41:13 127.0.0.1 - - [02/Dec/2020 19:41:13] "POST /api/v1/app/add HTTP/1.1" 502 -

Expected result:

2020-12-02 19:43:51,479 | Constructed String representation of Event to invoke Lambda. Event: {"body": null, "headers": {"Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Connection": "keep-alive", "Content-Length": "0", "Host": "127.0.0.1:4004", "User-Agent": "python-requests/2.23.0", "X-Forwarded-Port": "4004", "X-Forwarded-Proto": "http"}, "httpMethod": "POST", "isBase64Encoded": false, "multiValueHeaders": {"Accept": ["*/*"], "Accept-Encoding": ["gzip, deflate"], "Connection": ["keep-alive"], "Content-Length": ["0"], "Host": ["127.0.0.1:4004"], "User-Agent": ["python-requests/2.23.0"], "X-Forwarded-Port": ["4004"], "X-Forwarded-Proto": ["http"]}, "multiValueQueryStringParameters": null, "path": "/api/v1/app/add", "pathParameters": {"schema_app": "app", "schema_version": "v1"}, "queryStringParameters": null, "requestContext": {"accountId": "123456789012", "apiId": "1234567890", "domainName": "127.0.0.1:4004", "extendedRequestId": null, "httpMethod": "POST", "identity": {"accountId": null, "apiKey": null, "caller": null, "cognitoAuthenticationProvider": null, "cognitoAuthenticationType": null, "cognitoIdentityPoolId": null, "sourceIp": "127.0.0.1", "user": null, "userAgent": "Custom User Agent String", "userArn": null}, "path": "/api/{schema_version}/{schema_app}/add", "protocol": "HTTP/1.1", "requestId": "03b60fd2-78ab-4fc4-bcfa-930e57bfd9ee", "requestTime": "02/Dec/2020:15:43:40 +0000", "requestTimeEpoch": 1606923820, "resourceId": "123456", "resourcePath": "/api/{schema_version}/{schema_app}/add", "stage": "staging"}, "resource": "/api/{schema_version}/{schema_app}/add", "stageVariables": null, "version": "1.0"}
2020-12-02 19:43:51,479 | Found one Lambda function with name 'Locator'
2020-12-02 19:43:51,480 | Invoking handler_locator.handle (python3.7)
2020-12-02 19:43:51,480 | Environment variables overrides data is standard format
2020-12-02 19:43:51,480 | Loading AWS credentials from session with profile 'None'
2020-12-02 19:43:51,492 | Resolving code path. Cwd=/Users/BR0kEN/projects/SANITIZED/.aws-sam/build, CodeUri=Locator
2020-12-02 19:43:51,493 | Resolved absolute path to code is /Users/BR0kEN/projects/SANITIZED/.aws-sam/build/Locator
2020-12-02 19:43:51,493 | Code /Users/BR0kEN/projects/SANITIZED/.aws-sam/build/Locator is not a zip/jar file
2020-12-02 19:43:51,500 | Image was not found.
Building image.........
2020-12-02 19:43:51,831 | Requested to skip pulling images ...

2020-12-02 19:43:51,832 | Mounting /Users/BR0kEN/projects/SANITIZED/.aws-sam/build/Locator as /var/task:ro,delegated inside runtime container
2020-12-02 19:43:52,374 | Starting a timer for 180 seconds for function 'Locator'
START RequestId: e1cdfb2a-7475-1d51-6178-88c4176b300c Version: $LATEST
[ERROR]	2020-12-02T15:43:53.738Z	e1cdfb2a-7475-1d51-6178-88c4176b300c	POST: handler failed.
Traceback (most recent call last):
  File "/var/task/src/controller.py", line 130, in _get_data_document_prepared_for_storage
    document = self._schema.get_data_document(request.body, update)
  File "/var/task/src/schema/schema.py", line 117, in get_data_document
    raise ValidationError(["no data provided"])
src.validation.errors.ValidationError: ['no data provided']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/src/handler/handler.py", line 149, in handle_request
    return _handlers[method](HandlerRequest(method, resource, body, query))
  File "/var/task/src/controller.py", line 31, in create
    fields=self._get_data_document_prepared_for_storage(request, update=False),
  File "/var/task/src/controller.py", line 132, in _get_data_document_prepared_for_storage
    raise HandlerError(error_code, {"body": error.errors})
src.handler.handler.HandlerError: Invalid document
END RequestId: e1cdfb2a-7475-1d51-6178-88c4176b300c
REPORT RequestId: e1cdfb2a-7475-1d51-6178-88c4176b300c	Init Duration: 1349.40 ms	Duration: 46.97 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 32 MB
2020-12-02 19:43:53 127.0.0.1 - - [02/Dec/2020 19:43:53] "POST /api/v1/app/add HTTP/1.1" 400 -

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: macOS Big Sur 11.0.1 (20B29)
sam --version: 1.13.1

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 24
Comments: 37 (5 by maintainers)

Most upvoted comments

@sdwvit

pip3 install aws-sam-cli==1.12.0

BR0kEN- on Dec 7, 2020

Close the issue since we released two options --container-host and --container-host-interface. In short, if you would like to run SAM CLI in a docker container on macOS or Windows, use --container-host with value host.docker.internal will have SAM CLI work properly.

@xazhao Can you please give an example for Linux?

spmacdonald on Dec 1, 2021

I’m using aws-sam-cli inside a docker conatiner and still can’t get it working. Basically I want to run sam-app from official tutorial (https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html) inside docker container. So I have following Dockerfile:

FROM python:3.7.4-slim-buster

RUN pip install aws-sam-cli
RUN apt-get update && apt-get install -y git
RUN sam init --runtime python3.7 --dependency-manager pip --app-template hello-world --name sam-app --package-type Zip

WORKDIR sam-app

CMD sam build && sam local invoke "HelloWorldFunction" -e events/event.json

Running such Dockerfile (docker run -v /var/run/docker.sock:/var/run/docker.sock) ends with:

No response from invoke container for HelloWorldFunction

I’ve tried to run it in non-host network, but no success yet.

smalec on Jan 1, 2021

So I was doing some digging on #2436 and I think they are both related.

So between 1.12 and 1.13, we changed how we communicate with the container. In 1.12, we just execute the container to run the code. With 1.13, we updated to use aws-lambda-rie. aws-lambda-rie stands up an endpoint within the container to communicate with the runtime itself. Since we are not communicating over http with the container, SAM CLI exposes a port to communicate with. From what I can gather, Docker will not let you expose a port if you connect to the ‘host’ network. The problem is Docker doesn’t fail if you configure a port and run on the host network, why docker why!?!?

This is why SAM CLI cannot communicate with the local function. I need to do some deeper investigation to see how we can get around this docker limitation. I am not trilled to have to move back to executing the container. Communicating over http gives us much more control and allows us to integrate nicely with aws-lambda-rie. Open to suggestion on this.

This only appears to break for the host network, so if you drop the --docker-network host or provide a non host network thing should work as expected.

jfuss on Dec 10, 2020

Same issue here with 1.18.2 on Mac OS Catalina

MatteoGioioso on Mar 7, 2021

I’m getting the same error while trying to complete official tutorial: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html.

Downgrading to 1.12.0 is not an option for those who use Lambda Images.

smalec on Dec 8, 2020

i start sam cli in docker container with

  --host 0.0.0.0 \
  --container-host host.docker.internal \

It still does not work. Im on MacOS

dmngu9 on Sep 28, 2022

Close the issue since we released two options --container-host and --container-host-interface. In short, if you would like to run SAM CLI in a docker container on macOS or Windows, use --container-host with value host.docker.internal will have SAM CLI work properly.

I have dockerized SAM following similar posts above and the command worked after using this suggestion!!! thank you!!!

sam local start-api -p 3000 --host 0.0.0.0 --docker-network aws_backend --container-host host.docker.internal

marcellodesales on Aug 1, 2022

Close the issue since we released two options --container-host and --container-host-interface. In short, if you would like to run SAM CLI in a docker container on macOS or Windows, use --container-host with value host.docker.internal will have SAM CLI work properly.

xazhao on Apr 27, 2021

I’m seeing the same issue when trying to have follow this structure with DynamoDB

Basically the lambda works up until I enter DynamoDB to the mix, and the code is not that complicated:

const dynamoDb = require('aws-sdk/clients/dynamodb');

exports.handler = async (event, context) => {
  try {
    const params = {
      TableName: 'Main',
    };

    const docClient = new dynamoDb.DocumentClient({
      endpoint: 'http://localhost:8000',
    });

    const response = await docClient.scan(params).promise();

    return {
      statusCode: 200,
      headers: {
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        items: response,
      }),
    };
  } catch (err) {
    console.log(err);
    return {
      statusCode: 500,
    };
  }
};

This is how I started the container for dynamodb:

❯ docker run -d -p 8000:8000 --network=local-api-network --name dynamo-local amazon/dynamodb-local

And this is the response once I upload a schema with the NoSQL Workbench:

❯ aws dynamodb list-tables --endpoint-url http://localhost:8000
{
    "TableNames": [
        "Main"
    ]
}

But still, it’s not working and I’m running all this on linux. This is the dreaded error:

No response from invoke container for getAllItemsFn562F4A69
Invalid lambda response received: Lambda response must be valid json

fev4 on Apr 17, 2021

It seems that the suggested workarounds of providing a non-host --docker-network, or omitting that param entirely, do not work when running sam local from within a Docker container. I think this is a separate symptom of the same defect, since the error is the same.

To replicate this, put the following files into an AWS SAM project root:

Dockerfile:

FROM amazon/aws-cli:2.1.28

ARG AWS_SAM_CLI_VERSION=1.19.1

RUN yum -y update && \
    yum install -y python3 && \
    python3 -m pip --disable-pip-version-check install aws-sam-cli==${AWS_SAM_CLI_VERSION}

ENTRYPOINT [ "" ]

and docker-compose.yml :

version: "3.7"

services:
  testcase:
    build: .
    working_dir: $PWD
    command: sam local start-api --template template.yaml --host 0.0.0.0 -v $PWD
    ports:
      - 3000:3000
    volumes:
      - '$PWD:$PWD'
      - '/var/run/docker.sock:/var/run/docker.sock'

Now run the application locally:

docker-compose up

Hitting any of your API endpoints will result in a 502 BAD GATEWAY response with the error message Invalid lambda response received: Lambda response must be valid json.

This is pretty disruptive for us. We use Docker to ensure that all team members, and the CI server, use exactly the same version of the AWS SAM CLI binaries. The CLI project updates frequently enough that attempting to keep everyone in sync would be a bit of a nightmare.

Our solution at the moment is to pin our version at 1.12, the last version before this defect, but this can’t last. We will very soon need to upgrade to 1.17 for HTTP API Payload v2 support.

SamuelCook on Mar 1, 2021

We’re getting the same 502 response on 1.13.2. But we’re not using the --docker-network flag at all.

We’ve been forced to fall back to 1.12.0.

Is there any information I can provide about our setup to help with this?

BenVosper on Dec 16, 2020