aws-sam-cli: Sam local invoke (go) not honoring timeout in context

Description:

When running a golang lambda function with sam local invoke, and making a request (in the function) using the context results in RequestCanceled: request context canceled caused by: context deadline exceeded.

When using ListBuckets (without WithContext) results in a successful result.

Steps to reproduce:

hello.go

type Response events.APIGatewayProxyResponse

func Handler(ctx context.Context) (Response, error) {
	fmt.Printf("%+v\n", ctx)
	var buf bytes.Buffer

	sess := session.Must(session.NewSession(&aws.Config{Region: aws.String("us-west-2")}))
	svc := s3.New(sess)

	// Using the WithContext version of ListBuckets makes it apparent that the context isn't set to honor the function timeout
	b, err := svc.ListBucketsWithContext(ctx, &s3.ListBucketsInput{})
	if err != nil {
		fmt.Println(err)
	}
	fmt.Println(b)

	body, err := json.Marshal(map[string]interface{}{
		"message": "Go Serverless v1.0! Your function executed successfully!",
	})
	if err != nil {
		return Response{StatusCode: 404}, err
	}
	json.HTMLEscape(&buf, body)

	resp := Response{
		StatusCode:      200,
		IsBase64Encoded: false,
		Body:            buf.String(),
		Headers: map[string]string{
			"Content-Type":           "application/json",
			"X-MyCompany-Func-Reply": "hello-handler",
		},
	}

	return resp, nil
}

func main() {
	lambda.Start(Handler)
}

template.yml

AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: "Demo SAM Template"
Resources:
  Demo:
    Type: "AWS::Serverless::Function"
    Properties:
      Handler: bin/hello
      Runtime: go1.x
      MemorySize: 128
      Timeout: 30
      Environment:
        Variables:
          AWS_ACCESS_KEY_ID: myaccesskey
          AWS_SECRET_ACCESS_KEY: mysecretkey
          AWS_SESSION_TOKEN: mytoken
      Policies:
        - Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Action:
                - "s3:*"
              Resource: "*"

Observed result:

RequestCanceled: request context canceled caused by: context deadline exceeded

Expected result:

{
  Buckets: [
    {
      CreationDate: 2020-04-21 00:00:00 +0000 UTC,
      Name: "bucket-names"
    }
  ],
  Owner: {
    DisplayName: "...",
    ID: "..."
  }
}

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: macOS 10.15.7
  2. sam --version: SAM CLI, version 1.15.

Add --debug flag to command you are running

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 17
  • Comments: 17 (3 by maintainers)

Most upvoted comments

I rolled back to SAM CLI, version 1.12.0 to get this working again.

I’m experiencing this issue as well while it was working earlier on when I had v1.15. I’ve tried upgrading to v1.18.1 but that didn’t help. I have the exact same problem as @momotaro98 described.

If you do not set warm containers, they aren’t used. This is likely related to #2519

This issue appears to be fixed in the latest version: 1.37.0

Or at least my use case appears to be bug free. A non-expired context.Context is given to my Lambda with the correct timeout.

Go code:

package main

import (
	"context"
	"log"
	"os"

	"github.com/aws/aws-lambda-go/lambda"
)

type handler struct {
	logger *log.Logger
}

// Invoke implements the lambda.Handler interface and shows context.Context information.
func (h handler) Invoke(ctx context.Context, _ []byte) (response []byte, err error) {
	h.logger.Printf("%+v", ctx)

	select {
	case <-ctx.Done():
		h.logger.Println("Context expired.")
		err := ctx.Err()
		h.logger.Printf("Error: %s", err.Error())
		response = []byte(`{"statusCode":200,"headers":null,"multiValueHeaders":null,"body":"Failed due to expired context on invocation."}`)
	default:
		h.logger.Println("Context not expired.")
		response = []byte(`{"statusCode":200,"headers":null,"multiValueHeaders":null,"body":"Successfully returned response."}`)
	}

	return response, nil
}

func main() {
	logger := log.New(os.Stdout, "", log.LstdFlags)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	h := handler{
		logger: logger,
	}

	lambda.StartHandlerWithContext(ctx, h)
}

template.yml:

AWSTemplateFormatVersion: "2010-09-09"
Description: "An example Golang AWS SAM Lambda project."
Transform: "AWS::Serverless-2016-10-31"


Parameters:
  VaultToken:
    Type: "String"


Globals:
  Function:
    Handler: "main"
    Timeout: 10
    Runtime: "go1.x"


Resources:

  exampleOne:
    Type: "AWS::Serverless::Function"
    Properties:
      CodeUri: "cmd/one"
      Events:
        ApiEvent:
          Type: "HttpApi"
        Api:
          Type: "Api"
          Properties:
            Path: "/one"
            Method: "get"

Output:

$ sam --version && date && sam build && sam local invoke
SAM CLI, version 1.37.0
Sat 15 Jan 2022 04:19:18 PM EST
Building codeuri: /home/micah/GolandProjects/go-aws-sam-lambda-example/cmd/one runtime: go1.x metadata: {} architecture: x86_64 functions: ['exampleOne']
Running GoModulesBuilder:Build

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {stack-name} --watch
[*] Deploy: sam deploy --guided
        
Invoking main (go1.x)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-go1.x:rapid-1.37.0-x86_64.

Mounting /home/micah/GolandProjects/go-aws-sam-lambda-example/.aws-sam/build/exampleOne as /var/task:ro,delegated inside runtime container
START RequestId: ab0e8d22-adfb-4b9f-9d0e-bccea54000b9 Version: $LATEST
2022/01/15 21:19:22 context.Background.WithCancel.WithDeadline(2022-01-15 21:19:32.075598487 +0000 UTC [9.998813556s]).WithValue(type *lambdacontext.key, val <not Stringer>).WithValue(type string, val )
2022/01/15 21:19:22 Context not expired.
{"statusCode":200,"headers":null,"multiValueHeaders":null,"body":"Successfully returned response."}END RequestId: ab0e8d22-adfb-4b9f-9d0e-bccea54000b9
REPORT RequestId: ab0e8d22-adfb-4b9f-9d0e-bccea54000b9  Init Duration: 0.20 ms  Duration: 7.44 ms       Billed Duration: 8 ms   Memory Size: 128 MB     Max Memory Used: 128 MB

Notice the date is Sat 15 Jan 2022 04:19:18 PM EST and the context expires: 2022-01-15 21:19:32.075598487 +0000 UTC. The extra couple seconds are because of the time it took the sam build command to run.

This is the same behavior with sam local start-api and the Lambda is stopped early if the timeout in template.yml is reached.

++, got same bug now

For anyone wanting to know how to downgrade, you can use Pip3:

pip3 install --user 'aws-sam-cli==1.12.0'

This version seems to be working for me.