aws-sam-cli: Bug: Parallel Requests fail with 502 Bad Gateway
Description:
I have a lambda which I wish to test locally, using sam local start-api.
This works fine for single requests but when my UI makes multiple requests in quick succession one call usually fails with 502 Bad Gateway.
Steps to reproduce:
Observed result:
sam local start-api --debug
2023-10-26 14:42:42,764 | Config file location: /Users/mark/git/notifications-dao-lambda/samconfig.toml
2023-10-26 14:42:42,766 | Loading configuration values from [default.['local', 'start-api'].parameters] (env.command_name.section) in config file at
'/Users/mark/git/notifications-dao-lambda/samconfig.toml'...
2023-10-26 14:42:42,766 | Configuration values successfully loaded.
2023-10-26 14:42:42,767 | Configuration values are: {'stack_name': 'sam-pipeline-app', 'warm_containers': 'EAGER'}
2023-10-26 14:42:42,771 | Using SAM Template at /Users/mark/git/notifications-dao-lambda/.aws-sam/build/template.yaml
2023-10-26 14:42:42,789 | Using config file: samconfig.toml, config environment: default
2023-10-26 14:42:42,790 | Expand command line arguments to:
2023-10-26 14:42:42,790 | --template_file=/Users/mark/git/notifications-dao-lambda/.aws-sam/build/template.yaml --host=127.0.0.1 --port=3000
--static_dir=public --layer_cache_basedir=/Users/mark.pierce/.aws-sam/layers-pkg --warm_containers=EAGER --container_host=localhost --container_host_interface=127.0.0.1
2023-10-26 14:42:42,848 | local start-api command is called
2023-10-26 14:42:42,852 | No Parameters detected in the template
2023-10-26 14:42:42,867 | There is no customer defined id or cdk path defined for resource AwsApiGateway, so we will use the resource logical id as the resource id
2023-10-26 14:42:42,867 | Sam customer defined id is more priority than other IDs. Customer defined id for resource LambdaFunction is LambdaFunction
2023-10-26 14:42:42,868 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ProxyFunction is ProxyFunction
2023-10-26 14:42:42,868 | 0 stacks found in the template
2023-10-26 14:42:42,869 | No Parameters detected in the template
2023-10-26 14:42:42,878 | There is no customer defined id or cdk path defined for resource AwsApiGateway, so we will use the resource logical id as the resource id
2023-10-26 14:42:42,878 | Sam customer defined id is more priority than other IDs. Customer defined id for resource LambdaFunction is LambdaFunction
2023-10-26 14:42:42,878 | Sam customer defined id is more priority than other IDs. Customer defined id for resource ProxyFunction is ProxyFunction
2023-10-26 14:42:42,879 | 3 resources found in the stack
2023-10-26 14:42:42,879 | Found Serverless function with name='LambdaFunction' and CodeUri='LambdaFunction'
2023-10-26 14:42:42,880 | --base-dir is not presented, adjusting uri LambdaFunction relative to
/Users/mark/git/notifications-dao-lambda/.aws-sam/build/template.yaml
2023-10-26 14:42:42,880 | Found Serverless function with name='ProxyFunction' and CodeUri='ProxyFunction'
2023-10-26 14:42:42,880 | --base-dir is not presented, adjusting uri ProxyFunction relative to
/Users/mark/git/notifications-dao-lambda/.aws-sam/build/template.yaml
2023-10-26 14:42:42,881 | watch resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/template.yaml
2023-10-26 14:42:42,881 | Create Observer for resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/template.yaml with recursive True
2023-10-26 14:42:42,883 | watch resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/template.yaml's parent
/Users/mark/git/notifications-dao-lambda/.aws-sam/build
2023-10-26 14:42:42,884 | Create Observer for resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build with recursive False
2023-10-26 14:42:42,897 | Initializing the lambda functions containers.
2023-10-26 14:42:42,897 | Async execution started
2023-10-26 14:42:42,898 | Invoking function functools.partial(<function InvokeContext._initialize_all_functions_containers.<locals>.initialize_function_container at 0x104675940>,
Function(function_id='LambdaFunction', name='LambdaFunction', functionname='LambdaFunction', runtime='python3.11', memory=250, timeout=30, handler='lambda_function.lambda_handler',
imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/mark/git/notifications-dao-lambda/.aws-sam/build/LambdaFunction',
environment={'Variables': {'AWS_ENDPOINT_URL_DYNAMODB': 'http://docker.for.mac.localhost:8000', 'SAM_CLI_TELEMETRY': 0, 'pact_environment': 'true', 'database_name': 'TestDatabase'}},
rolearn=None, layers=[], events={'GetEvent': {'Type': 'HttpApi', 'Properties': {'ApiId': 'AwsApiGateway', 'Path': '/{proxy+}/api/v10/notifications', 'Method': 'GET'}}, 'PutEvent':
{'Type': 'HttpApi', 'Properties': {'ApiId': 'AwsApiGateway', 'Path': '/{proxy+}/api/v10/notifications/{creationDate}/read', 'Method': 'PUT'}}, 'DeleteEvent': {'Type': 'HttpApi',
'Properties': {'ApiId': 'AwsApiGateway', 'Path': '/{proxy+}/api/v10/notifications/{creationDate}', 'Method': 'DELETE'}}}, metadata={'SamResourceId': 'LambdaFunction'}, inlinecode=None,
codesign_config_arn=None, architectures=None, function_url_config=None, function_build_info=<FunctionBuildInfo.BuildableZip: ('BuildableZip', 'Regular ZIP function which can be build with
SAM CLI')>, stack_path='', runtime_management_config=None))
2023-10-26 14:42:42,900 | Invoking function functools.partial(<function InvokeContext._initialize_all_functions_containers.<locals>.initialize_function_container at 0x104675940>,
Function(function_id='ProxyFunction', name='ProxyFunction', functionname='ProxyFunction', runtime='python3.11', memory=512, timeout=60, handler='lambda_function.lambda_handler',
imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction',
environment={'Variables': {'SAM_CLI_TELEMETRY': 0}}, rolearn=None, layers=[], events={'GetEvent': {'Type': 'HttpApi', 'Properties': {'ApiId': 'AwsApiGateway', 'Path':
'/server/api/v9/{proxy+}', 'Method': 'GET'}}, 'PostEvent': {'Type': 'HttpApi', 'Properties': {'ApiId': 'AwsApiGateway', 'Path': '/server/api/v9/{proxy+}', 'Method': 'POST'}}},
metadata={'SamResourceId': 'ProxyFunction'}, inlinecode=None, codesign_config_arn=None, architectures=None, function_url_config=None, function_build_info=<FunctionBuildInfo.BuildableZip:
('BuildableZip', 'Regular ZIP function which can be build with SAM CLI')>, stack_path='', runtime_management_config=None))
2023-10-26 14:42:42,903 | Waiting for async results
2023-10-26 14:42:42,908 | Loading AWS credentials from session with profile 'None'
2023-10-26 14:42:42,909 | Loading AWS credentials from session with profile 'None'
2023-10-26 14:42:43,349 | Resolving code path. Cwd=/Users/mark/git/notifications-dao-lambda/.aws-sam/build,
CodeUri=/Users/mark/git/notifications-dao-lambda/.aws-sam/build/LambdaFunction
2023-10-26 14:42:43,353 | Resolved absolute path to code is /Users/mark/git/notifications-dao-lambda/.aws-sam/build/LambdaFunction
2023-10-26 14:42:43,356 | Resolving code path. Cwd=/Users/mark/git/notifications-dao-lambda/.aws-sam/build,
CodeUri=/Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction
2023-10-26 14:42:43,357 | Resolved absolute path to code is /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction
2023-10-26 14:42:44,079 | watch resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/LambdaFunction
2023-10-26 14:42:44,079 | Create Observer for resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/LambdaFunction with recursive True
2023-10-26 14:42:44,081 | watch resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/LambdaFunction's parent
/Users/mark/git/notifications-dao-lambda/.aws-sam/build
2023-10-26 14:42:44,081 | Code /Users/mark/git/notifications-dao-lambda/.aws-sam/build/LambdaFunction is not a zip/jar file
2023-10-26 14:42:44,794 | watch resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction
2023-10-26 14:42:44,795 | Create Observer for resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction with recursive True
2023-10-26 14:42:44,797 | watch resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction's parent
/Users/mark/git/notifications-dao-lambda/.aws-sam/build
2023-10-26 14:42:44,798 | Code /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction is not a zip/jar file
2023-10-26 14:42:47,404 | Local image is up-to-date
2023-10-26 14:42:47,428 | Using local image: public.ecr.aws/lambda/python:3.11-rapid-x86_64.
2023-10-26 14:42:47,430 | Mounting /Users/mark/git/notifications-dao-lambda/.aws-sam/build/LambdaFunction as /var/task:ro,delegated, inside
runtime container
2023-10-26 14:42:47,869 | Local image is up-to-date
2023-10-26 14:42:47,890 | Using local image: public.ecr.aws/lambda/python:3.11-rapid-x86_64.
2023-10-26 14:42:47,891 | Mounting /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction as /var/task:ro,delegated, inside
runtime container
2023-10-26 14:42:48,071 | Async execution completed
2023-10-26 14:42:48,072 | Containers Initialization is done.
2023-10-26 14:42:48,073 | Detected Inline Swagger definition
2023-10-26 14:42:48,073 | Parsing Swagger document using 3.0 specification
2023-10-26 14:42:48,074 | Lambda function integration not found in Swagger document at path='/{proxy+}/api/v10/notifications/{creationDate}/read' method='put'
2023-10-26 14:42:48,074 | Lambda function integration not found in Swagger document at path='/{proxy+}/api/v10/notifications' method='get'
2023-10-26 14:42:48,074 | Lambda function integration not found in Swagger document at path='/{proxy+}/api/v10/notifications/{creationDate}' method='delete'
2023-10-26 14:42:48,075 | Lambda function integration not found in Swagger document at path='/server/api/v9/{proxy+}' method='post'
2023-10-26 14:42:48,075 | Lambda function integration not found in Swagger document at path='/server/api/v9/{proxy+}' method='get'
2023-10-26 14:42:48,075 | Found '0' APIs in resource 'AwsApiGateway'
2023-10-26 14:42:48,076 | Found '0' authorizers in resource 'AwsApiGateway'
2023-10-26 14:42:48,076 | Found '3' API Events in Serverless function with name 'LambdaFunction'
2023-10-26 14:42:48,076 | Found '2' API Events in Serverless function with name 'ProxyFunction'
2023-10-26 14:42:48,077 | Removed duplicates from '5' Explicit APIs and '0' Implicit APIs to produce '5' APIs
2023-10-26 14:42:48,077 | 4 APIs found in the template
2023-10-26 14:42:48,079 | Mounting ProxyFunction at http://127.0.0.1:3000/server/api/v9/{proxy+} [GET, POST, OPTIONS]
2023-10-26 14:42:48,079 | Mounting LambdaFunction at http://127.0.0.1:3000/{proxy+}/api/v10/notifications/{creationDate} [DELETE, OPTIONS]
2023-10-26 14:42:48,080 | Mounting LambdaFunction at http://127.0.0.1:3000/{proxy+}/api/v10/notifications [GET, OPTIONS]
2023-10-26 14:42:48,080 | Mounting LambdaFunction at http://127.0.0.1:3000/{proxy+}/api/v10/notifications/{creationDate}/read [PUT, OPTIONS]
2023-10-26 14:42:48,080 | You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be
reflected instantly/automatically. If you used sam build before running local commands, you will need to re-run sam build for the changes to be picked up. You only need to restart SAM CLI
if you update your AWS SAM template
2023-10-26 14:42:48,081 | Localhost server is starting up. Multi-threading = True
2023-10-26 14:42:48 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on http://127.0.0.1:3000
2023-10-26 14:42:48 Press CTRL+C to quit
2023-10-26 14:42:52 127.0.0.1 - - [26/Oct/2023 14:42:52] "OPTIONS /server/api/v9/assets/getDataTable HTTP/1.1" 200 -
2023-10-26 14:42:52,275 | Constructed Event Version 2.0 to invoke Lambda. Event: {'version': '2.0', 'routeKey': 'POST /server/api/v9/{proxy+}', 'rawPath': '/server/api/v9/assets/getDataTable',
'rawQueryString': '', 'cookies': [], 'headers': {'Host': 'localhost:3000', 'Connection': 'keep-alive', 'Content-Length': '856', 'Sec-Ch-Ua': '"Chromium";v="118", "Google Chrome";v="118",
"Not=A?Brand";v="99"', 'Accept': 'application/json', 'Content-Type': 'application/json', 'Sec-Ch-Ua-Mobile': '?0', 'Authorization':
'eyJraWQiOiJENnA5U2YwVFpGajFiUEVJeXlZM1hrR1JkZHB2c0IrdU9oTldcL0FcL3pmTEU9IiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJiYWQwNWZjYy1jMWVlLTRiZDYtYTY1OS0yNmVmZjFjNDZlMzMiLCJjb2duaXRvOmdyb3VwcyI6WyJST0x
FX1BST1RFVVNfQURNSU4iLCJST0xFX1BST1RFVVNfRURJVE9SIiwiUk9MRV9QUk9URVVTX0RFUExPWUVSX0RFViIsIlJPTEVfUFJPVEVVU19ERVBMT1lFUl9VQVQiLCJST0xFX1BST1RFVVNfVklFV0VSIl0sImVtYWlsX3ZlcmlmaWVkIjp0cnVlLC
Jpc3MiOiJodHRwczpcL1wvY29nbml0by1pZHAuZXUtd2VzdC0xLmFtYXpvbmF3cy5jb21cL2V1LXdlc3QtMV9mZVlpUFk1YlEiLCJjb2duaXRvOnVzZXJuYW1lIjoibWFya3AiLCJhdWQiOiIzMWtzazdpc2xkcmo1MGduOGtmZWhnYXU4aSIsImV2Z
W50X2lkIjoiNjI0MDkxOGQtOThkMi00MGJhLTljZTQtMWE3ODBkYzQ1ODBlIiwidG9rZW5fdXNlIjoiaWQiLCJhdXRoX3RpbWUiOjE2OTgzMjc3NTYsIm5hbWUiOiJNYXJrIFBpZXJjZSIsImN1c3RvbTp0ZW5hbnQiOiJwZGlfbWFya3AiLCJleHAi
OjE2OTgzMzEzNTYsImlhdCI6MTY5ODMyNzc1NiwiZW1haWwiOiJtYXJrLnBpZXJjZUBjZGwuY28udWsifQ.pdQZqlNh9sWxbZx1KkfpGRrIlndhmxaRlIbbv8XmQskpwd7LOsWdJF7onI8IzmKwk3glk6WKL87ifZa3c8K88ER5MREhOD3baWyWY6z7
OOmp_o2bn3IWxJiWrRFpVkrISA3qhEh9EauTwE056tKbnwIgrjqOxDd0aAd96Nh8fgwgPeFytCbtQaqqX-EK_8QAotfZpb2iluF0bSNBIql6F0KLe9Nv-xRDbt2YyfpfoajP7DFt0HiMPb2cHQH1zDAysPbawlL8LWgtTTXMr4h8VxKQgLb99lGZJUL
dGCbNyZKZDroi3-_6LtEtvd5FwaGZ_zg5oYSG1nT34rLQNHESig', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0
Safari/537.36', 'Sec-Ch-Ua-Platform': '"macOS"', 'Origin': 'http://localhost:4200', 'Sec-Fetch-Site': 'same-site', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Dest': 'empty', 'Referer':
'http://localhost:4200/', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8', 'X-Forwarded-Proto': 'http', 'X-Forwarded-Port': '3000'},
'requestContext': {'accountId': '123456789012', 'apiId': '1234567890', 'http': {'method': 'POST', 'path': '/server/api/v9/assets/getDataTable', 'protocol': 'HTTP/1.1', 'sourceIp':
'127.0.0.1', 'userAgent': 'Custom User Agent String'}, 'requestId': 'c878a93e-9b6a-4e4d-924c-07e87906fca1', 'routeKey': 'POST /server/api/v9/{proxy+}', 'stage': 'dev', 'time':
'26/Oct/2023:13:42:42 +0000', 'timeEpoch': 1698327762, 'domainName': 'localhost', 'domainPrefix': 'localhost'}, 'body':
'{"assetTypeDto":null,"dataTablesInput":{"draw":1,"columns":[{"data":"id","name":"","searchable":true,"orderable":false,"search":{"value":"","regex":false}},{"data":"assetType","name":"",
"searchable":true,"orderable":true,"search":{"value":"","regex":false}},{"data":"fileName","name":"","searchable":true,"orderable":true,"search":{"value":"","regex":false}},{"data":"uploa
dedDateAndTime","name":"","searchable":true,"orderable":true,"search":{"value":"","regex":false}},{"data":"status","name":"","searchable":true,"orderable":false,"search":{"value":"","rege
x":false}},{"data":"action","name":"","searchable":true,"orderable":false,"search":{"value":"","regex":false}}],"order":[{"column":3,"dir":"desc"}],"start":0,"length":"TEN","search":{"val
ue":"","regex":false}},"mostRecentByName":false,"status":"ACTIVE","matchToAssetName":false,"dateBefore":null}', 'pathParameters': {'proxy': 'assets/getDataTable'}, 'stageVariables': None,
'isBase64Encoded': False}
2023-10-26 14:42:52,293 | Constructed Event Version 2.0 to invoke Lambda. Event: {'version': '2.0', 'routeKey': 'POST /server/api/v9/{proxy+}', 'rawPath': '/server/api/v9/assets/getDataTable',
'rawQueryString': '', 'cookies': [], 'headers': {'Host': 'localhost:3000', 'Connection': 'keep-alive', 'Content-Length': '762', 'Sec-Ch-Ua': '"Chromium";v="118", "Google Chrome";v="118",
"Not=A?Brand";v="99"', 'Accept': 'application/json', 'Content-Type': 'application/json', 'Sec-Ch-Ua-Mobile': '?0', 'Authorization':
'eyJraWQiOiJENnA5U2YwVFpGajFiUEVJeXlZM1hrR1JkZHB2c0IrdU9oTldcL0FcL3pmTEU9IiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJiYWQwNWZjYy1jMWVlLTRiZDYtYTY1OS0yNmVmZjFjNDZlMzMiLCJjb2duaXRvOmdyb3VwcyI6WyJST0x
FX1BST1RFVVNfQURNSU4iLCJST0xFX1BST1RFVVNfRURJVE9SIiwiUk9MRV9QUk9URVVTX0RFUExPWUVSX0RFViIsIlJPTEVfUFJPVEVVU19ERVBMT1lFUl9VQVQiLCJST0xFX1BST1RFVVNfVklFV0VSIl0sImVtYWlsX3ZlcmlmaWVkIjp0cnVlLC
Jpc3MiOiJodHRwczpcL1wvY29nbml0by1pZHAuZXUtd2VzdC0xLmFtYXpvbmF3cy5jb21cL2V1LXdlc3QtMV9mZVlpUFk1YlEiLCJjb2duaXRvOnVzZXJuYW1lIjoibWFya3AiLCJhdWQiOiIzMWtzazdpc2xkcmo1MGduOGtmZWhnYXU4aSIsImV2Z
W50X2lkIjoiNjI0MDkxOGQtOThkMi00MGJhLTljZTQtMWE3ODBkYzQ1ODBlIiwidG9rZW5fdXNlIjoiaWQiLCJhdXRoX3RpbWUiOjE2OTgzMjc3NTYsIm5hbWUiOiJNYXJrIFBpZXJjZSIsImN1c3RvbTp0ZW5hbnQiOiJwZGlfbWFya3AiLCJleHAi
OjE2OTgzMzEzNTYsImlhdCI6MTY5ODMyNzc1NiwiZW1haWwiOiJtYXJrLnBpZXJjZUBjZGwuY28udWsifQ.pdQZqlNh9sWxbZx1KkfpGRrIlndhmxaRlIbbv8XmQskpwd7LOsWdJF7onI8IzmKwk3glk6WKL87ifZa3c8K88ER5MREhOD3baWyWY6z7
OOmp_o2bn3IWxJiWrRFpVkrISA3qhEh9EauTwE056tKbnwIgrjqOxDd0aAd96Nh8fgwgPeFytCbtQaqqX-EK_8QAotfZpb2iluF0bSNBIql6F0KLe9Nv-xRDbt2YyfpfoajP7DFt0HiMPb2cHQH1zDAysPbawlL8LWgtTTXMr4h8VxKQgLb99lGZJUL
dGCbNyZKZDroi3-_6LtEtvd5FwaGZ_zg5oYSG1nT34rLQNHESig', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0
Safari/537.36', 'Sec-Ch-Ua-Platform': '"macOS"', 'Origin': 'http://localhost:4200', 'Sec-Fetch-Site': 'same-site', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Dest': 'empty', 'Referer':
'http://localhost:4200/', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8', 'X-Forwarded-Proto': 'http', 'X-Forwarded-Port': '3000'},
'requestContext': {'accountId': '123456789012', 'apiId': '1234567890', 'http': {'method': 'POST', 'path': '/server/api/v9/assets/getDataTable', 'protocol': 'HTTP/1.1', 'sourceIp':
'127.0.0.1', 'userAgent': 'Custom User Agent String'}, 'requestId': 'c878a93e-9b6a-4e4d-924c-07e87906fca1', 'routeKey': 'POST /server/api/v9/{proxy+}', 'stage': 'dev', 'time':
'26/Oct/2023:13:42:42 +0000', 'timeEpoch': 1698327762, 'domainName': 'localhost', 'domainPrefix': 'localhost'}, 'body':
'{"assetTypeDto":null,"dataTablesInput":{"draw":1,"columns":[{"data":"assetType","name":"","searchable":true,"orderable":true,"search":{"value":"","regex":false}},{"data":"fileName","name
":"","searchable":true,"orderable":true,"search":{"value":"","regex":false}},{"data":"uploadedDateAndTime","name":"","searchable":true,"orderable":true,"search":{"value":"","regex":false}
},{"data":"status","name":"","searchable":true,"orderable":false,"search":{"value":"","regex":false}},{"data":"action","name":"","searchable":true,"orderable":false,"search":{"value":"","
regex":false}}],"order":[{"column":2,"dir":"desc"}],"start":0,"length":"TEN","search":{"value":"","regex":false}},"mostRecentByName":false,"status":"ARCHIVED","matchToAssetName":false,"da
teBefore":null}', 'pathParameters': {'proxy': 'assets/getDataTable'}, 'stageVariables': None, 'isBase64Encoded': False}
2023-10-26 14:42:52,295 | Found one Lambda function with name 'ProxyFunction'
2023-10-26 14:42:52,309 | Found one Lambda function with name 'ProxyFunction'
2023-10-26 14:42:52,309 | Invoking lambda_function.lambda_handler (python3.11)
2023-10-26 14:42:52,310 | Invoking lambda_function.lambda_handler (python3.11)
2023-10-26 14:42:52,310 | Resolving code path. Cwd=/Users/mark/git/notifications-dao-lambda/.aws-sam/build,
CodeUri=/Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction
2023-10-26 14:42:52,311 | Resolving code path. Cwd=/Users/mark/git/notifications-dao-lambda/.aws-sam/build,
CodeUri=/Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction
2023-10-26 14:42:52,312 | Resolved absolute path to code is /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction
2023-10-26 14:42:52,312 | Lambda Function 'ProxyFunction' definition has been changed in the stack template, terminate the created warm container.
2023-10-26 14:42:52,311 | Resolved absolute path to code is /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction
2023-10-26 14:42:52,336 | Reuse the created warm container for Lambda function 'ProxyFunction'
2023-10-26 14:42:52,341 | Lambda function 'ProxyFunction' is already running
2023-10-26 14:42:52,343 | Starting a timer for 60 seconds for function 'ProxyFunction'
2023-10-26 14:42:52,466 | unwatch resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction
2023-10-26 14:42:52,477 | Unschedule Observer for resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction with recursive
True
2023-10-26 14:42:52,483 | unwatch resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction's parent
/Users/mark/git/notifications-dao-lambda/.aws-sam/build
Exception in thread Thread-10 (wait_for_logs):
Traceback (most recent call last):
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/docker/api/client.py", line 268, in _raise_for_status
response.raise_for_status()
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/containers/7023954718ffea56757b8028d457065a7e771440af272db501cdcc069340acc8/json
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python@3.12/3.12.0/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 1052, in _bootstrap_inner
self.run()
File "/opt/homebrew/Cellar/python@3.12/3.12.0/Frameworks/Python.framework/Versions/3.12/lib/python3.12/threading.py", line 989, in run
self._target(*self._args, **self._kwargs)
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/samcli/local/docker/container.py", line 388, in wait_for_logs
logs_itr = real_container.attach(stream=True, logs=True, demux=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/docker/models/containers.py", line 96, in attach
return self.client.api.attach(self.id, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/docker/api/container.py", line 61, in attach
response, stream, self._check_is_tty(container), demux=demux)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/docker/api/client.py", line 463, in _check_is_tty
cont = self.inspect_container(container)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/docker/api/container.py", line 784, in inspect_container
return self._result(
^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/docker/api/client.py", line 274, in _result
self._raise_for_status(response)
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e) from e
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.99.0_1/libexec/lib/python3.12/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.35/containers/7023954718ffea56757b8028d457065a7e771440af272db501cdcc069340acc8/json: Not Found ("No such container: 7023954718ffea56757b8028d457065a7e771440af272db501cdcc069340acc8")
2023-10-26 14:42:53,099 | No response from invoke container for ProxyFunction
2023-10-26 14:42:53,099 | Invalid lambda response received: Lambda response must be valid json
2023-10-26 14:42:53 127.0.0.1 - - [26/Oct/2023 14:42:53] "POST /server/api/v9/assets/getDataTable HTTP/1.1" 502 -
2023-10-26 14:42:53,113 | watch resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction
2023-10-26 14:42:53,113 | Create Observer for resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction with recursive True
2023-10-26 14:42:53,116 | watch resource /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction's parent
/Users/mark/git/notifications-dao-lambda/.aws-sam/build
2023-10-26 14:42:53,116 | Code /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction is not a zip/jar file
2023-10-26 14:42:53,138 | Using local image: public.ecr.aws/lambda/python:3.11-rapid-x86_64.
2023-10-26 14:42:53,138 | Mounting /Users/mark/git/notifications-dao-lambda/.aws-sam/build/ProxyFunction as /var/task:ro,delegated, inside
runtime container
2023-10-26 14:42:53,409 | Starting a timer for 60 seconds for function 'ProxyFunction'
START RequestId: 1a74d418-1724-4533-a679-32059afbaf75 Version: $LATEST
END RequestId: 1a74d418-1724-4533-a679-32059afbaf75
REPORT RequestId: 1a74d418-1724-4533-a679-32059afbaf75 Init Duration: 1.23 ms Duration: 1581.79 ms Billed Duration: 1582 ms Memory Size: 512 MB Max Memory Used: 512 MB
2023-10-26 14:42:56,059 | Unable to find Click Context for getting session_id.
2023-10-26 14:42:56 127.0.0.1 - - [26/Oct/2023 14:42:56] "POST /server/api/v9/assets/getDataTable HTTP/1.1" 200 -
Expected result:
I’d expect both calls to return 200.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
{
"version": "1.99.0",
"system": {
"python": "3.12.0",
"os": "macOS-13.6-arm64-arm-64bit"
},
"additional_dependencies": {
"docker_engine": "24.0.6",
"aws_cdk": "Not available",
"terraform": "1.3.7"
},
"available_beta_feature_env_vars": [
"SAM_CLI_BETA_FEATURES",
"SAM_CLI_BETA_BUILD_PERFORMANCE",
"SAM_CLI_BETA_TERRAFORM_SUPPORT",
"SAM_CLI_BETA_RUST_CARGO_LAMBDA"
]
}
About this issue
- Original URL
- State: closed
- Created 8 months ago
- Reactions: 6
- Comments: 19 (4 by maintainers)
Commits related to this issue
- Upgrade SAM client to 1.109.0 to avoid https://github.com/aws/aws-sam-cli/issues/6157. — committed to jg210/spring-experiments by jg210 5 months ago
Patch is released in v1.109.0. Closing
Thanks for providing all the updates here.
When you run your lambda function in your account, concurrent request spins up a new container if there is no free instance available. See more details here: https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html
However SAM CLI doesn’t support spinning up a new container per request. All requests are handled in single container. For that reason you might see the error below;
In order to fix this issue, we’ve raised a PR which will only allow single execution for a particular function, which should prevent this issue happening.
Thanks.
Reran with that option and it works a little better… worked first try, and then I get the 502 error.