powertools-lambda-python: SAM CLI Bug: APIGatewayRestResolver and APIGatewayHttpResolver behave differently in local environment

Expected Behaviour

APIGatewayRestResolver and APIGatewayHttpResolver should behave similar when StageName: prod is defined at in SAM at template.yaml.

    Api:
        Type: AWS::Serverless::Api
        Properties:
            StageName: prod

    HttpApi:
        Type: AWS::Serverless::HttpApi
        Properties:
            StageName: prod

    HelloWorldFunction:
        Type: AWS::Serverless::Function
        Properties:
            Events:
                HelloPath:
                    Type: HttpApi
                    Properties:
                        RestApiId: !Ref Api
                        ApiId: !Ref HttpApi
                        Path: /hello
                        Method: GET
@app.get("/hello")
@tracer.capture_method
def hello():

In both cases when running sam local start-api both API gateway instances should try to match /hello.

Current Behaviour

If StageName: prod is defined the APIGatewayProxyEventV2 (used by APIGatewayHttpResolver) will remove len(StageName) + 1 from the request path.

Hence, for this code, APIGatewayRestResolver will try to match /hello and APIGatewayHttpResolver will try to matcho (/hello - /hell = o).

This is due to: https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/aws_lambda_powertools/utilities/data_classes/api_gateway_proxy_event.py#L251-L255

If StageName is not defined (stage name has the default value) the APIGatewayProxyEventV2 will not change the request path making both APIGatewayRestResolver and APIGatewayHttpResolver match /hello.

Faking the stage name is a possible workaround to force APIGatewayHttpResolver to match /hello.

def lambda_handler(event: dict, context: LambdaContext) -> dict:
    event['requestContext']['stage'] = "$default"
    return app.resolve(event, context)

Code snippet

See: https://github.com/tinti/sam-api-httpapi

Possible Solution

I believe something has to be changed at: https://github.com/aws-powertools/powertools-lambda-python/blob/0523ff64606514ea3e59c07c8c69c83d751f61fa/aws_lambda_powertools/utilities/data_classes/api_gateway_proxy_event.py#L251-L255

If it is not a breaking change I would not remove the stage name at all.

 def path(self) -> str:
     return self.raw_path 

If it is a breaking change maybe create an option to control whether or not APIGatewayRestResolver and APIGatewayHttpResolver should remove the stage name.

Steps to Reproduce

Steps

Create a SAM template application from 1 - AWS Quick Start Templates then 3 - Hello World Example with Powertools for AWS Lambda then python3.10. See: https://github.com/tinti/sam-api-httpapi

Add explicit Api and HttpApi resources with StageName: prod property to the template.

Reference the Api or HttpApi resource in the Lambda’s event source.

Start local api with sam local start-api

Test with and without StageName: prod doing curl -v -X GET --silent localhost:3000/hello

Results

Type Stage Expected path Current path
Api $default /hello /hello
Api prod /hello /hello
HttpApi $default /hello /hello
HttpApi prod /hello /o
HttpApi + workaround prod /hello /hello

Workaround

def lambda_handler(event: dict, context: LambdaContext) -> dict:
    event['requestContext']['stage'] = "$default"
    return app.resolve(event, context)

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.10

Packaging format used

Lambda Layers

Debugging logs

$ # HttpApi with StageName=prod and @app.get('/hello')
$ sam local start-api 
Initializing the lambda functions containers.                                                                                                          
Local image is up-to-date                                                                                                                              
Using local image: public.ecr.aws/lambda/python:3.10-rapid-x86_64.                                                                                     
                                                                                                                                                       
Mounting <REDACTED>/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container                 
Containers Initialization is done.                                                                                                                     
Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]                                                                                       
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes 
will be reflected instantly/automatically. If you used sam build before running local commands, you will need to re-run sam build for the changes to be
picked up. You only need to restart SAM CLI if you update your AWS SAM template                                                                        
2023-07-13 06:05:28 WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:3000
2023-07-13 06:05:28 Press CTRL+C to quit
Invoking app.lambda_handler (python3.10)                                                                                                               
Reuse the created warm container for Lambda function 'HelloWorldFunction'                                                                              
Lambda function 'HelloWorldFunction' is already running                                                                                                
START RequestId: b1a452d8-a8ea-4d4c-8544-19bcd2c8df03 Version: $LATEST
/var/task/aws_lambda_powertools/package_logger.py:20: UserWarning: POWERTOOLS_DEBUG environment variable is enabled. Setting logging level to DEBUG.
  if powertools_debug_is_set():
2023-07-13 09:05:32,222 aws_lambda_powertools.tracing.tracer [DEBUG] Verifying whether Tracing has been disabled
[DEBUG] 2023-07-13T09:05:32.222Z                Verifying whether Tracing has been disabled
2023-07-13 09:05:32,518 aws_lambda_powertools.logging.logger [DEBUG] Adding filter in root logger to suppress child logger records to bubble up
[DEBUG] 2023-07-13T09:05:32.518Z                Adding filter in root logger to suppress child logger records to bubble up
2023-07-13 09:05:32,518 aws_lambda_powertools.logging.logger [DEBUG] Marking logger PowertoolsHelloWorld as preconfigured
[DEBUG] 2023-07-13T09:05:32.518Z                Marking logger PowertoolsHelloWorld as preconfigured
2023-07-13 09:05:32,518 aws_lambda_powertools.event_handler.api_gateway [DEBUG] Adding route using rule /hello and methods: GET
[DEBUG] 2023-07-13T09:05:32.518Z                Adding route using rule /hello and methods: GET
2023-07-13 09:05:32,519 aws_lambda_powertools.logging.logger [DEBUG] Decorator called with parameters
[DEBUG] 2023-07-13T09:05:32.519Z                Decorator called with parameters
2023-07-13 09:05:32,519 aws_lambda_powertools.metrics.base [DEBUG] Decorator called with parameters
[DEBUG] 2023-07-13T09:05:32.519Z                Decorator called with parameters
[WARNING]       2023-07-13T09:05:32.520Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    Subsegment ## lambda_handler discarded due to Lambda worker still initializing
2023-07-13 09:05:32,520 aws_lambda_powertools.tracing.tracer [DEBUG] Calling lambda handler
[DEBUG] 2023-07-13T09:05:32.520Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    Calling lambda handler
{"body":"","cookies":[],"headers":{"Accept":"*/*","Host":"localhost:3000","User-Agent":"curl/8.1.2","X-Forwarded-Port":"3000","X-Forwarded-Proto":"http"},"isBase64Encoded":false,"pathParameters":{},"rawPath":"/hello","rawQueryString":"","requestContext":{"accountId":"123456789012","apiId":"1234567890","domainName":"localhost","domainPrefix":"localhost","http":{"method":"GET","path":"/hello","protocol":"HTTP/1.1","sourceIp":"127.0.0.1","userAgent":"Custom User Agent String"},"requestId":"bb985f80-75a2-47a3-afd0-0467ac443ac9","routeKey":"GET /hello","stage":"prod","time":"13/Jul/2023:09:05:18 +0000","timeEpoch":1689239118},"routeKey":"GET /hello","stageVariables":null,"version":"2.0"}
2023-07-13 09:05:32,520 aws_lambda_powertools.event_handler.api_gateway [DEBUG] Converting event to API Gateway HTTP API contract
[DEBUG] 2023-07-13T09:05:32.520Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    Converting event to API Gateway HTTP API contract
2023-07-13 09:05:32,520 aws_lambda_powertools.event_handler.api_gateway [DEBUG] No match found for path o and method GET
[DEBUG] 2023-07-13T09:05:32.520Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    No match found for path o and method GET
2023-07-13 09:05:32,520 aws_lambda_powertools.metrics.base [DEBUG] Adding cold start metric and function_name dimension
[DEBUG] 2023-07-13T09:05:32.520Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    Adding cold start metric and function_name dimension
2023-07-13 09:05:32,520 aws_lambda_powertools.metrics.base [DEBUG] Adding metric: ColdStart with defaultdict(<class 'list'>, {'Unit': 'Count', 'StorageResolution': 60, 'Value': [1.0]})
[DEBUG] 2023-07-13T09:05:32.520Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    Adding metric: ColdStart with defaultdict(<class 'list'>, {'Unit': 'Count', 'StorageResolution': 60, 'Value': [1.0]})
2023-07-13 09:05:32,520 aws_lambda_powertools.metrics.base [DEBUG] Adding dimension: function_name:HelloWorldFunction
[DEBUG] 2023-07-13T09:05:32.520Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    Adding dimension: function_name:HelloWorldFunction
2023-07-13 09:05:32,520 aws_lambda_powertools.metrics.base [DEBUG] Adding dimension: service:PowertoolsHelloWorld
[DEBUG] 2023-07-13T09:05:32.520Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    Adding dimension: service:PowertoolsHelloWorld
2023-07-13 09:05:32,520 aws_lambda_powertools.metrics.base [DEBUG] {'details': 'Serializing metrics', 'metrics': {'ColdStart': defaultdict(<class 'list'>, {'Unit': 'Count', 'StorageResolution': 60, 'Value': [1.0]})}, 'dimensions': {'function_name': 'HelloWorldFunction', 'service': 'PowertoolsHelloWorld'}}
[DEBUG] 2023-07-13T09:05:32.520Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    {'details': 'Serializing metrics', 'metrics': {'ColdStart': defaultdict(<class 'list'>, {'Unit': 'Count', 'StorageResolution': 60, 'Value': [1.0]})}, 'dimensions': {'function_name': 'HelloWorldFunction', 'service': 'PowertoolsHelloWorld'}}
{"_aws":{"Timestamp":1689239132521,"CloudWatchMetrics":[{"Namespace":"Powertools","Dimensions":[["function_name","service"]],"Metrics":[{"Name":"ColdStart","Unit":"Count"}]}]},"function_name":"HelloWorldFunction","service":"PowertoolsHelloWorld","ColdStart":[1.0]}
/var/task/aws_lambda_powertools/metrics/base.py:418: UserWarning: No application metrics to publish. The cold-start metric may be published if enabled. If application metrics should never be empty, consider using 'raise_on_empty_metrics'
  self.flush_metrics(raise_on_empty_metrics=raise_on_empty_metrics)
2023-07-13 09:05:32,521 aws_lambda_powertools.tracing.tracer [DEBUG] Received lambda handler response successfully
[DEBUG] 2023-07-13T09:05:32.521Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    Received lambda handler response successfully
2023-07-13 09:05:32,521 aws_lambda_powertools.tracing.tracer [DEBUG] Annotating cold start
[DEBUG] 2023-07-13T09:05:32.521Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    Annotating cold start
[WARNING]       2023-07-13T09:05:32.521Z        b1a452d8-a8ea-4d4c-8544-19bcd2c8df03    No subsegment to end.
END RequestId: b1a452d8-a8ea-4d4c-8544-19bcd2c8df03
REPORT RequestId: b1a452d8-a8ea-4d4c-8544-19bcd2c8df03  Init Duration: 0.05 ms  Duration: 434.74 ms     Billed Duration: 435 ms Memory Size: 128 MB   Max Memory Used: 128 MB
2023-07-13 06:05:32 127.0.0.1 - - [13/Jul/2023 06:05:32] "GET /hello HTTP/1.1" 404 -

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 16 (14 by maintainers)

Most upvoted comments

Hey everyone! Thank you for patience while the SAM CLI team investigated this issue. They confirmed that fixing it would lead to a breaking change, so they’re unable to resolve it.

According to the SAM CLI team, given its complexities for a backwards compatible fix, they’ve marked as a feature request to think of an alternative way to solve this. From today onwards, I’d suggest anyone reading this to subscribe to the SAM CLI issue and add thumbs-up (👍) to help SAM CLI team prioritize it.

Closing this issue.

Hello @tinti! I was able to reproduce the error locally, but I couldn’t when deploying to AWS, which means this is not a bug in a production environment, it’s a bug in the way of emulating this locally.

However, the customer experience is extremely important to us and we know how important the feedback loop is in the software/dev process. I don’t know if it’s really a bug in Powertools or if we need to investigate with the SAM CLI team, but I’ll start a deeper analysis and bring feedback here as soon as I have something relevant.

Thanks for taking the time to create a project to test it out, it’s shortened the path by miles! 🚀

Hello @osjerick! I’m reopening this issue to monitor it and see what we can do to get it done.

The next steps will be: 1 - I’m going to create another issue in the SAM repository with more details, so they can analyze it and verify if it really is a bug and workarounds/solutions. 2 - We will keep in touch with the SAM CLI team.

I’ll update this issue when I have some news.

Thank you for bringing this issue to our attention.

I’ve opened an issue on the upstream project here https://github.com/aws/aws-sam-cli/issues/5579 and I ask you to track it. For the moment we’ll close this issue since the problem doesn’t appear to be caused by Powertools. But please feel free to re-open if you have any additional consideration!

Thank you so much for the well written bug report, it helped tremendously!

Not a problem at runtime.

I notice that the rawPath and path are not the same when comparing runtime and local.

IMG-20230713-WA0000

hi @heitorlessa, to be honest I just tested locally. Let me try at runtime and attach the results.

Thanks for the fast reply.

hey @tinti QQ - does this happen at runtime or only when running locally? Reason being is that we’ve seen a different behaviour during emulation and we don’t always account for those (given the plethora of options).

Thank you for the superb bug report - we’ll work on reproducing it now