datadog-lambda-js: TypeError - handler is not a function

Sorry for the late response, we noticed that certain Node14.x Lambda functions deployed in us-east-1 were observing intermittent errors (“handler is not a function”) between 7:20PM October 13th EDT and 8:00AM October 14th EDT. We are actively investigating the root cause with the AWS Lambda team, and will share the results with the community as soon as possible. Please subscribe this issue or contact Datadog support (support@datadoghq.com) for the investigation results and additional technical details.

I opened a support case with AWS and the initial response says the internal team “are actively investigating the increased errors when using Node.JS 14 in the US-EAST-1 Region.” so this may be an AWS issue. I recreated the lambda with the same code and version of the datadog layer and haven’t see any issues for last hour or so.

Support got back to us.

Furthermore,  Lambda team performed rollback to fix this issue but this may take some time to fully propagate. However, I am actively monitoring internal ticket and I will keep you updated on this case. 

Ok. Found a workaround for now. the cdk construct is changing the handlers by injecting /opt/nodejs/node_modules/datadog-lambda-js/handler.handler and replace it with DD_LAMBDA_HANDLER value (the original handler).

by changing the lambda handler to the local file path + function solve the issue (the value you have in DD_LAMBDA_HANDLER).

we don’t see more error the error no more

If it helps, my team has experienced the very same issue in ca-central-1 as well

@Kamitenshi have you tried rebuild your image using the latest of public.ecr.aws/lambda/nodejs:14 and redeploy your function?

Since the issue has been fixed, I’m going to close this thread. We are writing a postmortem next week, please contact support@datadoghq.com for a copy as you desire. Again, very sorry for everyone’s loss and inconvenience, we are doing everything we can to prevent similar issues from happening again! Thanks for your understanding and help! 🙇

LATEST UPDATES [12PM 10/15/21 EDT]: We no longer observe issues in ca-central-1 or functions using public.ecr.aws/lambda/nodejs:14. No customer is expected to experience this issue at this point time. If that’s not the case, please let us know.

@Kamitenshi and for whoever using container images, try use an older version of the base image by AWS public.ecr.aws/lambda/nodejs:14.2021.10.07.14-x86_64. According to our tests, any version after this one still experience the issue.

@alexis-caylent I don’t think aws:states:opt-out is related. Are you currently experiencing the same error in ca-central-1?

Same issue has been happening for us with a simple lambda for the last 12 hours - no datadog or layers. Stumbled upon this searching the internet for similar cases.

Same for us, we had Node 14.x and the layer 49. I switched to 64 just to make sure it wasn’t the layer version, but it quickly became evident that the version didn’t matter. Since there was no code change on our side before and after the errors started showing up, we believe it’s infra related.

The layer is still there, and you can’t modify it AFAIK so this must be AWS.

We opened a ticket with AWS. I think they’re are doing some changes to Lambda and this might be related

You are receiving this message because we identified that your account created or updated or invoked Lambda functions on or after July 1, 2021.

AWS Lambda is extending the capability to track the current state of a function through its lifecycle to all functions [1]. With this change, you may need to update your CLI and/or SDK-based automation workflows around creating and updating functions by adding a check that the function became active before performing additional actions that operate on the function.

Previously, states have been used in two primary use-cases. Firstly, to move the blocking setup of VPC resources out of the path of function invocation. Secondly, to allow the Lambda service to optimize new or updated container images for container-image based functions, also before invocation. By moving this additional work out of the path of the invocation, customers see lower latency and better consistency in their function performance.
Infrastructure as code and deployment automation tools such as AWS CloudFormation, AWS Serverless Application Model (SAM), AWS Cloud Development Kit (CDK), Serverless Framework,Hashicorp Terraform, AWS Chalice and Cloud Custodian already support states.

If you are using these tools and are already on the latest Software Development Kit (SDK), you do not need to take any action, and can stop reading now. If you are using an earlier SDK version, please update to the latest one.

We are rolling out this change over multi-phase periods to allow you to update existing tooling for deploying and managing Lambda functions. You can delay this change for your functions until December 5, 2021. Starting December 6, 2021, the delay mechanism expires and all customers see the Lambda states lifecycle applied during function create or update. Read this blog post [2] to learn about this change, timelines for different phases and a reference example on how to check your function state.

We’re experiencing this too and can confirm the workaround that @lielran posted above worked for us.

So is the problem here that the DD layer is generating the DD_LAMBDA_HANDLER environment variable using the relative path for the layer instead of the literal path for the bundled lambda? Did AWS indicate that this is something they actually intend to fix, or is the update they made going to be the new method for bundling to which we need to adapt?

Same for us, we had Node 14.x and the layer 49. I switched to 64 just to make sure it wasn’t the layer version, but it quickly became evident that the version didn’t matter. Since there was no code change on our side before and after the errors started showing up, we believe it’s infra related.

The layer is still there, and you can’t modify it AFAIK so this must be AWS.

We opened a ticket with AWS. I think they’re are doing some changes to Lambda and this might be related

You are receiving this message because we identified that your account created or updated or invoked Lambda functions on or after July 1, 2021.

AWS Lambda is extending the capability to track the current state of a function through its lifecycle to all functions [1]. With this change, you may need to update your CLI and/or SDK-based automation workflows around creating and updating functions by adding a check that the function became active before performing additional actions that operate on the function.

Previously, states have been used in two primary use-cases. Firstly, to move the blocking setup of VPC resources out of the path of function invocation. Secondly, to allow the Lambda service to optimize new or updated container images for container-image based functions, also before invocation. By moving this additional work out of the path of the invocation, customers see lower latency and better consistency in their function performance.
Infrastructure as code and deployment automation tools such as AWS CloudFormation, AWS Serverless Application Model (SAM), AWS Cloud Development Kit (CDK), Serverless Framework,Hashicorp Terraform, AWS Chalice and Cloud Custodian already support states.

If you are using these tools and are already on the latest Software Development Kit (SDK), you do not need to take any action, and can stop reading now. If you are using an earlier SDK version, please update to the latest one.

We are rolling out this change over multi-phase periods to allow you to update existing tooling for deploying and managing Lambda functions. You can delay this change for your functions until December 5, 2021. Starting December 6, 2021, the delay mechanism expires and all customers see the Lambda states lifecycle applied during function create or update. Read this blog post [2] to learn about this change, timelines for different phases and a reference example on how to check your function state.

Amazing @lielran that really seems to work - I think it’s the only thing I haven’t tried, I was going to just rip off datadog complitely