apm-agent-python: ElasticAPM is hanging lambda function after processing all events in an SQS queue
I believe ElasticAPM will hang the lambda function when no more events exist in the SQS queue.
To Reproduce
- Setup a lambda function to process events from an SQS queue
- Send events to an SQS queue
- Set
ELASTIC_APM_LOG_LEVEL: debug
Environment (please complete the following information)
- OS: debian:bullseye-slim (python:3.9-slim container image) x86
- Python version: 3.9
- Framework and version [e.g. Django 2.1]:
- APM Server version: 8.4.2
- Agent version: 6.13.2
Additional context
Cloudwatch logs (can dive deeper if needed) after execution is completed
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (16 by maintainers)
Commits related to this issue
- Updated docs with a note about aws lambda configuration, see GH-1704 for more info. — committed to brett-fitz/apm-agent-python by deleted user a year ago
- Config docs update with note for AWS Lambda (#1720) * Updated docs with a note about aws lambda configuration, see GH-1704 for more info. * Update docs/configuration.asciidoc LGTM Co-authore... — committed to elastic/apm-agent-python by deleted user a year ago
I love it when it’s easy! Just frustrated I missed the config issue originally. 😅 Thanks for opening a PR!
@brett-fitz Thanks for sharing the configuration and the debug logs.
I can see that you are setting
ELASTIC_APM_SERVER_URLto the APM Server’s URL which might be causing the current issue. APM-Agents are supposed to proxy their calls to APM-Server through the lambda extension but in the current case, the agents seem to be directly connected to the APM Server. Based on the extension’s logs, I can see that the extension doesn’t get any metadata or theflushsignal from the agent.Can you remove/unset the
ELASTIC_APM_SERVER_URLor set it tohttp://localhost:8200and try the extension again? (Only theELASTIC_APM_LAMBDA_APM_SERVERshould be set to point to APM Server’s URL).Yup, I can do that today. Due to this issue I had to gut APM from all of our serverless services running on lambda. It may take me some time to get this back up, stand by for logs.