symfony: [Messenger][SQS] Uncaught SQS Exception when worker stopped.
Symfony version(s) affected
5.4.23
Description
It’s not a problem that happens every time, the frequency is about 1 in 3.
SupervisorD is running and monitoring : bin/console messenger:consume mailer --no-tracking --time-limit=3600.
Every ~3600 sec, messenger worker will be stopped to be restarted automatically by supervisord.
When worker is stopping, SQS throw an uncaught exception (see additional context).
How to reproduce
I don’t know how to reproduce. This configuration has been in production for several months, and the problem appeared less than 2 months ago.
I spent many hours understanding the situation. So far, I only have thoughts.
Note that the problem is occurring only when the worker is stopping after “time-limit”.
Possible Solution
poll_timeout < wait_time
As a reminder, default SQS Messenger configuration:
- wait_time: 20
- poll_timeout: 0.1
- buffer_size: 9
From AWS SQS Documentation.
Important: To avoid HTTP errors, ensure that the HTTP response timeout for ReceiveMessage requests is longer than the WaitTimeSeconds parameter.
AmazonSqs\Transport\Connection__destruct() is calling already destructed useful objects.
aws-async is a library that is not easy to debug…
Long polling is not working
Additional Context
messenger.yaml
framework:
messenger:
reset_on_message: true # reset services after consuming messages
transports:
mailer:
dsn: "sqs://sqs.%amazon_region%.amazonaws.com"
options:
region: "%amazon_region%"
access_key: "%amazon_key%"
secret_key: "%amazon_secret%"
queue_name: "%queuing_queue_namespace%-emails.fifo"
retry_strategy:
max_retries: 3
delay: 1000
multiplier: 2
max_delay: 0
Two kind of exceptions:
ReceiveMessageResult.php on line 33
thrown in /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php on line 33
#7 {main}
#6 [internal function]: Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->__destruct()
#5 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(82): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->reset()
#4 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(371): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->fetchMessage()
#3 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(236): AsyncAws\Sqs\Result\ReceiveMessageResult->getMessages()
#2 /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php(26): AsyncAws\Core\Result->initialize()
#1 /home/ec2-user/acme/vendor/async-aws/core/src/Result.php(133): AsyncAws\Sqs\Result\ReceiveMessageResult->populateResult(Object(AsyncAws\Core\Response))
#0 /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php(33): SimpleXMLElement->__construct('')
Stack trace:
PHP Fatal error: Uncaught Exception: String could not be parsed as XML in /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php:33
AsyncResponse.php on line 287
thrown in /home/ec2-user/acme/vendor/symfony/http-client/Response/AsyncResponse.php on line 287
#8 {main}
#7 [internal function]: Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->__destruct()
#6 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(82): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->reset()
#5 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(371): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->fetchMessage()
#4 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(232): AsyncAws\Core\Result->resolve(0.1)
#3 /home/ec2-user/acme/vendor/async-aws/core/src/Result.php(63): AsyncAws\Core\Response->resolve(0.1)
#2 /home/ec2-user/acme/vendor/async-aws/core/src/Response.php(143): Symfony\Component\HttpClient\Response\ResponseStream->rewind()
#1 /home/ec2-user/acme/vendor/symfony/http-client/Response/ResponseStream.php(47): Generator->rewind()
#0 [internal function]: Symfony\Component\HttpClient\Response\AsyncResponse::stream(Array, 0.1, '...')
Stack trace:
PHP Fatal error: Uncaught LogicException: Instance of "Symfony\Component\HttpClient\Response\CurlResponse" is already consumed and cannot be managed by "Symfony\Component\HttpClient\RetryableHttpClient". A decorat
ed client should not call any of the response's methods in its "request()" method. in /home/ec2-user/acme/vendor/symfony/http-client/Response/AsyncResponse.php:287
supervisord output
CLI : bin/console messenger:consume mailer --no-tracking --time-limit=3600 -vv
02:25:35 INFO [messenger] Received message Symfony\Component\Mailer\Messenger\SendEmailMessage ["class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage"]
// Quit the worker with CONTROL-C.
// received a stop signal via the messenger:stop-workers command.
// The worker will automatically exit once it has been running for 3600s or
[OK] Consuming messages from transport "mailer".
thrown in /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php on line 33
#7 {main}
#6 [internal function]: Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->__destruct()
#5 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(82): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->reset()
#4 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(371): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->fetchMessage()
#3 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(236): AsyncAws\Sqs\Result\ReceiveMessageResult->getMessages()
#2 /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php(26): AsyncAws\Core\Result->initialize()
#1 /home/ec2-user/acme/vendor/async-aws/core/src/Result.php(133): AsyncAws\Sqs\Result\ReceiveMessageResult->populateResult(Object(AsyncAws\Core\Response))
#0 /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php(33): SimpleXMLElement->__construct('')
Stack trace:
PHP Fatal error: Uncaught Exception: String could not be parsed as XML in /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php:33
02:22:16 INFO [messenger] Worker stopped due to time limit of 3600s exceeded ["timeLimit" => 3600]
02:22:16 INFO [messenger] Stopping worker. ["transport_names" => ["mailer"]]
02:16:48 INFO [messenger] Symfony\Component\Mailer\Messenger\SendEmailMessage was handled successfully (acknowledging to transport). ["class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage"]
02:16:48 INFO [messenger] Message Symfony\Component\Mailer\Messenger\SendEmailMessage handled by Symfony\Component\Mailer\Messenger\MessageHandler::__invoke ["class" => "Symfony\Component\Mailer\Messenger\Send
EmailMessage","handler" => "Symfony\Component\Mailer\Messenger\MessageHandler::__invoke"]
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 17
- Comments: 32 (6 by maintainers)
Hi,
We encountered the same issue in our environments. The scale of some of our processes was so large that it occurred quite frequently, so we had to address it. We also found a related issue regarding unprocessed messages.
Environment
Issues
Instance of "Symfony\Component\HttpClient\Response\CurlResponse" is already consumed and cannot be managed by ...TL;DR: Messenger receives a
SIGTERMbefore processing all retrieved messages. The\Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connectionclass implements the_destructmethod, which is responsible for returning unprocessed messages to SQS using asynchronous CURL requests. However, these requests are sent asynchronously, and their responses compete with the termination of the main PHP thread.ApproximateAgeOfOldestMessage).TL;DR: Messenger sends an asynchronous CURL request to retrieve messages from the queue, waiting up to 20 seconds for new messages to arrive. In the meantime, Messenger receives a
SIGTERM. The\Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connectionclass implements the_destructmethod, which attempts to cancel any pending asynchronous requests. However, the CURL request continues listening even after PHP termination. If a new message appears, the long polling request completes, causing SQS to mark the message as “in flight.” Since PHP is already terminated, the response is ignored, and the worker does not process the message.Conclusions I believe that the
_destructmethod in\Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connectionshould not be used for executing complex operations, especially sending asynchronous API requests. Additionally, I checked the latest version ofsymfony/amazon-sqs-messenger(7.2), and the_destructmethod is still present. This suggests that the described behavior may still occur in the latest version of Messenger.Dirty fix
Here is a dirty hack that resolved the above issues in our environments. Let me know if this helped you as well - I hope it will 😃
Issue 1
Issues 2
Any update for this?
Instance of "Symfony\Component\HttpClient\Response\CurlResponse" is already consumed and cannot be managed by "Symfony\Component\HttpClient\Response\AsyncResponse". A decorated client should not call any of the response's methods in its "request()" method.This error causes some messages to randomly enter in a visibility timeout for
Xhours/minutes, depending on the AWS SQS configuration. In our case, it’s 12 hours. So, when this exception is thrown, the message is simply hidden from the queue and will only be available again after 12 hours, which adds a significant delay in processing certain tasks.This may be useful for those who are trying to suppress these errors.
We have the same issue. Using standard queues.
Same issue here:
Using
symfony/amazon-sqs-messenger6.3.0 andasync-aws/sqs1.7.0@jetrodn Yes, you are right. It’s issue 2 and as you mentioned will be resolved. But there is another place (issue 1) which emit the same exception when worker is shutting down and tries return back unprocessed messages. It may happen when you have
buffer_sizegreater than 1The exceptions
String could not be parsed as XMLorCould not parse response as arrayseems like depend on bothwait_timeandpoll_timeoutvalues, for some values it can be reproduced for other values everything works like intended.As example: for values
wait_time=2andpoll_timeout=0.2I am getting{}response that is valid JSON and no exception:For values
wait_time=4andpoll_timeout=0.6I am getting''response with exception:I assume it is related to Symfony HttpClient and
streammethod and how it works with chunks, especially I am getting thisErrorChunkand there are multiple places where chunk content is prefilled with''as content, like this for example. Maybe it happens because connection is getting cancelled by the client and then the content is''empty string instead of valid JSON or XML.The error
Uncaught Exception: String could not be parsed as XMLno longer occurs after upgrading fromasync-aws/sqs1.9.0 to 2.0.0.https://github.com/async-aws/sqs/commit/cef9f101f9b35b0a7e55cc9995e21ee4b1501c0b#diff-d2ab9925cad7eac58e0ff4cc0d251a937ecf49e4b6bf57f8b95aab76648a9d34L15
It seems there have been changes related to AWS JSON-1.0 Protocol, and JSON Protocol seems to be prioritized over XML.
(There might also be a possibility that related packages changed priorities due to
composer update symfony/amazon-sqs-messenger -W.)Instead, around the same time, the error
Uncaught Exception: Could not parse response as arraystarted occurring.(The frequency of its occurrence is occasional.)
StackTrace after the upgrade
Exception
Previous Exception
Environment
We started to receive same error after 6.4.0 update. No SQS changes. It is standard SQS queue worked for years as is. Here is stack trace from Sentry which is similar to provided in issue description.
Our worker is running with following arguments: