symfony: [Messenger][SQS] Uncaught SQS Exception when worker stopped.

Symfony version(s) affected

5.4.23

Description

It’s not a problem that happens every time, the frequency is about 1 in 3.

SupervisorD is running and monitoring : bin/console messenger:consume mailer --no-tracking --time-limit=3600. Every ~3600 sec, messenger worker will be stopped to be restarted automatically by supervisord.

When worker is stopping, SQS throw an uncaught exception (see additional context).

How to reproduce

I don’t know how to reproduce. This configuration has been in production for several months, and the problem appeared less than 2 months ago.

I spent many hours understanding the situation. So far, I only have thoughts.

Note that the problem is occurring only when the worker is stopping after “time-limit”.

Possible Solution

poll_timeout < wait_time

As a reminder, default SQS Messenger configuration:

wait_time: 20
poll_timeout: 0.1
buffer_size: 9

From AWS SQS Documentation.

Important: To avoid HTTP errors, ensure that the HTTP response timeout for ReceiveMessage requests is longer than the WaitTimeSeconds parameter.

AmazonSqs\Transport\Connection__destruct() is calling already destructed useful objects.

aws-async is a library that is not easy to debug…

Long polling is not working

Additional Context

messenger.yaml

framework:
    messenger:
        reset_on_message: true # reset services after consuming messages
        transports:
            mailer:
                dsn: "sqs://sqs.%amazon_region%.amazonaws.com"
                options:
                    region: "%amazon_region%"
                    access_key: "%amazon_key%"
                    secret_key: "%amazon_secret%"
                    queue_name: "%queuing_queue_namespace%-emails.fifo"
                retry_strategy:
                    max_retries: 3
                    delay: 1000
                    multiplier: 2
                    max_delay: 0

Two kind of exceptions:

ReceiveMessageResult.php on line 33

thrown in /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php on line 33
#7 {main}
#6 [internal function]: Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->__destruct()
#5 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(82): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->reset()
#4 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(371): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->fetchMessage()
#3 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(236): AsyncAws\Sqs\Result\ReceiveMessageResult->getMessages()
#2 /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php(26): AsyncAws\Core\Result->initialize()
#1 /home/ec2-user/acme/vendor/async-aws/core/src/Result.php(133): AsyncAws\Sqs\Result\ReceiveMessageResult->populateResult(Object(AsyncAws\Core\Response))
#0 /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php(33): SimpleXMLElement->__construct('')
Stack trace:
PHP Fatal error:  Uncaught Exception: String could not be parsed as XML in /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php:33

AsyncResponse.php on line 287

thrown in /home/ec2-user/acme/vendor/symfony/http-client/Response/AsyncResponse.php on line 287
#8 {main}
#7 [internal function]: Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->__destruct()
#6 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(82): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->reset()
#5 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(371): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->fetchMessage()
#4 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(232): AsyncAws\Core\Result->resolve(0.1)
#3 /home/ec2-user/acme/vendor/async-aws/core/src/Result.php(63): AsyncAws\Core\Response->resolve(0.1)
#2 /home/ec2-user/acme/vendor/async-aws/core/src/Response.php(143): Symfony\Component\HttpClient\Response\ResponseStream->rewind()
#1 /home/ec2-user/acme/vendor/symfony/http-client/Response/ResponseStream.php(47): Generator->rewind()
#0 [internal function]: Symfony\Component\HttpClient\Response\AsyncResponse::stream(Array, 0.1, '...')
Stack trace:
PHP Fatal error:  Uncaught LogicException: Instance of "Symfony\Component\HttpClient\Response\CurlResponse" is already consumed and cannot be managed by "Symfony\Component\HttpClient\RetryableHttpClient". A decorat
ed client should not call any of the response's methods in its "request()" method. in /home/ec2-user/acme/vendor/symfony/http-client/Response/AsyncResponse.php:287

supervisord output

CLI : bin/console messenger:consume mailer --no-tracking --time-limit=3600 -vv

02:25:35 INFO      [messenger] Received message Symfony\Component\Mailer\Messenger\SendEmailMessage ["class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage"]

 // Quit the worker with CONTROL-C.

 // received a stop signal via the messenger:stop-workers command.
 // The worker will automatically exit once it has been running for 3600s or

 [OK] Consuming messages from transport "mailer".

  thrown in /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php on line 33
#7 {main}
#6 [internal function]: Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->__destruct()
#5 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(82): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->reset()
#4 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(371): Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\Connection->fetchMessage()
#3 /home/ec2-user/acme/vendor/symfony/amazon-sqs-messenger/Transport/Connection.php(236): AsyncAws\Sqs\Result\ReceiveMessageResult->getMessages()
#2 /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php(26): AsyncAws\Core\Result->initialize()
#1 /home/ec2-user/acme/vendor/async-aws/core/src/Result.php(133): AsyncAws\Sqs\Result\ReceiveMessageResult->populateResult(Object(AsyncAws\Core\Response))
#0 /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php(33): SimpleXMLElement->__construct('')
Stack trace:
PHP Fatal error:  Uncaught Exception: String could not be parsed as XML in /home/ec2-user/acme/vendor/async-aws/sqs/src/Result/ReceiveMessageResult.php:33
02:22:16 INFO      [messenger] Worker stopped due to time limit of 3600s exceeded ["timeLimit" => 3600]
02:22:16 INFO      [messenger] Stopping worker. ["transport_names" => ["mailer"]]
02:16:48 INFO      [messenger] Symfony\Component\Mailer\Messenger\SendEmailMessage was handled successfully (acknowledging to transport). ["class" => "Symfony\Component\Mailer\Messenger\SendEmailMessage"]
02:16:48 INFO      [messenger] Message Symfony\Component\Mailer\Messenger\SendEmailMessage handled by Symfony\Component\Mailer\Messenger\MessageHandler::__invoke ["class" => "Symfony\Component\Mailer\Messenger\Send
EmailMessage","handler" => "Symfony\Component\Mailer\Messenger\MessageHandler::__invoke"]

About this issue

Original URL
State: open
Created a year ago
Reactions: 6
Comments: 16 (5 by maintainers)

Most upvoted comments

We have the same issue. Using standard queues.

jdelaune on Aug 9, 2023

Same issue here:

 // The worker will automatically exit once it has processed 9 messages, been running for 15s or received a stop signal 
 // via the messenger:stop-workers command.                                                                             

 // Quit the worker with CONTROL-C.                                                                                     

 // Re-run the command with a -vv option to see logs about consumed messages.                                           

PHP Fatal error:  Uncaught LogicException: Instance of "Symfony\Component\HttpClient\Response\CurlResponse" is already consumed and cannot be managed by "Symfony\Component\HttpClient\Response\AsyncResponse". A decorated client should not call any of the response's methods in its "request()" method. in /var/www/html/vendor/symfony/http-client/Response/AsyncResponse.php:284
Stack trace:
#0 /var/www/html/vendor/symfony/http-client/Response/AsyncResponse.php(66): Symfony\Component\HttpClient\Response\AsyncResponse::stream(Array, -0.0)
#1 /var/www/html/vendor/symfony/http-client/Response/CommonResponseTrait.php(142): Symfony\Component\HttpClient\Response\AsyncResponse::Symfony\Component\HttpClient\Response\{closure}(Object(Symfony\Component\HttpClient\Response\AsyncResponse), -0.0)
#2 /var/www/html/vendor/symfony/http-client/Response/AsyncResponse.php(96): Symfony\Component\HttpClient\Response\AsyncResponse::initialize(Object(Symfony\Component\HttpClient\Response\AsyncResponse))
#3 /var/www/html/vendor/async-aws/core/src/Response.php(141): Symfony\Component\HttpClient\Response\AsyncResponse->getStatusCode()
#4 /var/www/html/vendor/async-aws/core/src/Response.php(118): AsyncAws\Core\Response->resolve()
#5 [internal function]: AsyncAws\Core\Response->__destruct()
#6 {main}
  thrown in /var/www/html/vendor/symfony/http-client/Response/AsyncResponse.php on line 284

Fatal error: Uncaught LogicException: Instance of "Symfony\Component\HttpClient\Response\CurlResponse" is already consumed and cannot be managed by "Symfony\Component\HttpClient\Response\AsyncResponse". A decorated client should not call any of the response's methods in its "request()" method. in /var/www/html/vendor/symfony/http-client/Response/AsyncResponse.php:284
Stack trace:
#0 /var/www/html/vendor/symfony/http-client/Response/AsyncResponse.php(66): Symfony\Component\HttpClient\Response\AsyncResponse::stream(Array, -0.0)
#1 /var/www/html/vendor/symfony/http-client/Response/CommonResponseTrait.php(142): Symfony\Component\HttpClient\Response\AsyncResponse::Symfony\Component\HttpClient\Response\{closure}(Object(Symfony\Component\HttpClient\Response\AsyncResponse), -0.0)
#2 /var/www/html/vendor/symfony/http-client/Response/AsyncResponse.php(96): Symfony\Component\HttpClient\Response\AsyncResponse::initialize(Object(Symfony\Component\HttpClient\Response\AsyncResponse))
#3 /var/www/html/vendor/async-aws/core/src/Response.php(141): Symfony\Component\HttpClient\Response\AsyncResponse->getStatusCode()
#4 /var/www/html/vendor/async-aws/core/src/Response.php(118): AsyncAws\Core\Response->resolve()
#5 [internal function]: AsyncAws\Core\Response->__destruct()
#6 {main}
  thrown in /var/www/html/vendor/symfony/http-client/Response/AsyncResponse.php on line 284

Using symfony/amazon-sqs-messenger 6.3.0 and async-aws/sqs 1.7.0

jgmullor on Jun 19, 2023

The error Uncaught Exception: String could not be parsed as XML no longer occurs after upgrading from async-aws/sqs 1.9.0 to 2.0.0.

https://github.com/async-aws/sqs/commit/cef9f101f9b35b0a7e55cc9995e21ee4b1501c0b#diff-d2ab9925cad7eac58e0ff4cc0d251a937ecf49e4b6bf57f8b95aab76648a9d34L15

It seems there have been changes related to AWS JSON-1.0 Protocol, and JSON Protocol seems to be prioritized over XML.
(There might also be a possibility that related packages changed priorities due to composer update symfony/amazon-sqs-messenger -W.)

Instead, around the same time, the error Uncaught Exception: Could not parse response as array started occurring.
(The frequency of its occurrence is occasional.)

StackTrace after the upgrade

Exception

# AsyncAws\Core\Exception\UnparsableResponse

Uncaught Exception: Could not parse response as array

"file": "/var/www/app/vendor/async-aws/core/src/Response.php:353",

Previous Exception

# Symfony\Component\HttpClient\Exception\JsonException

Response body is empty.

"file": "/var/www/app/vendor/symfony/http-client/Response/CommonResponseTrait.php:80"

Environment

Symfony 6.4.5
PHP 8.3
async-aws/sqs 2.0.0
symfony/amazon-sqs-messenger 6.4.4

iamyukihiro on Mar 14, 2024

We started to receive same error after 6.4.0 update. No SQS changes. It is standard SQS queue worked for years as is. Here is stack trace from Sentry which is similar to provided in issue description.

Our worker is running with following arguments:

bin/console messenger:consume async --memory-limit=128M --time-limit=300 --limit=50 --failure-limit=10 -vv --env=prod

drekinov on Dec 21, 2023