framework: Database queue driver doesn't reconnect
Hi,
The database queue driver doesn’t seem to reconnect to the DB after losing connection. Although I am able to work around this issue it’s a big gotcha for other users of the queue daemon with similar settings. Normal DB operations already have a tryAgainIfCausedByLostConnection method so the same logic could be used for this case: https://github.com/laravel/framework/blob/5.2/src/Illuminate/Database/Connection.php#L691
System:
Laravel: 5.2.45 PHP: 7.0.10 Mac: OS X 10.11.6
Steps to reproduce:
- Start queue daemon:
QUEUE_DRIVER=database php artisan queue:work --daemon --quiet --sleep=1
- Restart MySQL:
brew services stop mysql # or "service mysql stop" on Ubuntu
sleep 10
brew services start mysql # or "service mysql start" on Ubuntu
Stack trace
PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in /Users/example/vendor/laravel/framework/src/Illuminate/Database/Connection.php:584
Stack trace:
#0 /Users/example/vendor/laravel/framework/src/Illuminate/Database/Connection.php(584): PDO->exec('SAVEPOINT trans...')
#1 /Users/example/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(162): Illuminate\Database\Connection->beginTransaction()
#2 /Users/example/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(180): Illuminate\Queue\DatabaseQueue->pop()
#3 /Users/example/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(149): Illuminate\Queue\Worker->getNextJob(Object(Illuminate\Queue\DatabaseQueue), NULL)
#4 /Users/example/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(111): Illuminate\Queue\Worker->pop(NULL, NULL, 0, '1', 0)
#5 /Users/example/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(85): Illuminate\Queue\Worker->runNextJobForDaemon(NULL, NULL, 0, '1', 0)
#6 /Users/example/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(119): Illuminate\Queue\Worker->daemon(NULL, NULL, 0, 128, '1', 0)
#7 /Users/example/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(78): Illuminate\Queue\Console\WorkCommand->runWorker(NULL, NULL, 0, 128, true)
#8 [internal function]: Illuminate\Queue\Console\WorkCommand->fire()
#9 /Users/example/vendor/laravel/framework/src/Illuminate/Container/Container.php(507): call_user_func_array(Array, Array)
#10 /Users/example/vendor/laravel/framework/src/Illuminate/Console/Command.php(169): Illuminate\Container\Container->call(Array)
#11 /Users/example/vendor/symfony/console/Command/Command.php(256): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /Users/example/vendor/laravel/framework/src/Illuminate/Console/Command.php(155): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /Users/example/vendor/symfony/console/Application.php(791): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /Users/example/vendor/symfony/console/Application.php(186): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /Users/example/vendor/symfony/console/Application.php(117): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /Users/example/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(107): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /Users/example/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 {main}
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 5
- Comments: 28 (17 by maintainers)
@vaidas-lungis Probably. My personal opinion is that the daemon mode is an abomination. Php is meant to die. The daemon mode introduces complex scenarios where you need to handle dead database connections, memory limits and garbage collection, and other nasty things regarding long running php scripts. All these need to be handled. Basically, it adds complexity to save some cpu.
The queue documentation has removed any mentions of the daemon workers in 5.3, and the supervisor configuration example no longer contains --daemon. I do not know if this was an oversight, or intentional.
Source: https://software-gunslinger.tumblr.com/post/47131406821/php-is-meant-to-die
@themsaid @GrahamCampbell Please take a look here
Note that the connection fail will only be caught if transactions count is zero. My logs point the error to the line 623 (the
exec
inside theelseif
condition).Since there’s not a try-catch block here, the lost connections won’t be detected and the Queue Woker will throw infinite errors on the log but will not reconnect.
@iget-esoares There’s your problem 😉
Laravel 5.2 does not have the #15511 fix.
Hmm, seems we only do that for query executions, not acquiring transactions. Ping @taylorotwell.
I am runnig Laravl 5.3.31 and I have the same problem with queue DB driver and connection lost. Shoul it by fixed in 5.3.18? Or where can I find the fix?
I’m running Laravel 5.3.9 with MySQL, and have the same issue as @nhowell . I’m also running several Laravel 5.1 projects, but none of them has such issue so far. Below is the stack trace.
This might be what’s causing my queue problems.