framework: supervised queue:listen stops working

I’m running the queue listen process via the following supervisor config (I’m using SQS).

[program:laravel-queue]
command=/usr/bin/php /var/www/server/current/artisan queue:listen --timeout 300 --tries=5 --env="production"
username=www-data
autostart=true
autorestart=true
stdout_logfile=/var/log/%(program_name)s.stdout
stderr_logfile=/var/log/%(program_name)s.stderr

Sometimes (~ every 5th day) the listen process “stops” and doesn’t consume any further tasks in the queue. The process doesn’t exit and thus won’t be restarted via supervisor. The stdout/stderr logfiles are empty.

When I restart the queue listen process via supervisorctl restart all queued up processes get worked off.

Is there a best practice to monitor this behavior correctly until this issue gets resolved? I’ve written a little package to check for me if the queue is jammed, but it feels not very clean: https://github.com/schickling/laravel-queue-checker

About this issue

Original URL
State: closed
Created 10 years ago
Reactions: 3
Comments: 38 (16 by maintainers)

Most upvoted comments

A, ok. I see which part you do not understand.

So, the Laravel workers work like this:

they check if there are jobs in the queue
first they check all the reserved jobs and check if any of them were reserved for more than expire property that you have in your config file. if there are expired jobs, they release them. they do no check if some other worker died or it’s just processing for so long.
they select the job and reserved it
they start processing their reserved job
they release it (or they die trying)

The bolded text is the most important part, in that step the worker can select an already processing job, if it’s processing for to long, and they think that the first worker died and it just forget to release it. So the easy fix is to set expire to a bit more, like an hour or so.

The second, better approach is the solution that I describe in previous comment.

I hope I made it a bit more clear for you now.

matejvelikonja on Apr 27, 2016

Just an FYI, mine was caused by using an incorrect php in command attribute in supervisord.conf file. I changed it from php to /usr/local/bin/php and it fixed it.

zenichanin on Jan 18, 2017

Are they making HTTP calls? If so make sure you set a max timeout in your HTTP client.

On Tue, Nov 17, 2015 at 3:01 AM, Matej Velikonja notifications@github.com wrote:

Has anoyone figured out anything regarding this? I’ve encountered the same issue on laravel 5.1.*. With supervisor I spawned 5 workers in daemon mode. They all start working for the first time, but after some time only one is grabbing new jobs, the rest are just there and they don’t do anything.

Reply to this email directly or view it on GitHub: https://github.com/laravel/framework/issues/4443#issuecomment-157312961

taylorotwell on Nov 17, 2015