framework: Laravel 5.3 Job fails without calling failed() method

I seriously would never use the DB driver for queues in production. It is no intended to be a production solution.

We’re doing an upgrade to 5.3 right now and coming across some weirdness around failed jobs with the database queue driver.

If I have an email job sitting in jobs table, and throw an intentional exception, then run php artisan queue:listen --tries=3, I expect the job to fail 3 times then move to the failed_jobs table (at least that’s how it worked for us in 5.2). However in 5.3, the job disappears off both tables with no valuable output by the listener, and no email sent.

So confused! Not sure if our problems are related, but it seems something is funky with failed job handling and database queues right now.

I had same issue that job wasn’t written into failed_jobs table after it falied. I finally found this error in log:

[2016-10-05 14:06:14] local.ERROR: PDOException: SQLSTATE[42S22]: Column not found: 1054 Unknown column ‘exception’ in ‘field list’ in …/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:77

I had to delete migration file and entry in migrations table for failed_jobs and rerun php artisan queue:failed-table command. This happened after I upgraded from 5.2 to 5.3. I didn’t find anything in documentation about this change and need to update structure of failed_jobs table.

Hope this helps at solving the issue.

Was jobs table structure changed as well?

2016_08_08_115250_create_failed_jobs_table.php.txt 2016_10_05_145707_create_failed_jobs_table.php.txt

@taylorotwell I work with @JesseLeite and I have been able to reproduce a use case where jobs will not make it to the failed jobs table.

Use Case

• Illuminate\Queue\Worker@process
• Illuminate\Queue\Jobs\Job@fire
  • Something is wrong with the payload, so an error is thrown.
• Illuminate\Queue\Worker@handleJobException
• Illuminate\Queue\Worker@markJobAsFailedIfHasExceededMaxAttempts
  • Assume the job has exceeded it’s max attempts.
• Illuminate\Queue\Worker@failJob
  • Job is deleted from the jobs table.
• Illuminate\Queue\Jobs\Job@failed
  • The same thing is wrong with the payload, so an error is thrown.
• Illuminate\Queue\Worker@failJob
  • Since an exception was thrown $this->raiseFailedJobEvent($connectionName, $job, $e); does not get executed.
  • raiseFailedJobEvent method fires off a Events\JobFailed event, which gets picked up down the line by Illuminate\Queue\Failed\DatabaseFailedJobProvider.
  • DatabaseFailedJobProvider is what is responsible for inserting the failed job into the failed_jobs table.

Summary

The problem as far as I can tell is that the fire method and the failed method on Illuminate\Queue\Jobs\Job are almost identical, so if an error is thrown in the fire method then it will surely be thrown in the failed method as well.

public function fire()
{
    $payload = $this->payload();

    list($class, $method) = $this->parseJob($payload['job']);

    $this->instance = $this->resolve($class);

    $this->instance->{$method}($this, $payload['data']);
}

public function failed($e)
{
    $payload = $this->payload();

    list($class, $method) = $this->parseJob($payload['job']);

    $this->instance = $this->resolve($class);

    if (method_exists($this->instance, 'failed')) {
        $this->instance->failed($payload['data'], $e);
    }
}

If there is something wrong with the payload then an exception could be thrown. For example a ReflectionException if a class does not exist or an exception is thrown when trying to unserialize the command. If something is wrong with the payload the job will never succeed, but we would still like to have it in the failed_jobs table, so that we can analyze the issue.

In our case we were overriding functionality in Illuminate\Queue\SerializesModels@__wakeup that was throwing an error when unserializing the command in Illuminate\Queue\CallQueuedHandler@call.

Since an error gets thrown in the fire method the failed method also throws an exception. The job is deleted and the event that inserts the record in the failed_jobs table is never called.

How to Replicate

The following can be replicated on a fresh install of Laravel:

• .env QUEUE_DRIVER=database
• php artisan queue:table
• php artisan queue:failed-table
• php artisan migrate
• Insert a record into the jobs table with an empty payload.
• In the command line run php artisan queue:listen --tries=1
• The job will be gone from the jobs table and not inserted into the failed_jobs table.
• In this specific instance a ReflectionException is thrown when it tries to make the class from the payload.

Honestly this is a bit frustrating. I’ve unknowingly been using Queues with the database driver since Laravel 5.1. It’s worked fine until 5.3. Now, from my perspective, it doesn’t work in 5.3, so I open an issue and the answer is basically - you shouldn’t have been using that.

Fine, but let’s at least document that if there are no plans to make the database driver production-ready. I doubt I’m the only person using the database driver in production. Clearly, there’s at least 4 of us 😉

After hours of tests and research, I just removed the “Exception” word from the function and it worked for me. Laravel 5.4.

Command “php artisan queue:work --tries=1”

Before (not working):

public function failed(Exception $exception)
{
    Log::info('FAILED');
}

After (working)

public function failed($e)
{
    Log::info('FAILED');
}

Could this be a bug?

@themsaid reading through this issue thread again, it seems everyone that has reported issues here was using the database driver (references to jobs and failed_jobs tables, or explicitly stating they are using the database driver).

Currently my job is failing because of the issue @jasonmccreary raised in the first place

ErrorException: Missing argument 1 for Illuminate\Queue\Jobs\Job::failed(), called in 
/vagrant/vendor/laravel/framework/src/Illuminate/Queue/InteractsWithQueue.php on line 47 
and defined in /vagrant/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php:143

I can find this in my failed_jobs table in the exception column.

Making sure that the failed method gets the correct exception as an argument (see below) I see my jobs disappearing after failing. So they can’t be found in the jobs or the failed_jobs table. There is also no exception in the log…

---

@GrahamCampbell The failed method in InteractsWithQueue is currently not matching the interface the given Job has.

public function failed()
{
    if ($this->job) {
        // This needs an Exception as argument according to it's interface.
        return $this->job->failed(); 
    }
}

Just to keep this up to date, the Database driver is now acceptable for production use cases alongside MySQL 8.0: https://divinglaravel.com/a-production-ready-database-queue-diver-for-laravel

Give me a fresh application that recreates that problem.

It is not intended to be a production solution.

To be fair, if failed jobs are unexpectedly lost to the ether, it’s not stable enough for any environment 😛