framework: Job not logged in failed_jobs if timeout occurs within database transaction
Laravel Version
10.37.3
PHP Version
8.2.7
Database Driver & Version
MariaDB 10.3.27
Description
We had a ‘phantom’ problem that jobs that failed, were not logged to the failed_jobs table. After some investigation we found out that by mistake we were doing a heavy process within a db transaction which lead the specific job to timeout.
After testing we found out that the issue only happens if the jobs timeout during an open db transaction
I reported this in laravel/horizon but it turns out it has nothing to do with horizon but seems to be a framework issue.
Steps To Reproduce
- Create a job with a timeout of 5 seconds
- In the job handle method sleep within a db transaction
public function handle(): void
{
DB::transaction(function (): void {
sleep(10);
});
}
We would expect a job in the failed_jobs table, however this is not the case, and the JobFailed event is also not dispatched
About this issue
- Original URL
- State: closed
- Created 7 months ago
- Comments: 23 (11 by maintainers)
Commits related to this issue
- [10.x] Test Improvements Verify #49389 Signed-off-by: Mior Muhammad Zaki <crynobone@gmail.com> — committed to laravel/framework by crynobone 6 months ago
- [10.x] Test Improvements (#49426) * [10.x] Test Improvements Verify #49389 Signed-off-by: Mior Muhammad Zaki <crynobone@gmail.com> * Apply fixes from StyleCI --------- Signed-off-by: M... — committed to laravel/framework by crynobone 6 months ago
@crynobone we use laravel horizon which automatically handles that. But please be aware: If the job had $tries=1 (which most of our jobs do) the job is NOT retried, and NOT logged, and therefore lost.
Please correct my if i’m wrong. This is the whole point of this ticket - loosing jobs
@gkmk Thank you for your insight and suggestions.
The reason i would tend to shy away from option 1) is because then behaviour would be different between jobs that are batchable, and ones that are not. This would be an unnecessary inconsistency in my opinion. (See my previous comment)
On a more subjective note, we have > 100 jobs, most of them have transactions and do not require failed methods. Option 2) would add failed methods just to handle this specific case to not loose jobs, which seems to be quite an overhead.
Will be interesting to see what the core of laravel developers have to say.