framework: MariaDB service crashes since updating to Laravel 11 oom-kill possible memory leak in Laravel 11?

Laravel Version

11.1.0

PHP Version

8.3.4

Database Driver & Version

MariaDB 10.11

Description

Hi 👋

I thought I’d just mention this in case anyone else has experienced this since updating to Laravel 11.

I use MariaDB 10.11 in production. Since updating, I’m seeing that my database has been crashing. I’ve been running my server for a few years now, and it’s always been operating on the latest Laravel version, and also has been fine sitting between 75% and 90% memory utilisation (8GB).

Steps To Reproduce

Two noticeable things are different in Laravel 11 compared to Laravel 10:

Removal of dbal package
Mariadb driver

I wanted to figure out whether I’m missing anything. As a result of this, at a cost, I’ve had to increase my server’s memory. I do wonder whether there’s a memory leak in how the driver works now in Laravel 11?

× mariadb.service - MariaDB 10.11 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: disabled)
     Active: failed (Result: oom-kill) since Thu 2024-03-28 04:00:36 EDT; 5min ago
   Duration: 1d 1h 19min 29.156s
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 766 ExecStartPre=/usr/libexec/mariadb-check-socket (code=exited, status=0/SUCCESS)
    Process: 825 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
    Process: 874 ExecStart=/usr/libexec/mariadbd --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=killed, signal=KILL)
    Process: 1443 ExecStartPost=/usr/libexec/mariadb-check-upgrade (code=exited, status=0/SUCCESS)
   Main PID: 874 (code=killed, signal=KILL)
     Status: "Taking your SQL requests now..."
        CPU: 2h 39min 52.288s

Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade[1473]:   1. Back-up your data before with 'mariadb-upgrade'
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade[1473]:   2. Start the database daemon using 'systemctl start mariadb.service'
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade[1473]:   3. Run 'mariadb-upgrade' with a database user that has sufficient privileges
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade[1473]: Read more about 'mariadb-upgrade' usage at:
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade[1473]: https://mariadb.com/kb/en/mysql_upgrade/
Mar 27 02:41:07 domain-monitor-2022 systemd[1]: Started MariaDB 10.11 database server.
Mar 28 04:00:36 domain-monitor-2022 systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
Mar 28 04:00:36 domain-monitor-2022 systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Mar 28 04:00:36 domain-monitor-2022 systemd[1]: mariadb.service: Failed with result 'oom-kill'.
Mar 28 04:00:36 domain-monitor-2022 systemd[1]: mariadb.service: Consumed 2h 39min 52.288s CPU time.

It’s been almost 3 hours since rebooting my database server, and I’m seeing the memory usage here at 1GB

● mariadb.service - MariaDB 10.11 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: disabled)
     Active: active (running) since Thu 2024-03-28 04:25:45 EDT; 2h 23min ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 834 ExecStartPre=/usr/libexec/mariadb-check-socket (code=exited, status=0/SUCCESS)
    Process: 887 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
    Process: 1182 ExecStartPost=/usr/libexec/mariadb-check-upgrade (code=exited, status=0/SUCCESS)
   Main PID: 946 (mariadbd)
     Status: "Taking your SQL requests now..."
      Tasks: 95 (limit: 100205)
     Memory: 1.0G
        CPU: 15min 19.823s
     CGroup: /system.slice/mariadb.service
             └─946 /usr/libexec/mariadbd --basedir=/usr

Mar 28 04:25:44 domain-monitor-2022 systemd[1]: Starting MariaDB 10.11 database server...
Mar 28 04:25:44 domain-monitor-2022 mariadb-prepare-db-dir[887]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Mar 28 04:25:44 domain-monitor-2022 mariadb-prepare-db-dir[887]: If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]: The datadir located at /var/lib/mysql needs to be upgraded using 'mariadb-upgrade' tool. This can be done using the fo>
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]:   1. Back-up your data before with 'mariadb-upgrade'
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]:   2. Start the database daemon using 'systemctl start mariadb.service'
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]:   3. Run 'mariadb-upgrade' with a database user that has sufficient privileges
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]: Read more about 'mariadb-upgrade' usage at:
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]: https://mariadb.com/kb/en/mysql_upgrade/
Mar 28 04:25:45 domain-monitor-2022 systemd[1]: Started MariaDB 10.11 database server.

About this issue

Original URL
State: closed
Created 3 months ago
Comments: 40 (15 by maintainers)

Most upvoted comments

Okay, I’ll run the same test as above, except have three separate tabs and run the following:

php artisan queue:work redis-short-running --queue=on-demand-runs-now,on-demand-stripe-listeners,cron,listeners,redis-short-running --tries=1 --max-jobs=1000 --max-time=300 --sleep=3

php artisan queue:work redis --queue=on-demand-runs-now,uptime,notifications,dns,certificates,domains-expires,default --tries=1 --max-jobs=1000 --max-time=300 --sleep=3

php artisan queue:work redis-long-running --queue=on-demand-runs-now,domains,blacklists,subscriptions,redis-long-running --tries=1 --max-jobs=1000 --max-time=300 --sleep=3

Essentially manually running them, I’ll report back the same types of screenshots

sts-ryan-holton on Mar 29, 2024

Whether this is any use:

Locally, using Laravel Sail, PHP 8.3, MySQL 5.7 (image: 'mysql/mysql-server:5.7'), Horizon, Redis etc, I’m running the Laravel Task Scheduler using schedule:work and Horizon is processing everything normally as if it was in production, I’m seeing:

Over a 5 minute period, looking within Docker Desktop on the MySQL container, the memory usage graph in here has gone up over the 5 minute period:

It started at: 226MB at 19:30, and and risen to: 236MB by 19:35, an increase of 10MB in 5 minutes? With very little workload

I don’t want this to distract from the issue. Maybe there’s something up with tasks being scheduled in Laravel 11.

Meanwhile, production now at 3.3GB memory usage

sts-ryan-holton on Mar 28, 2024

If you use mariadb you should best use that connection. This will make sure you don’t deviate from it in the future. You should update your uuid columns as in the upgrade guide. If it’s the same problem then that’s most likely not it. You’re also free to stay on mysql if you really want that.

@hafezdivandari @staudenmeir do any of you have any ideas what could cause the memory for the new MariaDB connection to blow up like above?

@sts-ryan-holton can you please add more info to reproduce this? What sort of queries/usage is causing this?

driesvints on Mar 28, 2024