magento2: Cron using too many resources.

We have been seeing stuck processes, but maybe its an effect of running out of ram resources on the VPS. Although I have made two changes that have so far seemed to improve things:

1, Truncated the cron log as it was getting very large 2, We were seeing errors in the system log relating to aqmp not being configured. We don’t use RabbitMQ so I just added the code below to env.php:

,
'cron_consumers_runner' => [
        'cron_run' => false,
        'max_messages' => 1000,
        'consumers' => [
            'async.operations.all'
        ]
    ] 

The stuck processes could be an effect of a few things happening at once, so it might require a longer period to see if things really have improved…

Hi @bobemoe. Thank you for your report. The issue has been fixed in magento/magento2#28007 by @driskell in 2.4-develop branch Related commit(s):

• aa12503923ac994873724ecaa759e1eb14b318dc
• b0b72b0f8486ca119015e84b7786dfab13d2e475
• 315bd3af8c5ed3dd711abd40db9a163b0c953855
• 96029b55f7cd00b58967637aedac67ddd1767bad
• 7df183b6c6bc3f57f0c3d29d279c3c6d631cf455
• 68d679b7bd7d371c5266d4803f03f204893f7af2
• a90161907aedae371cb307759b685cc254efe2d8
• dda5c72c9303f9476ce7041743c87079cada87f6
• 4d4fa685c0b507b510ad899f5afe94bd6c836115
• e95d2fefefb62150e566f000a4a17f58f6716af5
• 916bc93138f7a81f65c2e8b5069eeb0f35fceb43
• edd77531f8847f7b13dbaca9aefa315ba4aa5e57
• e70027f838f766b816bd1f7063514c1a9caf7a20
• 92dc76438063e68f7523d5ab036a57d7faf7bc03
• d1b22ff825052d836b56f8ce33760d42b2241980
• a76482c4614a7660ab2d324ca4c086f66906c701

The fix will be available with the upcoming 2.4.3 release.

Try to install each group separetely with locks This way you only run cron groups you want Processes are not overlaped and stay locked

https://github.com/magento/magento2/issues/25987#issuecomment-576034263 e.g. add in crontab smth like this

#RUN MAGENTO CRON GROUPS START

* * * * * chronic flock -w 0 /tmp/crongroup_default /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=default --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log
* * * * * chronic flock -w 0 /tmp/crongroup_index /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log
* * * * * chronic flock -w 0 /tmp/crongroup_import /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=import --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log

#do not run magento consumers in cron for now
#* * * * * chronic flock -w 0 /tmp/crongroup_consumers /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=consumers --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log
#do not run ddg automation (Engagement Cloud)
#* * * * * chronic flock -w 0 /tmp/crongroup_ddg_automation /usr/bin/php7.1 /var/www/magento/bin/magento cron:run --group=ddg_automation --bootstrap=standaloneProcessStarted=1 >> /var/www/magento/var/log/magento.cron.log

#RUN MAGENTO CRON GROUPS END

We’ve also disabled the cron_consumers_runner/cron_run = false but also added queue/consumers_wait_for_messages = 0 (both in env.php). This makes the processes stop when the’re not receiving any messages. We’re now running each individual process as a separate cron process. The enables you to control how often the process spawns (Magento will just spawn it on each cron run). For example the product_action_attribute.update is started every minute, but queue:consumers:start exportProcessor only runs once an hour (we don’t really use the export functionality, but did’t want to completely disable it).

Word of caution: This can delay certain processes. As an example product_action_attribute.update is responsible for processing mass attribute updates. If you only run this process once every hour, every mass update can take up to an hour before it’s actually processed.

Reading material: https://github.com/magento/community-features/issues/180 https://github.com/magento/magento2/issues/17951

Just a heads up (not sure if that’s the issue, but it might have been overlooked); Since Magento 2.3.3 the cron spawns several consumer processes (about 5). Each of these processes uses resources (~200Mb in RAM, not sure on the cpu but it’s more than nothing). In our case it will also occasionally leave queue:consumers:start async.operations.all processes running while starting new ones (resulting in 4-5 duplicate processes).

So all in all it could easily consume an additional ~1GB in memory compared to earlier versions.