magento2: OOM/Timeouts when saving a category with large catalog as all product URLs are regenerated
With a catalog of many thousands of products loaded, saving a category causes long execution times and high memory usage, when it isn’t necessarily required. In a production environment, the execution fails with max_execution_time or memory_limit errors.
It appears that the system is trying to regenerate all URLs of products contained within the category and subcategories; even if the url_key is not modified.
Preconditions
- Magento 2.1.1
max_execution_timeset to 300memory_limitset to 2G- Catalog of 140,000 products
- Top category which contains subcategories and 40,000+ products
Steps to reproduce
- Simply “Save” the top category without even making any changes
Expected result
- No errors
- A notice of
You saved the categoryshould appear
Actual result
- PHP errors for
max_execution_timeand/ormemory_limit
A workaround is to modify /vendor/magento/module-catalog-url-rewrite/Observer/CategoryProcessUrlRewriteSavingObserver.php temporarily to return; on line 1 of the execute() function to prevent URL rewrite changes.
A suggested performance improvement would be to check if the url_key was modified before running the regeneration of URLs.
This, though, cannot be used if the url_key is updated. At that point, the process should be handled by some background task.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 5
- Comments: 30 (10 by maintainers)
Commits related to this issue
- Fix: https://github.com/magento/magento2/issues/7874 >=M2.1.4 <= M2.1.9 — committed to ConvertGroupsAS/magento2-patches by deleted user 6 years ago
- Fix: https://github.com/magento/magento2/issues/7874 >=M2.1.4 <= M2.1.9 — committed to ConvertGroupsAS/magento2-patches by deleted user 6 years ago
Reopening as I have recently seen this happen on 2.2.7. There are a number of problems involved, one of them being that
\Magento\UrlRewrite\Model\Storage\DbStorage::deleteOldUrlsdoes not properly remove existing rewrites and allows clashes. It assumes that if there is notentity_idconflict inurl_rewritethen there will be no conflict on the table but that’s not always true (as evidenced by the integrity constraint errors reported). Another is that when removing a product from a category it seems that Magento goes and updates all the other products in that category to reflect their new positions with that product removed, but then when those products are saved all of their URL rewrites are regenerated because the products are considered “changed” even though theurl_keyhas not changed.Any update on this?
One of my 2.1.4 merchants is experiencing this exact issue. They have a category that contains 538 children/descendent categories and those child categories contain a total of 5k+ products. Upon saving the category, the
768Mnative Magento memory limit is hit. Magento’s memory limit should not need to be increased to accommodate saving this category.When the category is saved, the
\Magento\CatalogUrlRewrite\Observer\CategoryUrlPathAutogeneratorObserver::updateUrlPathForChildrenobserver calls\Magento\CatalogUrlRewrite\Observer\CategoryUrlPathAutogeneratorObserver::updateUrlPathForChildrenfor all ~583 child categories. This behavior happens of regardless of whether “By Category” is set to “Anchor == Yes” or “Anchor == No”.I profiled the category save using Xdebug and loaded the data in MacCallGrind and have included some screenshots below demonstrating how much of the processing time was used by the method mentioned above:
Hi @hostep Hi @sjb9774 this may be fixed by this commit -> https://github.com/magento/magento2/commit/83204d24e6911c605d40a4021035d85b22d07b1f
Our Upgraded version is 2.2.2 still having the same issue removing one product from the category and then saving category with 4k products taking 5mins to save and some time fails. can’t find this fix in new version too. checked in the file “CategoryProcessUrlRewriteSavingObserver.php” your code fix is not there. can you please verify?