magento2: AUTO_INCREMENT grows on every INSERT ... ON DUPLICATE on InnoDB tables

Preconditions (*)

  1. any magento version, mysql or mariaDB database
  2. Magento 2.4-develop
  3. InnoDB tables
  4. Magento uses queries like this to save new values or update existing in tables see method \Magento\Eav\Model\Entity\AbstractEntity::_processAttributeValues
INSERT INTO `catalog_product_entity_int` (`attribute_id`, `store_id`, `entity_id`, `value`)
VALUES (97, 1, 123, 1)
ON DUPLICATE KEY UPDATE `value` = VALUES(`value`);

The problem causes that the table reaches max autoincrement value too fast that leads to errors. Changing column type to bigint just temporalny resolves the problem but autoincrement still grows to infinity Problem is noticeable when products or some attributes are updated often and new products/attribute values (e.g. in store) added. New attributes have bigger autoincrement than expected. Moreover, the method \Magento\Eav\Model\Entity\AbstractEntity::_processAttributeValues always uses statements INSERT INTO ... ON DUPLICATE KEY UPDATE to insert new value or update existing. So updating existing values this way when innodb_autoinc_lock_mode > 0 causes the issue.

Steps to reproduce (*)

  1. Take auto_increment value from table catalog_product_entity_int
  2. Remember that value (e.g. 1000)
  3. Save one new product attribute (e.g. int type).
$this->productResourceModel->saveAttribute($product, 'status');

it generates query like above. INSERT INTO … ON DUPLICATE KEY UPDATE 4. New value inserted and auto_increment is 1001 5. Run the same query, or save the same attribute 4 more times 6. Save the same attribute under store (important that value not exist before insertion) 7. Check the auto_increment value of the table or MAX(value_id)

Expected result (*)

  1. The auto_increment value or MAX(value_id) is 1002
  2. Only one new record inserted that increased auto_increment by 1

Actual result (*)

  1. The auto_increment value or MAX(value_id) is 1006
  2. Only one new record inserted but auto_increment increased by 5

Workaround: (*)

Set parametr in my.cnf

innodb_autoinc_lock_mode=0

or recreate column

ALTER TABLE catalog_product_entity_int
drop column value_id;

ALTER TABLE catalog_product_entity_int
ADD value_id BIGINT(11) AUTO_INCREMENT NOT NULL FIRST,
ADD PRIMARY KEY (value_id);

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

  • Severity: S0 - Affects critical data or functionality and leaves users without workaround.
  • Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
  • Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
  • Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
  • Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 15
  • Comments: 35 (14 by maintainers)

Commits related to this issue

Most upvoted comments

@veloraven Here is the reproducer https://github.com/magento/magento2/pull/34339

I think that by default magento should suggest to use innodb_autoinc_lock_mode = 0 as insertOnDuplicate is widely used.

PS. Confirmed issues should not be closed by bots

@engcom-Delta, @sdzhepa: why did this issue got closed without any sort of extra information?

Hello,

As I can see this issue got fixed in the scope of the internal Jira ticket ACP2E-1358 by the internal team Related commits:https://github.com/search?q=repo%3Amagento%2Fmagento2+ACP2E-1358&type=commits

Based on the Jira ticket, the target version is 2.4.7-beta1.

Thanks

Any update on this issue? Just ran into this in production. Any timeframe on a fix?

change column type on value_id to bigint(20) or following this step https://gist.github.com/JesKingDev/af70e92740215009ab1bebe5a2cdc17f#2-get-the-current-max-value_id-used-in-the-table

Just hit this on a staging instance (thankfully)

This is a major issue in lot of Magento installations. Quite concerning this just gets closed without further explanation.

@JelleGe I created a gist for getting the database table back in shape if you’ve maxed out the auto_increment. https://gist.github.com/JesKingDev/af70e92740215009ab1bebe5a2cdc17f It’s very thorough and has a lot of checkpoints to ensure you know what’s being done.

This isn’t a long-term solution, but should definitely buy you back some time while the root cause is fixed.

It’s a fine temporary solution that works, but it’s a matter of time before it returns. So will there become a solution in a future Magento core update?

Any update on this issue? Just ran into this in production. Any timeframe on a fix?

@pmonosolo https://github.com/magento/magento2/issues/21890

The “workaround” section on that is largely valid

Due to declarative schema we need to drop the constraints on the existing table, and add properly named constraints on the new table before swapping the table names around.

But this way we end up back at auto_increment=1 with the original Magento schema intact

Thanks @pmonosolo

We’re just building a module that will fire an alert when we are over 90% capacity on auto increment IDs, to allow us to do maintenance.

It’s only every few years we’ve encountered it

Hello @ilnytskyi,

Thanks for the report and collaboration!

We have tried to reproduce the issue in Magento’s latest development branch ie 2.4-develop and the issue is still reproducible. We have followed the exact steps and refer to this PR https://github.com/magento/magento2/pull/34339 to reproduce the issue.

Hence confirming the issue.

Thanks

Any update on this issue? Just have run into it with a high-traffic Magento instance.

Oh yeah @pmonosolo that does indeed look to be the same symptoms as my issue from #21890

In that case it was fixed in 2.2, but it seems like some kind of regression since then.

Thanks for flagging this to me! 😱

@JelleGe try to recreate a column and change autoincrement lock mode to 0 as I described in workaround.