magento2: Missing paid orders and lock wait timeouts in Guest Checkout
Update on Jul 28 2020 The issue was not completely fixed in internal Jira tickets MC-29335 and MC-29206.
That solution looks like not completely fixes a problem, as sales rule usage update still potentially causes deadlock under high load when a lot of concurrent orders are placed at the same time. The only way to fix this problem is by introducing the append-only event log of sales rule counter adjustment log and using cronjob to update sales rule table.
https://github.com/magento/magento2/issues/25862#issuecomment-602170384 https://github.com/magento/magento2/issues/25862#issuecomment-665347649
During peak sales hour, one of my customers lost 400+ Magento orders but had charged credit card transactions at the PSP.
At first, I thought PSP implementation was a reason for missing orders, but after the investigation, it turned out this a result of a pull request being merged 2 years ago in Guest Place Order API endpoint of Magento related to this issue #6363.
That change added beginTransaction() and commit() around the whole place order process involving payment API calls and multi-entity save process ignoring that it results in lock contention on database level for an update of the same row in the same table (sales_rule). Although intentions were good (implementation to fix stock deduction issues), but it introduced huge performance bottleneck and rollbacks complete order history for paid and valid orders if wait-lock happens during the ordering process.
Preconditions (*)
- Magento 2.2.4+ or Magento 2.3.0+
- Enabled Guest Checkout
- Site-wide sales rule like “Free shipping above x amount”
Steps to reproduce (*)
- Stress test system with concurrent guest orders involving API payment PSP
Expected result (*)
- All paid payment transactions are saved to the database and visible in the admin panel
Actual result (*)
- A small fraction of orders is saved in the database while the majority of orders lost because of lock wait timeout.
Workaround
If you are using MSI it is possible to completely remove transactions and make it work in the same way as for logged-in user checkouts. Here is a patch that can be applied for “magento/module-checkout”:
Index: Model/GuestPaymentInformationManagement.php
<+>UTF-8
===================================================================
--- Model/GuestPaymentInformationManagement.php (date 1575083782000)
+++ Model/GuestPaymentInformationManagement.php (date 1575083782000)
@@ -98,33 +98,20 @@
\Magento\Quote\Api\Data\PaymentInterface $paymentMethod,
\Magento\Quote\Api\Data\AddressInterface $billingAddress = null
) {
- $salesConnection = $this->connectionPool->getConnection('sales');
- $checkoutConnection = $this->connectionPool->getConnection('checkout');
- $salesConnection->beginTransaction();
- $checkoutConnection->beginTransaction();
-
- try {
- $this->savePaymentInformation($cartId, $email, $paymentMethod, $billingAddress);
- try {
- $orderId = $this->cartManagement->placeOrder($cartId);
- } catch (\Magento\Framework\Exception\LocalizedException $e) {
- throw new CouldNotSaveException(
- __($e->getMessage()),
- $e
- );
- } catch (\Exception $e) {
- $this->getLogger()->critical($e);
- throw new CouldNotSaveException(
+ $this->savePaymentInformation($cartId, $email, $paymentMethod, $billingAddress);
+ try {
+ $orderId = $this->cartManagement->placeOrder($cartId);
+ } catch (\Magento\Framework\Exception\LocalizedException $e) {
+ throw new CouldNotSaveException(
+ __($e->getMessage()),
+ $e
+ );
+ } catch (\Exception $e) {
+ $this->getLogger()->critical($e);
+ throw new CouldNotSaveException(
__('An error occurred on the server. Please try to place the order again.'),
- $e
- );
- }
- $salesConnection->commit();
- $checkoutConnection->commit();
- } catch (\Exception $e) {
- $salesConnection->rollBack();
- $checkoutConnection->rollBack();
- throw $e;
+ $e
+ );
}
return $orderId;
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 32
- Comments: 45 (19 by maintainers)
Hello @IvanChepurnyi @PascalBrouwers @rhoerr
The internal Magento team is working on this issue right now. Fix is almost done in the scope of internal Jira tickets
MC-29335andMC-29206. Currently, all development work is done and its on the QA stage.I will keep updating this Issue with commits as soon as changes will be pushed to the public
magento/magento2repoThat solution looks like not completely fixes a problem, as sales rule usage update still potentially causes deadlock under high load when a lot of concurrent orders are placed at the same time. The only way to fix this problem is by introducing the append-only event log of sales rule counter adjustment log and using cronjob to update sales rule table.
@convenient This issue might have caused dozens of adverse effects since the introduction of transactions in guest order placement API. Your deadlock most probably a combination of both factors, this issue and
REPEATABLE-READtransaction mode in MySQL when the table gets locked for inserting new records if any transaction has a locking gap in the index for new increment value. I use for all merchantsREAD-COMMITEDmode that does not have any gap locks, so even if such problem exists inserting to table is possible.This is not compatible with EE 2.4.0 Please provide a patch which is compatible with EE 2.4.0
On Thu, Feb 11, 2021 at 4:12 AM Magento Quality Patches < notifications@github.com> wrote:
– Regards, Sodhan Manandhar
As soon as you fix this one
The transaction management in the guest orders payments has always been a huge pain in the … since we discovered it too.
In my opinion, you can’t simply wrap a whole placeOrder with a transaction where you allow for any third party extension to run on observers and roll back in case anything goes wrong allowing an ecommerce platform to fail in such a spectacular way.
Using lots of thirdparty extensions, I’ve seen many different situation where an undetected conditions in some code led to a _isRolledBack set to true in Magento\Framework\DB\Adapter\Pdo\Mysql, all subsequent tasks blocked and the previous ones to be reverted back.
No doubt for me this is a design bug that went hugely unnoticed.
The quickets “patches” are to fix all third party extensions causing that, or in the case of lots of deadlocks appearing to scale up platform specs.
Another option, is to temporarily dump all data to disk (dirty option, but as a bridge because of locked writing to database) when a rollback happens and subsequently analyse the reason of failure. All failed payment attempts on guest carts are treated in the same way by Magento, so, the reason in the exception at Magento\Checkout\Model\GuestPaymentInformationManagement.
But, again, “patches” for a problem that is still there.
Experiencing the same problem on Magento 2.4.4 - Customer placed an order, our payment provider (Stripe) charged the customer. The payment references the Magento order number, however that order number doesn’t exist
Confirming that this happens relatively often on EE 2.4.2 as well. The payment processor finish the transaction, but the quote for the given order doesn’t exists (anymore?), thus the order creation cannot be finished, leading to the customer being charged but order never created thus cannot be fulfilled. Its annoying because there is no specific rule to debug, it can happen with registered users as well with guest accounts, but no clear reason why exactly it happens. Not to mention that it ruins the reputation of the website…
@zcuric We have faced same issue with our client as well and as we can see we didn’t found issue in normal days but when there was a sale and there was lot of traffic on site then we faces this issue that Authorize.net have paid orders but we didn’t found those orders in system . So is there any solution for this ?
@magento-quality-patches
@zcuric ^^^
…lost 3% of payments…and tons of hours… will this fix be implemented in Magento 2.4.2? i cant find it in the release notes.
to be clear: I do have magento 2.4.1
Remark from Gene commerce, who are responsible for the bundlked braintree plugin:
We have seen numbers of merchants who faced the same issue where the Orders do not appear in Magento but they appear in Braintree Transaction. We had spent numbers of hours on this issue last year and at the end, it was found that this issue was from Magento end. We contacted Magento and they informed that this issue was fixed and fix is available from Magento 2.4 onwards.
@sodhancha The internal team resolved the issue - async processing for the coupon/rule usage was implemented. PR with the solution will be delivered in the 2.4-develop branch during the next 2 weeks.
Internal team start working on this issue