magento2: Varnish "Connection reset by peer" error when large catalog is reindexed on schedule

Addition information for Re-Open

The Issue was reopened based on several complaints that it was not properly fixed and it still can be reproduced on the 2.2.8 and 2.3.1 For more details and explanation see next comments:

Preconditions

Magento EE 2.1.5 with ~10K sample products
Server is running PHP version 7.0.15
All indexes are configured to “UPDATE BY SCHEDULE”
Cron job is running on server. Cron error email address is configured with your email so you get notifications of errors.
Server is configured to use Varnish for Full Page Cache
Varnish is configured with http_req_hdr_len and http_req_size configured at their default values (per the documentation)

Steps to reproduce

In the admin, go to SYSTEM > Export and export a CSV of all ~10K products
Import the CSV back into Magento using the SYSTEM > Import interface

Expected result

The import will be processed without any errors.

Actual result

The cron will have an error when reindexing and will send an error like this to the cron email address:

PHP Notice: fwrite(): send of 8192 bytes failed with errno=104 Connection reset by peer in /var/www/stage/releases/20170213214641/vendor/zendframework/zend-http/src/Client/Adapter/Socket.php on line 376

The error above is triggered by line 375 of vendor/zendframework/zend-http/src/Client/Adapter/Socket.php

In order to determine what request was causing the error, I edited the Socket.php file to add logging code at line 375 to log large requests, along with a backtrace:

if (strlen($request) > 8000) {
    file_put_contents('/var/www/stage/current/var/log/connection_error.log', date('c') . PHP_EOL . print_r(debug_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS, 100), true), FILE_APPEND);
    file_put_contents('/var/www/stage/current/var/log/connection_error.log', date('c') . PHP_EOL . print_r($request, true), FILE_APPEND);
}

The requests that were causing errors were ~1.1MB in size, which exceeded the http_req_size default Varnish setting (which is 32k). While we could crank up that setting to allow the massive requests to be sent to Varnish, this would increase the Varnish memory footprint and is not an acceptable solution. Read this for context on the memory implications: https://www.varnish-cache.org/docs/4.0/reference/varnishd.html#http-req-size

Here is a preview of what the 1.1MB request contained:

PURGE / HTTP/1.1
X-Magento-Tags-Pattern: ((^|,)catalog_product_6574(,|$))|((^|,)catalog_product_6575(,|$))|((^|,)catalog_product_6576(,|$))|((^|,)catalog_product_6577(,|$))|((^|,)catalog_product_6578(,|$))|((^|,)catalog_product_6579(,|$))|((^|,)catalog_product_6580(,|$))|((^|,)catalog_product_6581(,|$))|((^|,)catalog_product_6582(,|$))|((^|,)catalog_product_6583(,|$))|((^|,)catalog_product_6584(,|$))|((^|,)catalog_product_6585(,|$))|((^|,)catalog_product_6586(,|$))|((^|,)catalog_product_6587(,|$))|((^|,)catalog_product_6588(,|$))|((^|,)catalog_product_6589(,|$))|((^|,)catalog_product_6590(,|$))|((^|,)catalog_product_6591(,|$))|((^|,)catalog_product_6593(,|$))|((^|,)catalog_product_6594(,|$))|((^|,)catalog_product_6595(,|$))|((^|,)catalog_product_6596(,|$))|((^|,)catalog_product_6597(,|$))|((^|,)catalog_product_6598(,|$))|((^|,)catalog_product_6599(,|$))|((^|,)catalog_product_6600(,|$))|((^|,)catalog_product_6601(,|$))|((^|,)catalog_product_6602(,|$))|((^|,)catalog_product_6603(,|$))|((^|,)catalog_product_6604(,|$))|((^|,)catalog_product_6605(,|$))|((^|,)catalog_product_6606(,|$))|((^|,)catalog_product_6607(,|$))|((^|,)catalog_product_6608(,|$))|((^|,)catalog_product_6644(,|$))|((^|,)catalog_product_6645(,|$))|((^|,)catalog_product_6646(,|$))|((^|,)catalog_product_6647(,|$))|((^|,)catalog_product_6648(,|$))|((^|,)catalog_product_6649(,|$))|((^|,)catalog_product_6650(,|$))|((^|,)catalog_product_6651(,|$))|((^|,)

In case it is useful for troubleshooting this issue, here is the backtrace logged by the logging code I added to the Socket.php file:

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 31 (9 by maintainers)

Commits related to this issue

#8815 * send multiple small purge request instead of one big request to avoid reaching the varnish http_req_hdr_len threshold. — committed to magento/magento2 by deleted user 7 years ago
MAGETWO-95276: Varnish "Connection reset by peer" error when large catalog is reindexed on schedule — committed to magento/magento2 by zakdma 5 years ago
Fixed splitting purge request chunks #8815 — committed to ageno/magento2 by wujku 4 years ago
Merge pull request #8815 from adobe-commerce-tier-4/Tier4-Kings-PR-03-05-2024 [Support Tier-4-Kings glo16746] 03.05.2024 Regular delivery of bugfixes and improvements — committed to magento/magento2 by dhorytskyi 4 months ago

Most upvoted comments

I did a PR there https://github.com/magento/magento2/pull/8919 but they can’t reproduce the issue and as a result closed the PR x)

You can adapt a patch from this https://patch-diff.githubusercontent.com/raw/magento/magento2/pull/8919.diff and apply it on your project.

Vedrillan on Aug 18, 2017

The solution is to replace line in https://github.com/magento/magento2/commit/485ad0b67779ea3a59c89daa64f0e290e3311657 : $formattedTags = explode('|', $tagsPattern); with: $formattedTags = array_filter(preg_split('/(?<=\))\|(?=\()/', $tagsPattern));

pczerkas on Jul 22, 2019

@joaolisboa @gigadesign1 @pczerkas Unfortunately, I cannot reproduce issue on 2.3-develop CE and 2.3-develop EE. Testing scenario:

Generate 10K products
Configure varnish v.4.1.1
Set indexers to “UPDATE BY SCHEDULE”
Run bin/magento cron:install
In the admin, go to SYSTEM > Export and export a CSV of all ~10K products
Import the CSV back into Magento using the SYSTEM > Import interface

Result: ✔️ Import is finished without errors. #8810issue

Go to Catalog->Product
Check products using checkbox ( in my case I used Select All for testing)
Click Actions->Update Attributes
Set Advanced Inventory->Stock Availability=Out of Stock
Click Save

Result: ✔️ Stock Status is updated without errors #8815issueMassAction

Could you take a look if my steps are correct?

engcom-Delta on Oct 17, 2019

I have the same issue and can confirm that the issue was not properly fixed.

We were experiencing the varnish cache to be fully flushed every time our warehouse updated the stocks of a larger number of products.

After some debugging I found this in the varnish log: ReqHeader X-Magento-Tags-Pattern: $))|((^|,)cat_p_769(,|$))|((^|,)cat_p_776(,|$))|((^|,)cat_p_768(,|$))|((^|,)cat_p_891(,|$))|((^|,)cat_p_97(,|$))|((^|,)cat_p_193(,|$))|((^|,)cat_p_949(,|$))|((^|,)cat_p_950(,|$))|((^|,)cat_p_312(,|$))|((^|,)cat_p_884(,|$))|((^|,)c VCL_Error ban(): Regex compile error: unmatched parentheses

Apparently the regex being incorrect results in the varnish cache being fully flushed.

I’m on Magento 2.3.1 and varnish 4.1.1

gigadesign1 on Jun 23, 2019