magento2: Elasticsearch pagination does not work on 2.3.3

Preconditions (*)

Magento version 2.3.3 Elasticsearch 6.0 Stores->Configuration->Catalog->Catalog->Catalog Search->Search Engine = Elasticsearch 6.0+

Steps to reproduce (*)

NEW steps used for REOPEN on 2.4-develop -> https://github.com/magento/magento2/issues/25038#issuecomment-588881091

Set Elasticsearch 6.0 as search engine
bin/magento indexer:reindex
Go to category page that have more than one page
Navigate to second page

Expected result (*)

Products from the second page are displayed.

Actual result (*)

No products found.

More details

I’ve digged up in Magento’s ES implementation and found really strange thing. Magento is requesting elasticsearch with from and size parameters that limits search results but additionally there is another limit in \Magento\Eav\Model\Entity\Collection\AbstractCollection::_loadEntities $this->getSelect()->limitPage($this->getCurPage(), $this->_pageSize);

Result is as follow for second page of category listing:

Elasticsearch returns second page of products limited by from and size and magento is adding product IDs to the collection. Example ElasticSearch request:

POST /magento2_product_1/document/_search
{ 
   "from":9,
   "size":9,
   "stored_fields":[ 
      "_id",
      "_score"
   ],
   "sort":[ 
      { 
         "position_category_5":{ 
            "order":"asc"
         }
      }
   ],
   "query":{ 
      "bool":{ 
         "must":[ 
            { 
               "term":{ 
                  "category_ids":"5"
               }
            },
            { 
               "terms":{ 
                  "visibility":[ 
                     "2",
                     "4"
                  ]
               }
            }
         ]
      }
   },
   "aggregations":{ 
      "price_bucket":{ 
         "extended_stats":{ 
            "field":"price_0_1"
         }
      },
      "category_bucket":{ 
         "terms":{ 
            "field":"category_ids",
            "size":500
         }
      },
      "manufacturer_bucket":{ 
         "terms":{ 
            "field":"manufacturer",
            "size":500
         }
      },
      "color_bucket":{ 
         "terms":{ 
            "field":"color",
            "size":500
         }
      }
   }
}

Collection executes _loadEntities and add yet another limit which goes way out of collection s range and this leads to 0 results found.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 6
Comments: 29 (8 by maintainers)

Commits related to this issue

25038 Only set collection to toolbar once in ListProduct block Fix for: https://github.com/magento/magento2/issues/25038 When using ElasticSearch, pagination does not work in some cases if paginat... — committed to wigman/magento2 by wigman 4 years ago

Most upvoted comments

Hi @piotrusin. Thank you for your report. Seems Issue is already reported please review at #24047

krishprakash on Oct 14, 2019

Hello everyone! We are closing this issue as @piotrusin temporary fix is working fine: #25038 (comment) #25038 (comment).

@engcom-Charlie I personally find it unacceptable to close a ticket if there’s a workaround, without a real fix. Does this remain on an internal backlog, or will this bug remain in core?

From the people experiencing the issue: could someone check the following? In \Magento\Catalog\Block\Product\ListProduct::configureToolbar, wrap $toolbar->setCollection($collection); with a check to see if there’s already a collection set, as follows:

\Magento\Catalog\Block\Product\ListProduct::configureToolbar:520

        if(!$toolbar->getCollection()) {
            $toolbar->setCollection($collection);
        }

I my case, pagination breaks when $toolbar->setCollection() is called multiple times. If anyone can confirm this is not unique to my project, I will create a PR to core.

wigman on Feb 18, 2020

I can confirm, @piotrusin temporary fix is working fine:

etc/frontend/di.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd">
    <type name="Magento\Catalog\Model\ResourceModel\Product\Collection">
        <plugin name="dexanet-catalog-plugin-collection-elastic-bug" type="Dexanet\Catalog\Plugin\Collection"
                sortOrder="1"/>
    </type>
</config>

Plugin\Collection

public function around_loadEntities(\Magento\Catalog\Model\ResourceModel\Product\Collection $subject, callable $proceed, $printQuery = FALSE, $logQuery = FALSE) {
    $pageSize = $subject->getPageSize();
    $subject->setPageSize(0);
    $result = $proceed($printQuery, $logQuery);
    $subject->setPageSize($pageSize);
    return $result;
  }

sedax90 on Nov 11, 2019

There is a patch for this, but it will also cause the pager to stop showing: example Instead of “Items 1-9 of 13” just “13 Items”

domeglic on Oct 14, 2019

Same as @ArthurSCD. I have applied the patch, the issue still there…

@ArthurSCD have you found a solution? @piotrusin May you be more explicit by adding before/after _loadEntities?

Please reopen this issue.

facetimer on Nov 6, 2019

I’ve handled it by myself by adding before/after _loadEntities plugin on Magento\Catalog\Model\ResourceModel\Product\Collection that sets pageSize to 0 and resets it back after querying if using elasticsearch. This does not break pager.

piorus on Oct 14, 2019

@simonmaass Did the pagination still work after applying the patch? When I apply the patch it breaks it.

Before: magento pagination counter before After: magento pagination counter after

domeglic on Oct 16, 2019

Hello @piotrusin

Thank you for contribution and collaboration!

The hotfix for this Issue is published and available for download on magento.com portal here https://magento.com/tech-resources/download#download2331

Could you please apply patch and confirm that the issue is fixed?

sdzhepa on Oct 15, 2019