magento2: Large configurable products or bundles do not show up in product listings with ES

Preconditions (*)

1. Magento 2.3.5 or Magento 2.4.0
2. Any elastic search version

Steps to reproduce (*)

1. Create a bundle or a configurable product with a few hundred child products
2. Attach the bundle or configurable to a visible frontend category
3. Make sure that the total summed amount of characters in all child product names exceeds 32766 characters
4. Make sure that Elasticsearch is configured as the search engine

Expected result (*)

1. The product is visible in the category and search.

Actual result (*)

1. The product is not visible in the category or search

---

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

Tagged it as S0 since on 2.4 there is no workaround anymore: the MySQL search engine is gone

Cause of the problem:

See the Elasticsearch logs for the cause of the problem:

[2020-10-14T13:04:22,816][DEBUG][o.e.a.b.TransportShardBulkAction] [di-u1v-] [magento2-prod_product_3_v4418][0] failed to execute bulk item (index) index {[magento2-prod_product_3_v4418][document][13421], source[n/a, actual length: [106.5kb], max length: $
java.lang.IllegalArgumentException: DocValuesField "name.sort_name" is too large, must be <= 32766

That explains why only large products go missing: child product names are imploded in that field, so if there are too many child products the 32766 field size limit will be hit.

Place in the code that causes this: https://github.com/magento/magento2/blob/fa6d291646b1ca0ebc0ebd3798ec8aaae634d486/app/code/Magento/Elasticsearch/Model/Adapter/BatchDataMapper/ProductDataMapper.php#L298

I drafted a quick patch that caps the sortable field on 32766 characters, you can apply it to magento/module-elasticsearch in your composer.json. Have not yet found any side effects.

Patch: fix-for-too-long-sortable-attributes.txt