medusa: TypeORM produces bad query for /products route

Bug report

Describe the bug

The routes GET /products stop working properly when dealing with a large number of products (~100.000-200.000). The ProductService.listAndCount method has a temporary fix for typeorm’s query strategy mode, but it generates a peculiar SQL query that completely hangs the database due to a problematic subquery.

System information

Medusa version (including plugins): 1.11.0 Node.js version: 18.16.0 Database: PostgreSQL Operating system: Ubuntu 22.04 Browser (if relevant): Google Chrome

Steps to reproduce the behavior

  1. Add plenty of products (~100-200 thousands)
  2. GET /admin/products?offset=0&limit=1

Expected behavior

Modify productRepo.findWithRelationsAndCount in ProductService.listAndCount to prevent it from generating incorrect queries.

Code snippets

Example of query:

select
    distinct "distinctAlias"."product_id" as "ids_product_id",
    "distinctAlias"."product_created_at"
from
    (
    select
        "product"."id" as "product_id",
        "product"."created_at" as "product_created_at"
    from
        "product" "product"
    left join "product_category_product" "product_categories" on
        "product_categories"."product_id" = "product"."id"
    left join "product_category" "categories" on
        "categories"."id" = "product_categories"."product_category_id"
    where
        "product"."deleted_at" is null
        ) "distinctAlias"
order by
    "distinctAlias"."product_created_at" desc,
    "product_id" asc
limit 1;

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Reactions: 1
  • Comments: 21 (11 by maintainers)

Most upvoted comments

I am testing on GET /admin/products?limit=12 as it seems to be what you was doing, the result is a list of 12 products including 600 variants with their prices etc. For the tool I am using https://www.npmjs.com/package/medusa-plugin-sentry which I have developed sometimes ago to help me monitor and investigate 😊

That seems a bit too much, you should have something like this. Maybe some more if you have more currencies but I wouldn’t expect that high. Most of them being parallelised when possible and some the prices are cached if already computed.

Also, I ll push something to remove unnecessary count

Screenshot 2023-05-30 at 16 04 13

When you add a lot of data in your db, the indexes might not have been updated properly yet and you might need to ask postgres to update the indexes data to take into account your newly added data especially when you add a lot of it (REINDEX maybe). It might be why at first it didn’t worked and then after some tests it ends up working as by selecting the data the indexes are updated

After several attempts, I managed to make it work. Even the entire route /admin/products is functioning. It is quite slow, but it works. Therefore, if we cannot influence TypeORM’s querying directly, perhaps adding indexes could help improve the performance.