sequelize: Memory leak using findAll method

Issue Description

What are you doing?

Iterating over Postgres DB data via findAll method with limit and offset params. Here is the minimal reproducible code:

import { Model } from './models';

void (async () => {
  let currentOffset = 0;

  const queryData = async (offset = 0) => {
    await Model.findAll({
      limit: 10000,
      offset,
    });

    await queryData(currentOffset += 10000);
  };

  await queryData();
})();

What do you expect to happen?

The code runs fine and doesn’t cause memory leaks.

What is actually happening?

Node.js process grows in memory consumption over time and causes a memory leak and process exit.

Output here

Additional context

Here is the similar issue, which was closed by the bot.

Environment

  • Sequelize version: 6.3.5
  • Node.js version: 14.15.4
  • Operating System: Ubuntu 20.04
  • TypeScript version: 4.1.3

Issue Template Checklist

How does this problem relate to dialects?

  • I think this problem happens regardless of the dialect.
  • I think this problem happens only for the following dialect(s):
  • I don’t know, I was using PUT-YOUR-DIALECT-HERE, with connector library version XXX and database version XXX

Would you be willing to resolve this issue by submitting a Pull Request?

  • Yes, I have the time and I know how to start.
  • Yes, I have the time but I don’t know how to start, I would need guidance.
  • No, I don’t have the time, although I believe I could do it if I had the time…
  • No, I don’t have the time and I wouldn’t even know how to start.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 8
  • Comments: 33 (11 by maintainers)

Most upvoted comments

I ran some load testing.

Here’s a heap snapshot before, with Sequelize taking around 1.5MB

select1

And then here’s 6 hours after the load testing was over:

select

As you can see, Sequelize hasn’t released 114MB worth of memory (~48% of the Node process).

handleSelectQuery() seems to be the culprit.

Note: We are using Sequelize 6.6.2, Postgres 9.6, Node 14.16.0 and Ubuntu 16.0.4

I think I can confirm.

image image

I couldn’t manage to reproduce a memory leak. I queries /get 3443 times, each time it returned 57 items. The memory usage fluctuated between 19MB and 35MB while the queries were being run then once I stopped it fell back to 19MB

image

Exploring the heap, the only new elements in the heap related to sequelize after running the 3443 queries were Executing (default): SELECT "id", "userName", "email", "password", "createdAt", "updatedAt" FROM "Users" AS "User"; which was still in memory because it was used by chrome devtools. Clearing the devtools purged all of those strings from memory

Let me send you boilerplate repo with docker and details how you can reproduce them.

Actually, I rewrote it with while loop (e.g. while (hasMoreData) { do await stuff }, and now it seems to work fine. I believe @nawlbergs suggestion may be true but still want to know what the best approach here is.