sequelize: Memory leak using findAll method
Issue Description
What are you doing?
Iterating over Postgres DB data via findAll
method with limit
and offset
params. Here is the minimal reproducible code:
import { Model } from './models';
void (async () => {
let currentOffset = 0;
const queryData = async (offset = 0) => {
await Model.findAll({
limit: 10000,
offset,
});
await queryData(currentOffset += 10000);
};
await queryData();
})();
What do you expect to happen?
The code runs fine and doesn’t cause memory leaks.
What is actually happening?
Node.js process grows in memory consumption over time and causes a memory leak and process exit.
Output here
Additional context
Here is the similar issue, which was closed by the bot.
Environment
- Sequelize version: 6.3.5
- Node.js version: 14.15.4
- Operating System: Ubuntu 20.04
- TypeScript version: 4.1.3
Issue Template Checklist
How does this problem relate to dialects?
- I think this problem happens regardless of the dialect.
- I think this problem happens only for the following dialect(s):
- I don’t know, I was using PUT-YOUR-DIALECT-HERE, with connector library version XXX and database version XXX
Would you be willing to resolve this issue by submitting a Pull Request?
- Yes, I have the time and I know how to start.
- Yes, I have the time but I don’t know how to start, I would need guidance.
- No, I don’t have the time, although I believe I could do it if I had the time…
- No, I don’t have the time and I wouldn’t even know how to start.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 8
- Comments: 33 (11 by maintainers)
I ran some load testing.
Here’s a heap snapshot before, with Sequelize taking around 1.5MB
And then here’s 6 hours after the load testing was over:
As you can see, Sequelize hasn’t released 114MB worth of memory (~48% of the Node process).
handleSelectQuery() seems to be the culprit.
Note: We are using Sequelize 6.6.2, Postgres 9.6, Node 14.16.0 and Ubuntu 16.0.4
@ephys here is a repo and way to reproduce https://github.com/hironate/sequelize-memory-leak
I think I can confirm.
I couldn’t manage to reproduce a memory leak. I queries
/get
3443 times, each time it returned 57 items. The memory usage fluctuated between 19MB and 35MB while the queries were being run then once I stopped it fell back to 19MBExploring the heap, the only new elements in the heap related to sequelize after running the 3443 queries were
Executing (default): SELECT "id", "userName", "email", "password", "createdAt", "updatedAt" FROM "Users" AS "User";
which was still in memory because it was used by chrome devtools. Clearing the devtools purged all of those strings from memoryLet me send you boilerplate repo with docker and details how you can reproduce them.
Actually, I rewrote it with
while
loop (e.g.while (hasMoreData) { do await stuff }
, and now it seems to work fine. I believe @nawlbergs suggestion may be true but still want to know what the best approach here is.