bluebird: Memory leaking with Promise.map concurrency

What version of bluebird is the issue happening on? Bluebird 3.5.20 with bluebird-global 3.5.5
What platform and version? (For example Node.js 0.12 or Google Chrome 32) Node.js 8.9.
Did this issue happen with earlier version of bluebird? As far as I know

I have code that looks like this:

const aBunchOfIds = [ /* a few hundred thousand elements */ ]
Promise.map(aBunchOfIds, async id => {
    const anObj = await getFromDatastore(id);
    const anotherObj = await process(anObj);
    if (isSpecial(anotherObj)) { console.log('so special'); }
}, { concurrency: 1000 });

This dies because node runs out of heap space. However the following modification resolves the issue:

const aBunchOfIds = [ /* a few hundred thousand elements */ ]
for (let i = 0; i < aBunchOfIds.length; i += 10000) {
    const someIds = aBunchOfIds.slice(i, i + 10000);
    Promise.map(someIds, async id => {
        const anObj = await getFromDatastore(id);
        const anotherObj = await process(anObj);
        if (isSpecial(anotherObj)) { console.log('so special'); }
    }, { concurrency: 1000 });
}

So it seems that Promise.map doesn’t free the memory used by its promises until all promises have returned, even when a concurrency is specified.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 7
Comments: 15 (1 by maintainers)

Most upvoted comments

I’m seeing memory-leak issues that are very similar, if not the same.

Came up with the following test-case which shows the issue clearly:

running below code as-is will result in OOM in a couple of seconds
running code with then() added (see below) will allow GC to free up mem.

Lodash 3.5.2


/*
 Below gives an OOM in a couple of seconds. 
 Adding the commented-out then() solves it. Why?
 */
const Promise = require("bluebird");
const obj = {a: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"};
return Promise.map(new Array(100000), () => {
  return Promise.resolve()
  .then(() => new Array(10000).map(() => Object.assign({},obj))) //consume some mem
  // .then(() => { //passing in an argument here, makes the OOM-issue reappear
  //   //adding this then(), will allow objects to be GC'ed, thus solving OOM-issue. WHY?
  // });
}, { concurrency: 10 })

It seems that since the result isn’t passed in as argument to the last then() the object can be freed from mem. This seems to suggest that not only the promises themselves but all objects created inside those promises are kept locked in mem until the entire Promise.map is finished.

0xgeert on Feb 2, 2019

That… makes no sense 😅

benjamingr on May 7, 2019

Just a heads up that I got “Maximum call stack size exceeded” errors until I finally downgraded this package to 3.5.2

I think it could be related to this issue.

krunkosaurus on Jan 3, 2019

Seems to me that as long as memory usage is contained within the callback it gets GCed correctly: https://jsfiddle.net/w03ykabx/3/

spion on Nov 2, 2018