node-fetch: MaxListenersExceededWarning: Possible EventEmitter memory leak detected.

Hi there,

I think there is a regression in version 3.0.

When I’m using Custom Agent (https) with keepAlive: true option and then just make a bunch of requests. Pretty soon such warnings show up:

(node:24083) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 data listeners added to [TLSSocket]. Use emitter.setMaxListeners() to increase limit MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 data listeners added to [TLSSocket]. Use emitter.setMaxListeners() to increase limit at _addListener (node:events:469:17) at TLSSocket.addListener (node:events:491:10) at TLSSocket.Readable.on (node:internal/streams/readable:876:35) at ClientRequest.<anonymous> (…/node_modules/node-fetch/src/index.js:316:10) at ClientRequest.emit (node:events:394:28) at tickOnSocket (node:_http_client:755:7) at onSocketNT (node:_http_client:815:5) at processTicksAndRejections (node:internal/process/task_queues:84:21)

The same thing with http, just listeners added not to TLSSocket but to just Socket.

Reproduction

Here is a really simple example.

import https from 'node:https'
import fetch from 'node-fetch'

const agent = new https.Agent({ keepAlive: true })

async function start () {
  for (let i = 1; i < 1000; i++) {
    await fetch('https://example.com', { agent })
  }
}

process.on('warning', e => console.warn(e.stack))

start().catch(e => console.error(e))

Steps to reproduce the behavior:

  1. Run this code with node-fetch@3 installed and you will start getting warnings.
  2. Install node-fetch@2 as a dependency, run the same code, and boom, no warnings.

Expected behavior

Version 2.6.2 does not produce any warning on the same code. Version 3.0.0 produces warnings.

Your Environment

I have been running this on node.js 14 (LTS) and 16 (latest) with the same results.

software version
node-fetch 3.0.0
node v14.17.6 & v16.9.1
npm 6.14.15 & 7.21.1
Operating System MacOS 11.5.2 (20G95) & Docker with alpine linux

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 5
  • Comments: 25 (4 by maintainers)

Commits related to this issue

Most upvoted comments

@jimmywarting The PR has been inactive for 2 months. Can you chose the other fix and get out a release to unblock anyone experiencing the issue?

Is it possible to get an update on this issue? We eagerly await a new version without the memory leak.

The undici project and we might have a problem where the body isn’t consumed https://github.com/nodejs/undici#garbage-collection

what would happend if you try out

import https from 'node:https'
import fetch from 'node-fetch'

const agent = new https.Agent({ keepAlive: true })

async function start () {
  for (let i = 1; i < 1000; i++) {
    const res = await fetch('https://example.com', { agent })
    for await (const chunk of res.body) {} // do nothing
  }
}

process.on('warning', e => console.warn(e.stack))

start().catch(e => console.error(e))

I’ve found the problem.

In fixResponseChunkedTransferBadEnding() two signals are added to the socket in every request, but they are never removed.

In #1474 I fix that problem by removing both signals on the socket’s close() event.

🎉 This issue has been resolved in version 3.2.5 🎉

The release is available on:

Your semantic-release bot 📦🚀

I took a look at both pull requests. Check my comments there. https://github.com/node-fetch/node-fetch/pull/1325 https://github.com/node-fetch/node-fetch/pull/1474

I think https://github.com/node-fetch/node-fetch/pull/1474 by @felipec should be accepted. It does the correct things.

Could you make a release with this change?

@tamtamchik @kristoffer-zliide I pushed up a change to remove the fix when keep-alive is enabled. Please give it a try when you have a moment: https://github.com/tekwiz/node-fetch/tree/fix/chunked-encoding-keepalive

After some experimentation, it turns out that this solution is not compatible with keep-alive. Because the socket is the only place that can be used to verify that a chunked transfer has properly finished, the only option is to disable fixResponseChunkedTransferBadEnding when keep-alive is enabled. We should also advise users in the README that they need to be sure to implement timeouts with keep-alive to avoid the kinds of issues described in #1055 and #968. I can work on a change for this tomorrow.

We just spent more than a week upgrading our code base to ESM and can finally upgrade to node-fetch 3, but was immediately hit by this issue.

@jimmywarting: while not consuming the body is probably causing leaks, I don’t think this issue is related to that. https://github.com/node-fetch/node-fetch/commit/51861e98a8f87e0905e71bb101b506f9512a9d7f#diff-bfe9874d239014961b1ae4e89875a6155667db834a410aaaa2ebe3cf89820556R310 looks fundamentally suspicious; a close listener is attached to the socket on each request and only removed when the socket is aborted, but that doesn’t work well with keep alive, right? Also, is the data listener ever removed? Our problem seems to largely go away if we disable keep-alive. This guy seems to also be hit by it, even though they consume the body: https://stackoverflow.com/questions/67168047/node-js-memory-leak-am-i-doing-something-wrong-or-is-it-a-problem-with-a-packag