undici: Leaking array buffers

Not sure if this is a user, undici, node or V8 bug but I have case where I have lot’s of ArrayBuffers allocated and never freed in our Node based proxy:

{rss: 4519784448, heapTotal: 25522176, heapUsed: 20927688, external: 722839852, arrayBuffers: 721332770}

@mcollina @addaleax any hints on how to debug this?

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 15 (15 by maintainers)

Commits related to this issue

Most upvoted comments

My current guess it that it has something to do with pause/resume multiple times. This seems to be growing arrayBuffer usage infinitiely:

'use strict'

const { Client } = require('..')
const { createServer } = require('http')
const { Readable } = require('stream')

const requests = 128

const server = createServer()

const limit = 1e9

server.on('request', (req, res) => {
  let bytesWritten = 0

  const interval = setInterval(() => {
    console.log(process.memoryUsage().arrayBuffers / 1e6, bytesWritten, bytesWritten / limit)
  }, 1e3)

  const buf = Buffer.allocUnsafe(16384)
  new Readable({
    read () {
      bytesWritten += buf.length
      this.push(buf)
      if (bytesWritten >= limit) {
        clearInterval(interval)
        this.push(null)
      }
    }
  }).pipe(res)
})

server.listen(0, () => {
  const client = new Client(`http://localhost:${server.address().port}`, {
    pipelining: 1
  })

  for (let n = 0; n < requests; ++n) {
    client.request({ path: '/', method: 'GET', opaque: 'asd' }, (err, data) => {
      if (err) {
        console.error(err)
      }
      data.body
        .resume()
        .on('data', () => {
          data.body.pause()
          setTimeout(() => data.body.resume(), 1e3)
        })
    })
  }
})

`Client.request``

I think it might be broken back pressure.

Seem to have the same problem on a totally different service which recently started to use undici

14.8, HTTP

Happens at multiple locations with different node versions.

I see that V8 recently did something with GC of ArrayBuffers. Maybe coincidence…