apm-agent-nodejs: memory leak problem

We have add the apm agent with express , our codes is:

require('elastic-apm-node').start({
    serviceName: 'activity-service',
    serverUrl: 'http://es-apm-svc.infra.svc.cluster.local:8200',
    transactionSampleRate: 0.2,
    transactionMaxSpans: 32,
    flushInterval: 5,
    serverTimeout: 10,
    maxQueueSize: 50,
})

global.Promise = require('bluebird')

const PORT = process.env.PORT || 7000
const app = require('./app')

app.listen(PORT, function () {
    console.info('Node app is running, port:', PORT)
})

But we find the memory usage is increasing until hit the memory limit .

We also put the same code in Koa applications , but the problem is still existed .

We find the issues elastic/apm-agent-nodejs#243 But @Qard said that it has been fixed ?

We also try to change our config like ：

require('elastic-apm-node').start({
    serviceName: 'activity-service',
    serverUrl: 'http://es-apm-svc.infra.svc.cluster.local:8200',
    transactionSampleRate: 0.2,
    transactionMaxSpans: 32,
    flushInterval: 5,
    serverTimeout: 10,
    maxQueueSize: 50,
    sourceLinesErrorAppFrames: 0,
    sourceLinesErrorLibraryFrames: 0,
})

But it’s not work .

Our dependencies is :

    "mongoose": "4.9.9",
    "express": "4.12.4",
    "bluebird": "3.1.1",

and the applications run in kubernets .

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 39 (16 by maintainers)

Most upvoted comments

Hi. It appears to be ok for us too ! Thanks !

Thetortuga on Nov 21, 2018

yea. it seems to be fixed. we have only tested it in our performance testing environment. hopefully we’ll get it to our production environment soon. thanks @watson @Qard

junereycasuga on Nov 15, 2018

Thanks for being patient with us on this. It’s been a complicated process to debug and narrow down the culprit.

But we now think that we have a fix. We just released version 1.14.3 of the agent which fixes the memory leak that we identified. Please try it out and let us know if it also fixed it for you 😃

watson on Nov 13, 2018

Hello, We are experiencing the same kind of problem on our production server too.

Like joway, we use these packages : “bluebird”: “^3.4.6”, “request”: “^2.81.0”, “request-promise”: “^4.2.1”,

Also using these: “hapi”: “^17.6.0”, “apollo-engine”: “^0.5.4”, “apollo-server-hapi”: “^2.1.0”, “elastic-apm-node”: “^1.12.0” “mysql2”: “^1.4.1”, “sequelize”: “^4.38.0”, “dataloader”: “^1.3.0”,

bluebird override Promise: global.Promise = require(‘bluebird’).getNewLibraryCopy()

node version 8.12.0.

Agent is loaded with default configuration. A custom transaction is created each time a graphQl resolver is access. Some span are created using sequelize model hooks (before and after). The transaction is ended at the end of the graphQl resolver. The status is set to “success”. The custom context is set with the GraphQl query. Multiples transaction are used in parrallel.

After less than 24h of uptime: heap was 328mb ~230mb was from Transaction (639 objects) The transactions Map had ~25k entries (with multiple reference to Transaction objects). Transaction were of type custom but also request.

After analysing some heap dump of the process, Some Transaction are not released from the Map that is in the async-hook.js file. There’re a lot of key with null value and a lot of key referencing the same Transaction object (same memory address) that are staying in the Map eternally (never destroyed nor garbage).

From what I understand, some PROMISE hooks type are init in the async-hook.js but are not destroyed, this might be link to promise cache used in dataloader package (storing a Promise in a map or a variable) or / and in local Promise cache, that store resolved Promise for reused OR PromiseWrap. it seems that some internal Transaction (intercepted http requests) are also not destroyed and staying.

A way to release these PROMISE might be to use the promiseResolve callback of async hooks (tried it on a local solution and seems to do the job but can’t really tell as it’s not in production) ? and this might move the leak somewhere else.

Note that, for now I can’t reproduce on a test project, so this might also be linked in wrong use of apm client in our production software.

Hope this can help.

TitTurtle on Nov 2, 2018

Thanks @PestoP. Yes it would be super useful for us to have a small app where the problem can be reproduced 👍

watson on Jul 4, 2018

Hi, @watson

we using the latest version (1.6.0)
no , it’s the first line above all other code and we don’t use any other apm
yes , it’s still leak when i set captureSpanStackTraces= false

joway on Jun 15, 2018