mongoose: How to release Mongoose model from memory? (memory leak)
How to release the model from memory after accessing it? I’m using global.gc()
with --expose-gc
option to see that this is eating my memory.
The following code creates 10000 collections:
// mongoose connection
var db = mongoose.createConnection(...);
// amount of collections
var amount = 10000;
// create collections (100/per second)
var sync = async.queue(function(n, cb) {
var schema = new mongoose.Schema({
data: mongoose.Schema.Types.Mixed
});
var collection = 'model_'+n;
var model = db.model(collection, schema);
setTimeout(function() { cb(); }, 10);
}, 1);
// push to queue
for(var i=0; i<amount; i++) {
sync.push(i);
}
// done
sync.drain = function(err) {
console.log('all '+amount+' models done');
};
// garbage collector (every second)
setInterval(function() {
try { global.gc(); } catch(gcerr) { }
}, 1000);
The memory usage is increasing as collections are created and the memory is never released:
1 - Memory used: 30 MB
2 - Memory used: 36 MB
3 - Memory used: 42 MB
4 - Memory used: 48 MB
5 - Memory used: 54 MB
6 - Memory used: 61 MB
7 - Memory used: 65 MB
8 - Memory used: 71 MB
9 - Memory used: 77 MB
10 - Memory used: 82 MB
all 10000 models done
11 - Memory used: 86 MB
12 - Memory used: 86 MB
Any ideas how to purge the model from the memory manually without closing the connection?
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 4
- Comments: 27 (3 by maintainers)
These lines did the trick and got all my memory back:
@weichtg3 that’s an interesting use case, thanks for elaborating. We’ll look into whether we can get rid of
_originalSchema
.Leaving this snippet here in case anyone else finds it useful. It’s mostly just a variation of what’s already been discussed.
We create our models directly on connection objects, and our Jest tests were quickly chewing through available memory. We were able to programmatically remove references to all models/connections with this code in an
afterAll
hook:Try:
The other solution that works perfectly for me is to not use global moongoose, instantiated with
var moongoose = require('moongoose');
use the following instead:
so that you can manage references to Moongoose() instances just like any other objects and it will be garbage collected when needed.
I know this is a Closed issue, but I would like to add what we discovered recently.
BLUF: if you are repetitively building a model using model.discriminator(), the schema is getting exponentially bigger due to schema._originalSchema = schema.clone().
Against recommendations, we are doing open/close with each request instead of caching connections. This is because our multi-tenancy adds an additional level of security in that each tenant gets their own database with their own set of user credentials. Mongoose useDb() did not seem to support that, nor any other mod/hack I could find on Google. (Note: I have since modified a library called mobgoose to cache down to that level and may use it going forward).
We initially tried deleting everything under the sun as suggested above to no avail. If you use connection.model(), we could get things to clean themselves up. However, we were using model.discriminator(). Looking inside the mongoose library code and seeing how the Model.discriminator() and Schema prototypes work, we see a lot of clone() calls. The exponential memory leak seems to come from schema._originalSchema = schema.clone().
Our current solution, if we are to continue with open/close per request, is to re-generate the Schema on each call too. Mucking around inside the Schema to delete individual properties seemed too risky, and this gets us past the current issue while we explore a better solution going forward.
@QuotableWater7 do you have a repo example of this with jest?
@vkarpov15 should mongoose provide a helper to cleanup models from memory?
any other place where memory can leak from mongoose?
related to https://github.com/facebook/jest/issues/6787