google-cloud-node: pubsub: app with default settings eventually runs out of memory (but does not crash)
Environment details
- OS: alpine 3.6
- Node.js version: 9.8.0
- npm version: 5.6.0
- google-cloud-node version:
@google-cloud/pubsub@0.16.5
Steps to reproduce
Run the following on GKE, pump 30 million messages into pubsub:
const PubSub = require('@google-cloud/pubsub')
const express = require('express')
console.log('Node starting up')
PROJECT_ID = 'some-project'
TOPIC_ID = 'some-topic'
SUBSCRIPTION_ID = 'some-node-subscription'
const pubsub = new PubSub({
projectId: PROJECT_ID,
})
const topic = pubsub.topic(TOPIC_ID)
topic.createSubscription(SUBSCRIPTION_ID, function (err, subscription) {
if (err) {
console.log('Node subscription already created, moving on')
}
})
const subscription = topic.subscription(SUBSCRIPTION_ID)
subscription.on('error', function (err) {
console.error('Node error!', err)
})
function onMessage(message) {
console.log(`Node received ${message.id} ${message.data}`)
setTimeout(function () {
console.log('Node acking')
message.ack()
}, 5000)
}
console.log('Node receiving..')
subscription.on('message', onMessage)
// because I have no idea how to wait forever..
const app = express()
app.listen(3000, () => console.log('Example app listening on port 3000!'))
// Remove the listener from receiving `message` events.
// subscription.removeListener('message', onMessage)
Once the app runs out of memory (1.5GB), reception slows from ~20k/min to something like 200/min:
(removing the timeout, it still runs out of memory)
Let me know if I’m missing something silly here.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 22 (22 by maintainers)
@jadekler hooray! Thank you for all your help, I definitely would not have resolved this issue as quickly without it.
@callmehiphop Happy to report that I saw no problems over the weekend, and that the node pubsub library chugged through 67M messages at a rate of 61,000 acks / minute.