nodejs-pubsub: Error [ERR_STREAM_WRITE_AFTER_END]: write after end

Environment details

  • OS: Google Cloud Container-Optimized OS
  • Node.js version: 10.17.0
  • npm version: 6.12.0
  • @google-cloud/pubsub version: 1.x (1.1.5, tried reverting down to 1.0.0, same error, currently using 0.21.1)

Steps to reproduce

Unclear so far but have been seeing it after publishing 10k+ messages to various topics. Will get back with more details when we can but if anyone else is seeing this to confirm this isn’t a config issue let us know.

We are using our own wrapper around pubsub which you can find here: https://github.com/starchup/google-pubsub-wrapper, extremely bare bones

Error stacktrace


E 2019-11-19T16:49:45.701927239Z events.js:174
E 2019-11-19T16:49:45.702039214Z       throw er; // Unhandled 'error' event
E 2019-11-19T16:49:45.702047675Z       ^
E 2019-11-19T16:49:45.702054717Z 
E 2019-11-19T16:49:45.702060707Z Error [ERR_STREAM_WRITE_AFTER_END]: write after end
E 2019-11-19T16:49:45.702068184Z     at writeAfterEnd (_stream_writable.js:248:12)
E 2019-11-19T16:49:45.702074930Z     at ClientHttp2Stream.Writable.write (_stream_writable.js:296:5)
E 2019-11-19T16:49:45.702081282Z     at filterStack.sendMessage.then.message (/node_modules/@grpc/grpc-js/build/src/call-stream.js:372:34)
E 2019-11-19T16:49:45.702088817Z     at process._tickCallback (internal/process/next_tick.js:68:7)
E 2019-11-19T16:49:45.702094878Z Emitted 'error' event at:
E 2019-11-19T16:49:45.702121372Z     at errorOrDestroy (internal/streams/destroy.js:107:12)
E 2019-11-19T16:49:45.702164237Z     at onwriteError (_stream_writable.js:436:5)
E 2019-11-19T16:49:45.702169699Z     at onwrite (_stream_writable.js:461:5)
E 2019-11-19T16:49:45.702174990Z     at /node_modules/async-listener/glue.js:188:31
E 2019-11-19T16:49:45.702180323Z     at process._tickCallback (internal/process/next_tick.js:63:19)

Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 34 (2 by maintainers)

Commits related to this issue

Most upvoted comments

We are actually able to go up to v0.29.1 without this issue happening so it seems possible that v0.30.0 introduced it (or exposed it) - https://github.com/googleapis/nodejs-pubsub/releases/tag/v0.30.0

Maybe one of the changes to grpc extensions & dependencies has something to do with it. Will try to identify more clearly where it gets thrown tomorrow

Yeah, and this happened in 0.30.0, too:

https://github.com/googleapis/nodejs-pubsub/pull/658

We’re looking at just reverting back to the gRPC C++ bindings for now. I’m talking with some folks on Monday before we commit to that. In the meantime, it’s possible to manually switch to them and see if it helps:

const {PubSub} = require('@google-cloud/pubsub');
const grpc = require('grpc');
const pubsub = new PubSub({grpc});

@shufengh It looks like 2.5.0 is the one that pulled in the new gax version explicitly. Maybe give that one a try?

https://github.com/googleapis/nodejs-pubsub/pull/1085

We are seeing this in v2.3.0 as well

grpc-js has never had a dependency on readable-stream, so I don’t know where that could be coming from.

Regardless, a similar issue was filed in the google-gax repository and it looks like grpc/grpc-node#1436 will address it.