grpc-node: Express App crashing citing "Stream Removed"
Apologies in advance if this is on my end, struck out at StackOverflow and another resource using this package similarly.
I saw similar issues posted here in my search that were upstream resolutions.
Problem description
Express App crashing citing “Stream Removed”
events.js:167
throw er; // Unhandled 'error' event
^
Error: 2 UNKNOWN: Stream removed
at Object.exports.createStatusError (/home/justin/singles-api/node_modules/grpc/src/common.js:91:15)
at ClientReadableStream._emitStatusIfDone (/home/justin/singles-api/node_modules/grpc/src/client.js:233:26)
at ClientReadableStream._receiveStatus (/home/justin/singles-api/node_modules/grpc/src/client.js:211:8)
at Object.onReceiveStatus (/home/justin/singles-api/node_modules/grpc/src/client_interceptors.js:1272:15)
at InterceptingListener._callNext (/home/justin/singles-api/node_modules/grpc/src/client_interceptors.js:568:42)
at InterceptingListener.onReceiveStatus (/home/justin/singles-api/node_modules/grpc/src/client_interceptors.js:618:8)
at /home/justin/singles-api/node_modules/grpc/src/client_interceptors.js:1029:24
Emitted 'error' event at:
at ClientReadableStream._emitStatusIfDone (/home/justin/singles-api/node_modules/grpc/src/client.js:234:12)
at ClientReadableStream._receiveStatus (/home/justin/singles-api/node_modules/grpc/src/client.js:211:8)
[... lines matching original stack trace ...]
at /home/justin/singles-api/node_modules/grpc/src/client_interceptors.js:1029:24
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! singles-api@0.0.0 start: `node ./bin/www`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the singles-api@0.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Reproduction steps
grpc installed via npm for use in express app, application works fine in short-timeframe testing but eventually crashes in production tests.
Environment
- Ubuntu 18.04 LTS x64
- Node version 10.15.0 (also 10.13.0)
- grpc@1.17.0 via NPM (also tried --build-from-source)
Additional context
Usage context:
var lnrpcDescriptor = grpc.load("./rpc.proto");
var lnrpc = lnrpcDescriptor.lnrpc;
var lnd = new lnrpc.Lightning(hostport, credentials);
var call = lnd.subscribeInvoices(request)
call.on('data', function(response) {
if (response.settle_index === '0') {
console.log("New Invoice Issued: " + response.payment_request)
}
else {
//iterate through array to find who paid their invoice and update the db
for (var i = 0; i < loadedDB.db.Node.length; i++) {
if (loadedDB.db.Node[i].add_index == response.add_index) {
console.log("Node " + loadedDB.db.Node[i].Id + " has settled their invoice.");
loadedDB.db.Node[i].isSettled = true;
saveDB.writeEntry();
}
}
}
});
call.on('status', function(status) {
console.log(status);
});
call.on('end', function() {
console.log('subscribeInvoices stream ended')
});
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 14
- Comments: 36 (15 by maintainers)
Commits related to this issue
- Repro for https://github.com/grpc/grpc-node/issues/708#issuecomment-532522513 — committed to paambaati/node-liftbridge by paambaati 5 years ago
We have this
Stream Removed
(and anotherTCP Read failed
reported by #692) error frequently in production, but none in test env.Our production is run on k8s cluster, and access grpc service via service name(dns resolver is CoreDNS).
Before we set service host to
core-grpc:50000
, whichcore-grpc
is the domain,50000
is port, and CoreDNS will resolve it to internal ip like:172.21.0.121:50000
, then this error occurred very frequently.After we change service host by ENV variables(inject by k8s) to
${process.env.CORE_GRPC_SERVICE_HOST || 'core-grpc'}:50000
, then still occurred but it’s better than before.Our test env is quite simple, services run in docker container expose by different port. access each other by
localhost:<port>
, and no this kind of error.So is this a network problem?
Same here, only on production env 😦
I guess this problem happen beacause env setting
https_proxy=
Try to remove it env run the commandexport https_proxy=
orunset https_proxy
I experienced the same issue with the EKS cluster using AWS NLB. I tried to play with keepalive timeouts in gRPC options on both sides. I solved the issue by creating a secure gRPC server on the app side.
I’m sorry, but I honestly have no idea what could make errors bypass the error listener like that. And it might not be a grpc-specific problem.
Currently, I am not sure what the root cause of that problem might be. One thing that might help is to try using the
@grpc/proto-loader
and@grpc/grpc-js
libraries in place of thegrpc
library. You will have to change how the.proto
file is loaded but otherwise you can use the same code. I don’t know if that will solve the problem, but it should at least result in a different error that may help with understanding the problem.