nodejs-datastore: Error: 13 INTERNAL: Received RST_STREAM with code 2

Hi!

Getting the following error periodically (1-2 times a day)

A 2020-06-10T18:35:53.594123Z (node:14) UnhandledPromiseRejectionWarning: Error: 13 INTERNAL: Received RST_STREAM with code 2 
A 2020-06-10T18:35:53.594173Z     at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:30:26) 
A 2020-06-10T18:35:53.594187Z     at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:175:52) 
A 2020-06-10T18:35:53.594267Z     at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141) 
A 2020-06-10T18:35:53.594293Z     at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181) 
A 2020-06-10T18:35:53.594306Z     at Http2CallStream.outputStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74) 
A 2020-06-10T18:35:53.594318Z     at Http2CallStream.maybeOutputStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22) 
A 2020-06-10T18:35:53.594330Z     at Http2CallStream.endCall (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18) 
A 2020-06-10T18:35:53.594342Z     at ClientHttp2Stream.<anonymous> (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22) 
A 2020-06-10T18:35:53.594356Z     at ClientHttp2Stream.emit (events.js:315:20) 
A 2020-06-10T18:35:53.594367Z     at emitCloseNT (internal/streams/destroy.js:69:8) 
A 2020-06-10T18:35:53.594394Z (node:14) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1) 
A 2020-06-10T18:35:53.594405Z (node:14) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code. 
A 2020-06-10T18:35:54.059739Z Error: 13 INTERNAL: Received RST_STREAM with code 2
    at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at ClientHttp2Stream.<anonymous> (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22)
    at ClientHttp2Stream.emit (events.js:315:20)
    at emitCloseNT (internal/streams/destroy.js:69:8) { 
A 2020-06-10T18:35:54.059769Z   code: 13, 
A 2020-06-10T18:35:54.059794Z   details: 'Received RST_STREAM with code 2', 
A 2020-06-10T18:35:54.059803Z   metadata: Metadata { internalRepr: Map {}, options: {} }, 
A 2020-06-10T18:35:54.059813Z   note: 'Exception occurred in retry method that was not classified as transient' 
A 2020-06-10T18:35:54.059821Z }

Environment details

OS: Cloud Run
Node.js version: node:lts-slim
npm version: 6.14.5
@google-cloud/datastore version: 6.0.0

Thanks!

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 19
Comments: 29 (5 by maintainers)

Most upvoted comments

We are working on this with multiple engineers and trying to come with mitigation strategies. Unfortunately, this is more difficult than anticipated.

schmidt-sebastian on Feb 4, 2021

Saw one of these in a firebase async onRequest call (Node 10 runtime) on 02/15/2021

Here’s the stack …

Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)
    at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
    at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:342:141)
    at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:305:181)
    at process.nextTick (/workspace/node_modules/@grpc/grpc-js/build/src/call-stream.js:124:78)
    at process._tickCallback (internal/process/next_tick.js:61:11)

tohagan on Mar 9, 2021

@schmidt-sebastian Does this look like the same error we’re seeing in Firestore?

BenWhitehead on Jun 15, 2020

Had this error happen after a couple of deletes on datastore, resolved it by ensuring that the process exited properly with process.exit(0). hope this helps someone, cheers 😃

odira1 on Jun 20, 2021

Same issue here, when try to migrate db from realtimedatabe to firestore, on localhost with small sample of db the migration function works very well, but when i use entire legacy db i got this errors and emulators got shutdown:

Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error) at Object.callErrorFromStatus (workspace\functions\node_modules\@grpc\grpc-js\build\src\call.js:31:26) at Object.onReceiveStatus (workspace\functions\node_modules\@grpc\grpc-js\build\src\client.js:176:52) at Object.onReceiveStatus (workspace\functions\node_modules\@grpc\grpc-js\build\src\client-interceptors.js:336:141) at Object.onReceiveStatus (workspace\functions\node_modules\@grpc\grpc-js\build\src\client-interceptors.js:299:181) at workspace\functions\node_modules\@grpc\grpc-js\build\src\call-stream.js:145:78 at processTicksAndRejections (internal/process/task_queues.js:75:11) Caused by: Error at WriteBatch.commit (workspace\functions\node_modules\@google-cloud\firestore\build\src\write-batch.js:413:23) at DocumentReference.set (workspace\functions\node_modules\@google-cloud\firestore\build\src\reference.js:350:14) at userIterator (workspace\functions\dbMigrator.js:270:28) at workspace\functions\dbMigrator.js:276:5 at local\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:604:16 at runFunction (local\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:583:15) at runHTTPS (local\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:603:11) at handler (local\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:532:23) at Layer.handle [as handle_request] (local\AppData\Roaming\npm\node_modules\firebase-tools\node_modules\express\lib\router\layer.js:95:5) at next (local\AppData\Roaming\npm\node_modules\firebase-tools\node_modules\express\lib\router\route.js:137:13)

Chm-vinicius on Jun 9, 2021

Facing exactly the same issue with Cloud Tasks:

Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: read ECONNRESET
    at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
    at /usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
    at processTicksAndRejections (internal/process/task_queues.js:79:11)

This happens quite rarely, but when happening, it’s not in a context of massive task creation, just a single task created :

await this.client.createTask({
      parent: queueName,
      task: {...}
});

tfalvo on May 17, 2021

@schmidt-sebastian ok thanks for the info. Good luck with that 😃 Was unsure as I see Open bug with no assignee.

zdenda-online on Feb 4, 2021

Any update?

By the way, I don’t agree this should have priority p2 = moderate importance. This breaks the base use of the service.

zdenda-online on Feb 4, 2021

I am seeing this as well. I am running a Typescript React App with a node.js server. The server is throwing this error fairly regularly. Something like once an hour. I also get this similar error: @firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Stream refused by server but about 20% as frequently.

I had a crash on my site on Monday where I was not able to read or write the database and I am trying to figure out if its related.

These are the 3 types of errors I am receiving of this type: @firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 2

@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Stream refused by server

@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 0

eli-stewart on Dec 10, 2020