nodejs-datastore: Error: 13 INTERNAL: Failed to start HTTP/2 stream

Hi,

This error seems to happen intermittently.

Error: 13 INTERNAL: Failed to start HTTP/2 stream
    at Object.callErrorFromStatus (/src/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/src/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/src/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/src/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/src/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/src/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/src/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at Http2CallStream.cancelWithStatus (/src/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:457:14)
    at /src/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/channel.js:209:48
    at runMicrotasks (<anonymous>)

Environment details

OS: Cloud Run
Node.js version: v14.8.0
npm version: 6.14.7
@google-cloud/datastore version: 4.5.2

Thanks!

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 6
Comments: 33 (8 by maintainers)

Most upvoted comments

@Joshandrews43 nodejs-datastore/firebase-admin or an upstream Google dependency is pretty clearly broken. There is no fix other than restarting it. Google Support says this has been a known issue for a long time. I have no idea how to get it taken care of.

Looking through Google’s internal bug tracking system, development group email lists and different GitHub repository issues, this issue just bounces around back and forth between teams. I tried contacting Google’s premium cloud support about this issue and they report that this is a known issue with numerous customers reporting it, but provide no solution and no ETA.

stx on Mar 5, 2021

@stx The issue with this error causing subsequent requests to fail looks like a bug that was fixed in @grpc/grpc-js version 1.1.2.

murgatroid99 on Mar 11, 2021

@crwilcox Retries absolutely do not fix it (and really, have nothing to do with it.) The requests perpetually timeout forever until instances are restarted. I’m sure there are multiple causes of this, but the issue @Joshandrews43 and I are describing isn’t a random irritating message, it’s an outage. So we might not be talking about the same issue here.

Here’s a complete reproduction:

const express = require('express');
const admin = require('firebase-admin');
var app = module.exports = express();

app.get('/test', async (_, res) => {
  await admin.firestore().collection('users').where('thing', '==', 'true').get();
  res.send("Success");
});

app.get('/works', async (_, res) => {
  res.send("Success");
});

/test will randomly begin to fail 100% of the time, forever, with the message outlined in this issue. /works will work forever.

Replace Firestore with Datastore (and firestore-admin with nodejs-datastore) and the problem remains.

stx on Mar 11, 2021

No, haven’t tried.

stx on Feb 24, 2021

@stephenplusplus

This also happens on Node 12 using the node:12-slim image and connecting to Firestore. Isn’t incurred by load, inactivity or volume. This is an extraordinarily severe issue to be happening randomly on stable production environments.

I guarantee there are a number of customers experiencing this because we’re not doing anything remotely special and 100% of requests fail until the instances are replaced. Our code is literally just an Admin SDK call through an Express route on a vanilla Docker image. Routes without Admin SDK calls perform correctly.

stx on Dec 10, 2020