nodejs-datastore: runQuery() fails with timeout after some time of inactivity

Environment details

OS: Linux 4.15.0-64-generic #73~16.04.1-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux
Node.js version: v10.16.0
npm version: 6.11.3
@google-cloud/datastore version: 4.3.0

Steps to reproduce

Simple app mixing express and datastore:

“use strict”;

const { Datastore } = require(“@google-cloud/datastore”); const express = require(“express”);

const ds = new Datastore();

const app = express();

app.disable(“etag”); app.set(“trust proxy”, true);

app.get(“/”, (req, res, next) => { const q = ds.createQuery(“Tasks”).order(“taskName”);

return ds.runQuery(q).then((data) => { console.log(“Callback from runQuery()”); let entities = data[0]; return res.json(entities); }).catch((err) => { console.log(“runQuery() failed:”, err.message); return next(err); }); });

// Basic 404 handler app.use((req, res) => { res.status(404).send(“Not Found”); });

// Basic other error handler app.use((err, req, res, next) => { console.log(“500 Error:”, err); res.status(500).send(err.message || “Something is broken!”); });

if (module === require.main) { // Start the server const server = app.listen(“8080”, () => { const port = server.address().port; console.log(App listening on port ${port}); }); }

module.exports = app;

Start it as usual: node app
Open in a browser: http://localhost:8080/

You will get a list of entities.

Wait for more than 5 minutes. Do not stop the webapp and do not use the webapp.
Open in a browser again: http://localhost:8080/
No results and after some time (very long) you will get the following errors:

runQuery() failed: Retry total timeout exceeded before any response was received 500 Error: { Error: Retry total timeout exceeded before any response was received at repeat (./node_modules/google-gax/build/src/normalCalls/retries.js:80:31) at Timeout.setTimeout [as _onTimeout] (./node_modules/google-gax/build/src/normalCalls/retries.js:113:25) at ontimeout (timers.js:436:11) at tryOnTimeout (timers.js:300:5) at listOnTimeout (timers.js:263:5) at Timer.processTimers (timers.js:223:10) code: 4 }

Notes: it’s repeatable in the cloud and running locally (all the environment details for the local run).

Thanks!

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 6
Comments: 24 (4 by maintainers)

Most upvoted comments

Since v0.6.6 is the latest one, just doing rm -rf node_modules package-lock.json && npm install (for those of you who use npm) also brings the latest dependencies to you, including @grpc/grpc-js v0.6.6.

Thank you folks for confirming that it works for you!

alexander-fenster on Oct 8, 2019

Does anybody know the latest version that doesn’t have this issue? We are also using some gcloud libraries, e.g. Datastore, Pub/Sub, and recently tried to upgrade the Datastore package as the previous version was fairly outdated. With version 4.1.4 or the latest one (4.3) we are seeing the same errors after 5 minutes of inactivity.

mrksbnch on Oct 7, 2019

Using

"resolutions": {
    "@grpc/grpc-js": "^0.6.6"
}

as a temporary fix, works for me too.

Tolgor on Oct 8, 2019

@dann95 Thanks, I edited my previous comment: Adding @grpc/grpc-js separately (v0.6.6) seems to work for us thus far.

mrksbnch on Oct 8, 2019

@alexander-fenster I’m happy to test that. However, I’m not totally sure how the @grpc/grpc-js is related to the @google-cloud/datastore package. We currently only have a @google-cloud/datastore entry in the package.json file. Do we need to add @grpc/grpc-js separately?

@dann95 Yes, we updated package.json (without ^), package-lock.json and no cache. However, we were still seeing errors. We can probably give it a try again.

EDIT: So, we’ve added the package @grpc/grpc-js (v0.6.6) as an additional dependency. This version is running for a couple of hours already and no new “timeout” errors popped up thus far (will keep monitoring for errors). Without this change, the errors usually popped up quite soon (depending on the amount of inactivity between requests which can vary in our case).

mrksbnch on Oct 8, 2019

@bcoe Yes, it seems pretty much like that. All the issues that look like “it’s all going well but then after some time everything stops working” point to https://github.com/grpc/grpc-node/issues/1064 at this time.

@rescan-dl @dann95 @mrksbnch Can you folks try it with the latest (released just minutes ago) @grpc/grpc-js v0.6.6 (just rm -rf node_modules package-lock.json && npm install), and let us know if it helps? If it still fails, running your code with the following environment variables set:

export GRPC_TRACE=all
export GRPC_VERBOSITY=DEBUG

and sharing the results with us will be really appreciated.

alexander-fenster on Oct 7, 2019

@alexander-fenster do you think this might be related to some of the gprc-js issues we’ve been seeing recently?

bcoe on Oct 7, 2019

I can observe the same issue in my project. After 4-5 minutes of inactivity, my queries fails on runQuery().

"@google-cloud/datastore": "^4.3.0",
"@google-cloud/error-reporting": "^1.1.2",
"@google-cloud/storage": "3.3.1",

Moving “new Datastore()” to local scope solves the problem but I don’t think that this is clean approach. Or maybe it is?

Thanks!

Kamil7S on Oct 7, 2019