nodejs-datastore: Total timeout of API google.datastore.v1.Datastore exceeded 60000 milliseconds

Hello, we are getting this error from nodejs-datastore-sdk in our clusters in production from time to time:

Error: Total timeout of API google.datastore.v1.Datastore exceeded 60000 milliseconds before any response was received.

We are having this error for more than one year and no one seems to know what’s going on, can you guys at least enlighten us on what could be possibly happening???

  1. Is this a client library issue or a product issue? Yes, it looks like a bug into nodejs-datastore-sdk

  2. Did someone already solve this?

  • Search the issues already opened: not-found
  • Search the issues on our “catch-all” repository: not-found
  • Search or ask on StackOverflow (engineers monitor these tags): not-found

Environment details

  • OS: debian10
  • Node.js version: 14.21.1
  • npm version: 9.5.0
  • @google-cloud/datastore version: 7.0.0

Steps to reproduce

  1. Unfortunately not reproduceable, it happens completely random times in our VM instances at google-cloud-platform

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 9
  • Comments: 20 (5 by maintainers)

Commits related to this issue

Most upvoted comments

Lowering priority to P3 since the issue is now just limited to REST fallback.

I’d like to remind senior-developers of this lib and also program-managers that THIS IS NOT A P3 PROBLEM

REST fallback is SLOW and we are moving away from Datastore and going to ScyllaDB because of timeout/reset problems

We are also thinking about leaving the entire google-cloud-platform, so I hope you guys realize how bad this problem is for your customers

I opened a support ticket and was told that, while the issue isn’t resolved, there is a workaround. It was confirmed to be an issue with the Datastore backend and not an issue with this library, but it looks like you can now add a fallback to the options object when creating a new Datastore instance.

const ds = = new Datastore({
  fallback: 'rest',
});

Reference PR

Unfortunately, I also need to use the @google-cloud/connect-datastore library and that is currently locked to @google-cloud/datastore v7 so I’m unable to upgrade at this time. I’m currently using our Cloud SQL database as a fallback workaround.

Hi guys, the same thing is happening to me, it’s very random. It started happening today and we haven’t made any changes.

  Error: Total timeout of API google.datastore.v1.Datastore exceeded 60000 milliseconds before any response was received.
  
  at .repeat ( /node_modules/google-gax/build/src/normalCalls/retries.js:66 )
  at .Timeout._onTimeout ( /node_modules/google-gax/build/src/normalCalls/retries.js:101 )
  at .listOnTimeout ( node:internal/timers:569 )
  at process.processTimers ( node:internal/timers:512 )

I have updated my version to see if it solved the problem but it keeps happening.

@google-cloud/datastore 8.2.1 to 8.2.2

Node version: v18.17.0 npm version: 9.6.7

Lowering priority to P3 since the issue is now just limited to REST fallback.

@danieljbruce awesome, thanks

We migrated 50% of our data to scylladb and the timeouts dissapeared, you guys should make more explicit in the docs what are the real limits of datastore

We made a change recently to expose the original error that occurs with https://github.com/googleapis/gax-nodejs/pull/1740/files instead of just reporting the [Total timeout of API google.datastore.v1.Datastore exceeded 60000 milliseconds] error. If you upgrade nodejs-datastore to the latest version then you should see Previous Errors in the error message. Please paste the error message you see here.

Also ran into this today internally on one instance, the rest of them are fine. Feels like some form of GRPC handling bug where it loses connection and unable to reconnect and then the queue of things to process just get stuck.

Wish we could reproduce this easier to contribute some form of fix for this, as P3 seems pretty low for something which is pretty severely broken? REST is somewhat OK for our usecase, 15-20 RPS, but seeing 500ms latency now instead of <150ms.

Also the terminology fallback for configuring this seems wrong? It’s not a fallback if it uses REST all the time, I’d expect it to prefer GRPC but once it fails, then try REST?

thanks for your help @rossjs we are gonna try this fallback parameter