google-cloud-node: Spanner - noStatus - TypeError: requestStream.destroy is not a function

OS: Mac OSX 10.12.4 Node.js version: 6.9.3 npm version: 5.0.0 version: 0.4.4

After closing #2176 i run my data export in spanner. When a number of requests were in parallel the following error occurred several times.

noStatus - TypeError: requestStream.destroy is not a function
    at StreamProxy.<anonymous> (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/@google-cloud/spanner/src/transaction.js:375:19)
    at emitOne (events.js:96:13)
    at StreamProxy.emit (events.js:188:7)
    at ClientReadableStream.<anonymous> (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/google-gax/lib/streaming.js:130:17)
    at emitOne (events.js:96:13)
    at ClientReadableStream.emit (events.js:188:7)
    at ClientReadableStream._emitStatusIfDone (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/grpc/src/node/src/client.js:204:12)
    at ClientReadableStream._receiveStatus (/Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/grpc/src/node/src/client.js:180:8)
    at /Users/Chipintoza/GSS Projects/accounts.gss.ge/node_modules/grpc/src/node/src/client.js:592:14

After certain quantity of data export, the following error kept occurring constantly:

req.url - http://accounts.gss.local:9001/gss-api/environments/upsert InternalServerError: err.code: 8
err.message: Too many active sessions in database projects/spanner-gss-ge/instances/business-data/databases/system-development, limit is 30000. Increase the node count to allow more sessions.
err.status: undefined
err location: Environment.upsert\database.runTransaction((err, tx) => {…
data:

{
  "environmentId": "5e9d511f-e300-49d8-80ba-21c421f98b03",
  "isActive": true,
  "created": 1325376000000,
  "timestamp": 1496655116694,
  "modifierId": "system-id",
  "modifierName": "system",
  "name": "ka:ჩემი 'GSS გარემო' - paab@mail.ru|en:My 'GSS Environment' - paab@mail.ru|ru:Моя 'GSS Среда' - paab@mail.ru",
  "databaseID": "39aa56bc-f926-4df2-afe9-619c1e527afe",
  "isUnderControl": false,
  "environmentSyncURL": "",
  "linkedGSSContactID": null,
  "linkedContactID": null,
  "inputLanguages": "en ka ru",
  "usersInputLanguages": "",
  "registrationCountry": "",
  "rsgeServiceUser": "",
  "rsgeServiceUserPassword": "",
  "mainCurrencySignification": null,
  "accountingCurrencySignifications": "",
  "domain": null,
  "gssEducationIsActive": false
}

apiResponse:

I stop my process and during 20 minute from console i can not run any queries… there was same error:

Too many active sessions in database projects/spanner-gss-ge/instances/business-data/databases/system-development, limit is 30000. Increase the node count to allow more sessions.

When will close sessions? look at pictures

I was wait around 40 minute and then delete database. https://cloud.google.com/spanner/docs/limits Sessions per database per node 10,000 What does it mean? how longer do i need to wait session closing?

Thank you

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 21 (11 by maintainers)

Most upvoted comments

@danoscarmike @lukesneeringer @callmehiphop can we please get this resolved asap. This is causing customers to get locked out of spanner for upto an hour.

vkedia on Jul 10, 2017

@stephenplusplus I am also seeing this error in our production environment. let me know if there’s any more info you need that would be helpful, or if there’s a bad practice on the application that leads to this error, that would be helpful for me to know.

JoshFerge on Jun 9, 2017

@Chipintoza If there are other problems in https://github.com/GoogleCloudPlatform/google-cloud-node/issues/2356#issuecomment-314768751 that are still present in the 0.6.0 release, can you create separate issue(s) for them, and tag me and @lukesneeringer. Thanks!

bjwatson on Jul 18, 2017

The final fix (we hope!) is out under @google-cloud/spanner@0.6.0 – please let us know if there are any issues.

stephenplusplus on Jul 17, 2017

@bjwatson working on this right now.

For anyone experiencing a similar issue, you’ll want to check that you’re not creating multiple Database instances for the same Database. Everytime a Database instance is created it also creates a new session pool (hence why we see the number of sessions get really high).

The PR I’m working on is going to cache the pools, that way users don’t have to be bothered with caching database instances and we won’t spin up sessions unnecessarily.

callmehiphop on Jul 12, 2017

@callmehiphop @lukesneeringer What’s the ETA for fixing this issue?

bjwatson on Jul 11, 2017