nodejs-spanner: spanner: Root cause of "Error: Unable to release unknown resource"

Hi @olavloite @hengfengli

Thanks for the suggestion. I will give this a try and test the same in the staging environment. Will keep you guys updated about the same

Thanks

Hi @codemart786

If I understand you correctly, you are (sort of) able to reproduce this problem on your staging environment, right? If so, would it be possible for you to try the following:

1. Make a build of your application that uses the current master branch of this client library. master now contains https://github.com/googleapis/nodejs-spanner/pull/759 which will include the stacktraces of potentially leaked sessions in the error that is thrown when the session pool is exhausted.
2. Deploy this build on your test/staging environment with the following session pool settings:
   1. max=100
   2. fail=true (This setting will cause the session pool to throw an error instead of waiting indefinitely when the maximum number of sessions in the pool has been reached. You should therefore be careful with enabling this setting on your production environment, as these errors could bubble up to the users.)
3. This will hopefully still trigger the problem that you are running into, but instead of the application just hanging, it will start throwing errors that include valuable debugging information.

If you could then inspect the stacktraces (messages) that are included in the SessionPoolExhaustedError, it should be possible to determine whether the problem is caused by a session leak or not, and when it is, what part of the code is causing it.

@AVaksman @skuruppu @callmehiphop @mf2199

Hey guys, few things I want to ask here, related to this issue .

We did load testing in our staging environment, and have seen session count increasing from 3k to 100k within 30 mins. We have an api which uses spanner locking read-write transaction. We are calling transaction.end() function also. I believe this is a session-leak problem, but could’t understand why. We use rps of 4000 for 30 minutes on one of service apis.

Having said that, here are my some of doubts which I would like to clarify:

1. Do I need to call transaction.end() even after successful commit of transaction? Right now I am not doing that . I am calling transaction.end() only if commit operation throws an error or if we decided to forego commit due to some reason . I believe on successful commit, transaction.end() is called to release the session back to the pool, by the spanner module itself. (Please correct me if I am wrong here)

2. We added the code snippet as suggested by @callmehiphop on database.close(), before pod shuts-down . The error which we see in the logs is given below:

jsonPayload: {
  message: " got error message trace"   
  metadata: {
   err: "Session leak detected!
    at _acquire (/testProject/node_modules/@google-cloud/spanner/build/src/session-pool.js:278:44)
    at null (null:null:null)"    
   service: "testProject-abc"
  }
 }

Please help me know when the @google-cloud/spanner node module throws this stack trace? If I know in which direction to proceed further, it would be very helpful.

PS: Our application has suffered several production outages, which I believe is mostly due to this issue. Please help me in solving this issue. Any help is deeply appreciated 😃

Thanks