nodejs-spanner: Error 14 UNAVILABLE: EOF
Environment details
- OS: CoreOS
- Node.js version: v8.9.6
- npm version: v5.6.0
@google-cloud/spannerversion: v1.4.1
Steps to reproduce
Will attempt to create some if needed.
Hello 👋
I have a set of APIs using this spanner library and as a transfer over a substantial amount of load (a few thousand requests per second) to these APIs I start to get errors from spanner with the following message: 14 UNAVAILABLE: EOF.
Looking around it seems I should simply have to retry this request to have a good chance at succeeding. Curious why this is not a case the library handles for me (I believe I saw some mentions of auto-reties for other transient errors somewhere in the docs). Additionally I would like to better understand what is going on here so I can solve the real underlying issue.
Is this spanner trying to tell me that I need to scale up my cluster in order to provide more reliable throughput? Or is this simply something I should expect to run into from time to time?
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 6
- Comments: 52 (32 by maintainers)
We have 2 projects using
nodejs-spannerin production getting exactly the same errors nightly.Since we haven’t heard much in the way of @stephenplusplus fix we are going to proceed to PR and merge it in. Thanks!
@walshie4 @Duncan00 would you please give my branch a test:
This should retry on these types of errors.
I’m currently running with 10 Spanner nodes. Haven’t dug into which query is causing this issue (because the volume and impact of this bug is lesser than the others).
The data in this database is only about ~250mb of data currently, a mix of String, Int64, and ARRAY<String> values. There is a main table with ~40k rows, with ~8 interleaved tables each containing at most a couple thousand rows in those interleaved tables (most are in the ballpark of 0-20 rows per interleaved table). One of the interleaved tables has another table interleaved onto those rows, again small datasets, only 10 rows in that table per parent at most (most are 1-4).
This was just off-hand as I’m currently in the middle of something so if there are specific things I didn’t cover which could be valuable let me know and I’ll get back to y’all shortly when I have a bit more time.
@callmehiphop can you please investigate based on @vkedia’s assertion in the comment above? Thanks!
@stephenplusplus has merged his fix for gax-nodejs
I think it does, but @stephenplusplus should be able to give us a definitive answer.