valkey-glide: Nodejs StartSocketConnection is not a function
Inquiry
We’re using esbuild to bundle and minify handler code for AWS Lambdas producing this kind of output.
-rw-r--r--@ 1 paulhawxby staff 1852129 Jan 1 1980 eventData.js
-rw-r--r--@ 1 paulhawxby staff 4081730 Feb 17 13:24 eventData.js.map
-rwxr-xr-x@ 1 paulhawxby staff 4800864 Feb 17 13:24 glide-rs.darwin-arm64.node
-rwxr-xr-x@ 1 paulhawxby staff 4949752 Feb 17 13:24 glide-rs.linux-arm64-gnu.node
-rwxr-xr-x@ 1 paulhawxby staff 4986584 Feb 17 13:24 glide-rs.linux-arm64-musl.node
-rwxr-xr-x@ 1 paulhawxby staff 5756848 Feb 17 13:24 glide-rs.linux-x64-gnu.node
-rwxr-xr-x@ 1 paulhawxby staff 5752728 Feb 17 13:24 glide-rs.linux-x64-musl.node
When executed on AWS we get the following error.
{
"name": "TypeError",
"location": "/var/task/eventData.js:13",
"message": "(0 , nT.StartSocketConnection) is not a function",
"stack": "TypeError: (0 , nT.StartSocketConnection) is not a function\n at GlideClient.<anonymous> (/var/task/eventData.js:13:66296)\n at Generator.next (<anonymous>)\n at /var/task/eventData.js:13:29846\n at new Promise (<anonymous>)\n at F (/var/task/eventData.js:13:29598)\n at GlideClient.createClientInternal (/var/task/eventData.js:13:66224)\n at GlideClient.<anonymous> (/var/task/eventData.js:13:68101)\n at Generator.next (<anonymous>)\n at /var/task/eventData.js:13:67321\n at new Promise (<anonymous>)"
}
I absolutely cannot figure out where this error is coming from.
"@valkey/valkey-glide": "^1.2.1",
"@valkey/valkey-glide-darwin-arm64": "^1.2.1",
"@valkey/valkey-glide-linux-musl-arm64": "^1.2.1",
"@valkey/valkey-glide-linux-arm64": "^1.2.1",
"@valkey/valkey-glide-linux-x64": "^1.2.1",
"@valkey/valkey-glide-linux-musl-x64": "^1.2.1",
Language
Typescript
Language Version
5.1.6
Engine Version
Node 22
Operating System
ARM & x86
Additional Technical Information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 79 (41 by maintainers)
@hassanbsalimi @dcaruso-pathpoint @LinusU @galaxyfeeder 1.3.5-rc9 is the new working structure and should solve the issues mentioned, including yarn and TS issues, with cleaner package, structure, better size and better performance. If you chose to use the RC please hard-code it, since I’m going to keep testing the new CD and more RC will go out, the mentioned passed the whole testing matrix, for the other to come there is no guarantee.
Note that this is an RC, and altho it’s stable and passed tests, the official release not yet happened and processes we take before releases didn’t go on this one.
@galaxyfeeder i apologize for the inconvenience. The issue is not been fixed yet. But it is my main focus now, including minimizing further the package size along with a little more performance enhancements.
I believe that in the next few days I’ll be able to release an RC (tested one, but not the whole content of the next version), would it work for you, or solution is urgently needed?
Fix was successfully merged. 1.3.5-rc13 is the last test performed successfully. It will be released as stable in v2.
@LinusU @hassanbsalimi v1.3.5-rc11 Please try it out and let me know.
For public awareness of what this specific issue appears to come down to.
Valkey glide is structured in such a way that there is an entry package which based on the architecture loads specific sub dependency packages, for example @valkey/valkey-glide-linux-arm64. Unfortunately those sub dependency packages also include the full JS wrapper around the rust native binary and not just the binary. This means that if you’re trying to bundle the js with esbuild (and especially in our case where we want to bundle for multiple architectures at the same time) you end up with the same wrapper code bundled in multiple times which inflates the output bundle size and also appears to break things.
There are possible workarounds with lambda layering, manual esbuild steps to copy files around, but that would require a lot of rework of our build pipeline. Also @avifenesh has had success using serverless-esbuild.
It’s worth noting most other libraries where native binaries are required are not structured this way, the arch specific binary is in a package on its own and the wrapper code it self acts as the parent entry package.
Hi @avifenesh , sure I’ll join the Valkey Slack and send you a message there.
@BhupinderSinghYouScience Can you please join Valkey Slack and message me so we can chat directly?
@BhupinderSinghYouScience First, I’m happy that the first issue is solved. Second, don’t be sorry, doing it with pleasure. When you run it, do you able to reproduce it locally, or you need to deploy to an actual lambda? I want to know whether I need to set up a real env with lambda and everything. If I have to run to create the lambda, I would appreciate it if you have a CFN ready so I can create the whole system needed the same.
Hi @avifenesh, Thanks for the fix — that error is now resolved. The issue was with my package.json (it worked for me after using
"@valkey/valkey-glide": "1.3.5-rc13"). I really appreciate your help, and sorry for the delay in getting back to you. I was trying to debug it on my end because I kept running into other runtime errors.Sorry to bother you again, but if you get a chance, could you please run
npm run deploy(build + deploy) and if the/cacheendpoint works for you without any errors. The repo now uses CodePipeline to build on an x86 architecture, and it’s deployed to a Lambda function running on x86_64. I’m still hitting the error.Hi @avifenesh, Thank you so much for taking the time to help. I really appreciate it. I’ll test the change shortly and let you know if it resolves the issue.
@BhupinderSinghYouScience Hi 😃 thanks for the repo! Pushed a small change as a PR, please let me know if it solves your issue. If it does, please consider posting here the solution.
@avifenesh , I’ve set up a minimal private repo that reproduces the issue and sent you an invite. Let me know if you have any trouble accessing it. Looking forward to your thoughts when you get a chance.
Thanks, I’ll try to reproduce when I’m back.
Ok great, will work on it ASAP. Can you please share the architecture of the lambda? And whether you remove the musl binary?
@avifenesh, When I run it locally, everything works fine.
Architecture: x86_64 Engine Version: Node.js 18.x
My package.json does not explicitly mention the musl binary. It currently includes:
Yes, we can wait a day no problem at all. I really appreciate your help and look forward to your suggestions when you’re available. Thanks so much for taking the time to look into this
@BhupinderSinghYouScience Yes, it’s probably what caused the last mentioned error. The valkey-glide-{target} is explicitly direct to the .node file, same as it would be setting the index.js of a generic nodejs package. If a package points to index.js but the index now is index-hash.js you would have a crash. Can you explain the hash behavior?
I’m sorry that I can’t do some research myself, my laptop is not in reach today. Is there an option to set to a hard coded name, or to a bin instead of file?
Have a meeting about it soon, will come back with better answers. I don’t believe we will get under 1 MB for the zipped native, but we are looking for improvements with the size.
All working fine for me. Thanks for the quick resolution. Much appreciated.
Added also tests to the CI specifically mimicking those cases, both yarn usage, and TS usage that check dependencies types. I generally, I’m my project, don’t perform type checking in transpile time to node modules, probably why i didnt face the error, so added it so from now no type issues snick in.
Thank you. Please let us know when the RC is available. I am blocked heavily by it. Eager to block myself with RC and switch to full release whenever its available.
Hi @hassanbsalimi Yes it is resolved, tested it locally, what is left is to finish fighting with the CD, and I’ll release an RC to unblock everyone. It will later be an official release, after the merge completes, but the code is passing our heavy testing matrix (16-23 node versions X 6.2-8.1 Valkey/Redis versions X 6 platforms), and producing better performance and smaller code overall, with clean separation between the TS package and the native package.
quite bust today - i’ll send over a minimal repo when I can
No problem! Works for me! Just wanted to understand better the previous conversation with Linus and what to expect from the last release!
Thanks for your help! I need to run from the office now, but will still try to keep an eye on this issue!
Thanks for all the time you put in providing information.
@LinusU The size of glide will be cut by half with the patch from around 10 to 5-6, depending on host target, and hopefully by more with the new versions. The double package is because NPM respects LIBC type from v11 only, in compare to yarn that does that for a long time ago. ATM excludes the wrong target, in a month upgrade NPM. And in a month you’ll be able to esbuild all but the binary more easily, in compare to now having one package with both. So it will shrink the size even more. But note, the client, in best practice, shouldn’t be fire up on request, but be global and alive, while the functions are fired up, so following that, the size is not a big deal. If you’d like, ill hand to you the first RC before the release itself, in two weeks maybe, no promises, so you can see how is it working for you, and ill be able to know that it is actually working.
Want to jump to valkey slack to have faster debugging?