nodejs-storage: getSignedUrl is blocking the event loop
We have an Express.js API which is using signed URLs for managing media files (images, video and files). Some endpoints can return from 0 to many signed URLs and we are having huge problems with the performance of storage.bucket(...).file(...).getSignedUrl(...) because it is delaying the Node.js event loop (or at least this is what I think right now). The more signed URLs, the worst performance. With a very low usage, the app performance is degrading so much that it is unusable.
Using clinic.js for profiling, we get the following event loop delay times:
and this just commenting getSignedUrl line:
The tool is making:
- 50 req/s calling to an endpoint
- the endpoint is calling in parallel (
await Promise.all([ ... ])) 10 times togetSignedUrlmethod
Environment details
- OS: Linux
- Node.js version: 16.6.2
- npm version: 7.20.3
@google-cloud/storageversion: 5.16.0
Any clue is welcome! Thanks!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 5
- Comments: 18 (10 by maintainers)
Currently prefixes are not supported by the storage library. The documentation you had linked above is actually for Cloud CDN.
Hi!
Yes, to clarify, this was with usage of getSignedUrl.
For us, it was the number of documents we were signing in “parallel” using Promise.all
For something like 3 or 4, all the promises resolved in a total of less than a second
For something like 600 calls to getSignedUrl, the promise.all was finishing in 13+ seconds.
Changing to manually setting the private key resulted in the promises to finish in ~3 to 4 seconds, which is acceptable for our use-case.
Thank you for the details @oleurud. I will try and reproduce this using the steps/tools you outlined. I will let you know what I find.