aws-sdk-js-v3: Assertion parser->current_buffer_.IsEmpty() failed in Node when using Upload in @aws-sdk/lib-storage in cloud environment
Confirm by changing [ ] to [x] below to ensure that it’s a bug:
- I’ve gone through Developer Guide and API reference
- I’ve checked AWS Forums and StackOverflow for answers
- I’ve searched for previous similar issues and didn’t find any solution
Describe the bug This is actually an issue in https://github.com/nodejs/node/issues/39671
When using @aws-sdk/lib-storage
to perform multipart upload to S3. NodeJs throw the following exception:
node[1]: ../src/node_http_parser.cc:510:static void node::{anonymous}::Parser::Execute(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `parser->current_buffer_.IsEmpty()' failed.
This is from node/http
module. Interestingly, this only happen on cloud env but not local dev env work fine. (Further investigation is needed to verify). It run fine almost every time in my local machine (Macbook Pro 16" Intel running node v14.17.5) with only one time I have seen the exception. But in the production env running EKS over EC2, it fail consistancly at the same point no matter the data size.
This MAY NOT be an issue of @aws-sdk/lib-storage
but it seems to be only happen when using aws sdk together in cloud env.
I have tried changing my application logic to only perform 1 upload at a time and it did not crash even in cloud env (AWS EKS running on EC2). So I suspect it is a concurrent issue of some sort in either aws-sdk or nodejs level. It would be great if we (user, aws-sdk, nodejs) can work together and find a re-producer first.
Is the issue in the browser/Node.js? Node.js and possible only happens on cloud (AWS?) ENV only
If on Node.js, are you running this on AWS Lambda? No
Details of the browser/Node.js version
- machine type: m5.xlarge
- container base image tag: node:14.17.5-alpine3.14 and 16.9.1-alpine3.14 have been tried (same node version as my local but use alpine instead. I am not sure does it matter)
- resources on k8s pod
resources:
limits:
cpu: 1000m
memory: 1024Mi
requests:
cpu: 1000m
memory: 1024Mi
SDK version number
@aws-sdk/lib-storage@3.33.0
for me and I am not sure others
To Reproduce (observed behavior)
No reproducer at this moment. Looking for someone to reproduce it.
Expected behavior
No exception thrown
Screenshots
No
Additional context
Code snippet for reference
import { Upload } from '@aws-sdk/lib-storage'
import { PassThrough } from 'stream'
const multipartUpload = new Upload({
client: s3Client,
params: {
Bucket: 'some-s3-bucket',
Key: 'path/to/some.json',
Body: inputStream.pipe(new PassThrough()),
ContentType: 'application/octet-stream',
},
})
await multipartUpload.done()
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 11
- Comments: 15
Commits related to this issue
- revert to using SBT to upload to riffraff A bug in Node 16 means that `@guardian/actions-riff-raff` can fail to upload assets to S3. See https://github.com/aws/aws-sdk-js-v3/issues/2843 for more info... — committed to guardian/frontend by georgeblahblah a year ago
- revert to using SBT to upload to riffraff A bug in Node 16 means that `@guardian/actions-riff-raff` can fail to upload assets to S3. See https://github.com/aws/aws-sdk-js-v3/issues/2843 for more info... — committed to guardian/frontend by georgeblahblah a year ago
NodeJS v18.6.0 has a fix for nodejs/node#39671 (nodejs/node@491c7619c4)
Yes, we use
@aws-sdk/lib-storage
to do multipart uploads to S3 on Lambda.queueSize
is a parameter that controls concurrency in@aws-sdk/lib-storage
uploads; i.e.We first ran into this issue when we switched from v2 to the v3 SDK. We couldn’t reproduce it when we set
queueSize
to 1, so this is how we “fixed” it, since this was sufficient for our needs. But we recently encountered the issue in our production environment even withqueueSize
set to 1, so it seems like it can still happen (albeit much less frequently) without concurrent uploads.