aws-sdk-go: [S3] RequestError with UploadPart call
Please fill out the sections below to help us address your issue.
Version of AWS SDK for Go?
v1.18.2
Version of Go (go version)?
go version go1.10.4
What issue did you see?
We are using MultipartUpload for uploading some mp3 files to S3. At times, the S3.UploadPart function throws the following error:
RequestError: send request failed
caused by: Put https://<bucket_name>.s3.ap-southeast-1.amazonaws.com/filename.mp3?partNumber=1&uploadId=FQEnFylcgwfiSyqAFJSAaBPujuG_ooLOCrPnHv5vO_Un0W5_Ml8DYeqB4xx7US_IFbcdkjrWZqizTemAKyx3MNYwku6BPRqvLz3eGxAqndFUUw--: http: Request.ContentLength=4674720 with nil Body
Here is the code the code that handles the uploading part:
func uploadPart(svc *s3.S3, resp *s3.CreateMultipartUploadOutput, fileBytes []byte, partNumber int) (*s3.CompletedPart, error) {
tryNum := 1
body := bytes.NewReader(fileBytes)
partInput := &s3.UploadPartInput{
Body: body,
Bucket: resp.Bucket,
Key: resp.Key,
PartNumber: aws.Int64(int64(partNumber)),
UploadId: resp.UploadId,
ContentLength: aws.Int64(int64(len(fileBytes))),
}
for tryNum <= maxRetries {
uploadResult, err := svc.UploadPart(partInput)
if err != nil {
if tryNum == maxRetries {
if aerr, ok := err.(awserr.Error); ok {
return nil, aerr
}
return nil, err
}
log.Printf("Retrying to upload part #%v\n", partNumber)
tryNum++
} else {
return &s3.CompletedPart{
ETag: uploadResult.ETag,
PartNumber: aws.Int64(int64(partNumber)),
}, nil
}
}
return nil, nil
}
Steps to reproduce
Not reproducable. Happens only some times. The body is not actually nil. Made sure of it by checking it in the logs.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 55 (23 by maintainers)
Commits related to this issue
- service/s3/s3manager: Add unit test for S3 Upload manager part retries Adds a new unit test to S3 Upload Manager to verify the SDK will correctly retry concurrent multipart uploads. Related to #2525 — committed to jasdel/aws-sdk-go by jasdel 5 years ago
- service/s3/s3manager: Add unit test for S3 Upload manager part retries (#2636) Adds a new unit test to S3 Upload Manager to verify the SDK will correctly retry concurrent multipart uploads. Relat... — committed to aws/aws-sdk-go by jasdel 5 years ago
- aws/request: Fix SDK error checking when seeking readers Fixes the SDK handling of seeking a reader to ensure errors are not lost, and are bubbled up. In several places the SDK ignored Seek errors w... — committed to jasdel/aws-sdk-go by jasdel 5 years ago
- aws/request: Fix SDK error checking when seeking readers Fixes the SDK handling of seeking a reader to ensure errors are not lost, and are bubbled up. In several places the SDK ignored Seek errors w... — committed to jasdel/aws-sdk-go by jasdel 5 years ago
- aws/request: Fix SDK error checking when seeking readers Fixes the SDK handling of seeking a reader to ensure errors are not lost, and are bubbled up. In several places the SDK ignored Seek errors w... — committed to jasdel/aws-sdk-go by jasdel 5 years ago
- aws/request: Fix SDK error checking when seeking readers (#2696) Fixes the SDK handling of seeking a reader to ensure errors are not lost, and are bubbled up. In several places the SDK ignored S... — committed to aws/aws-sdk-go by jasdel 5 years ago
thanks for the update @knrt10 Let us know if you run into the issue in the future.
@arjunmahishi I created PR #2696 that fixes the SDK’s request handling to always check for error when seeking a request body. there were several cases where an error that occurred would be eaten leading to unexpected behavior. In addition the Pr includes an Example for wrapping your file with a logger that will log stack traces when read/seek errors occur.
Thanks for the the additional information @knrt10 . I’m working on a utility to help debug this issue by logging the SDK’s seek/read usage of a file while uploading it. I’m hoping to have this finished early next week.
@arjunmahishi are you also running your application in Kubernetes as well?
@arjunmahishi, we’re still attempting to reproduce this issue. I’m working on a io.ReadSeeker wrapper with additional debug/logging information that will attempt to log Seeks and reads of the underlying reader in hopes of providing insight into what/where the issue is occurring.
The change in #2636 Added unit tests to the S3 Upload manager’s error/retry behavior, asserting that the retry’s offset seeking behavior works as expected.
I think the best bet we’ll have to gain more insight into what the issue is, is to get this debug wrapper out to you to log how the file handle is being used.
@arjunmahishi Yes, conversely if the file size is smaller than the
PartSizespecified for the uploader then the file will be uploaded in a singlePUTrequest rather than a multipart upload.Thanks for reaching out to us @arjunmahishi. Do you see this issue without the retry logic around the UploadPart operation call? The SDK will automatically retry the operation if it fails to upload.
Is it possible that the
bodybytes.Readervalue needs to be seeked to the beginning of the byte buffer before retrying?Also, have you checked out the SDK’s provided S3 Upload Manager? It provides the logic to wrap multi-part upload. The uploader is also able to handle multi-part uploads from a input file. The Uploader will automatically split the file into parts for concurrent uploads.