minio-go: File corruption on S3 upload
Hi,
I’m working for Exoscale, a cloud company the offers an S3-compatible Object Storage service named SOS. We’ve noticed that content corruption occurs when uploading large files on SOS with the minio-go package, which doesn’t happen with the same file on the same service endpoint using s3cmd:
$ md5sum largefile
e3fcf94aad137e4d8da9ef04c5647d57 largefile
$ s3cmd put largefile s3://marc-templates/largefile-s3cmd
upload: 'largefile' -> 's3://marc-templates/largefile-s3cmd' [part 1 of 83, 15MB] [1 of 1]
15728640 of 15728640 100% in 2s 7.22 MB/s done
...
upload: 'largefile' -> 's3://marc-templates/largefile-s3cmd' [part 83 of 83, 14MB] [1 of 1]
14811296 of 14811296 100% in 1s 7.09 MB/s done
$ exo sos upload -p largefile-exo marc-templates largefile
done! 100 % [==============================================================] largefile
$ s3cmd ls --list-md5 s3://marc-templates/
2019-06-19 14:52 1304559776 ca211e79fdd8c34a1231093eaa0dfa78-20 s3://marc-templates/largefile-exo
2019-06-19 14:41 1304559776 e3fcf94aad137e4d8da9ef04c5647d57 s3://marc-templates/largefile-s3cmd
It is possible that we may have messed up in our CLI implementation (available here), but it may also be a bug in this package. Note that we don’t see this issue with smaller files.
Could you please have a look?
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 18 (12 by maintainers)
The correct fix is sent to exos/cli https://github.com/exoscale/cli/pull/157/files - closing this issue.
Btw you should also depend on the v6 path
https://github.com/exoscale/cli/blob/master/go.mod#L21
Should be
github.com/minio/minio-go/v6and use the latest.