minio-go: File corruption on S3 upload

Hi,

I’m working for Exoscale, a cloud company the offers an S3-compatible Object Storage service named SOS. We’ve noticed that content corruption occurs when uploading large files on SOS with the minio-go package, which doesn’t happen with the same file on the same service endpoint using s3cmd:

$ md5sum largefile
e3fcf94aad137e4d8da9ef04c5647d57  largefile

$ s3cmd put largefile s3://marc-templates/largefile-s3cmd
upload: 'largefile' -> 's3://marc-templates/largefile-s3cmd'  [part 1 of 83, 15MB] [1 of 1]
 15728640 of 15728640   100% in    2s     7.22 MB/s  done
...
upload: 'largefile' -> 's3://marc-templates/largefile-s3cmd'  [part 83 of 83, 14MB] [1 of 1]
 14811296 of 14811296   100% in    1s     7.09 MB/s  done

$ exo sos upload -p largefile-exo marc-templates largefile
done! 100 % [==============================================================] largefile

$ s3cmd ls --list-md5 s3://marc-templates/
2019-06-19 14:52 1304559776   ca211e79fdd8c34a1231093eaa0dfa78-20  s3://marc-templates/largefile-exo
2019-06-19 14:41 1304559776   e3fcf94aad137e4d8da9ef04c5647d57  s3://marc-templates/largefile-s3cmd

It is possible that we may have messed up in our CLI implementation (available here), but it may also be a bug in this package. Note that we don’t see this issue with smaller files.

Could you please have a look?

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 18 (12 by maintainers)

Most upvoted comments

The correct fix is sent to exos/cli https://github.com/exoscale/cli/pull/157/files - closing this issue.

Btw you should also depend on the v6 path

https://github.com/exoscale/cli/blob/master/go.mod#L21

Should be github.com/minio/minio-go/v6 and use the latest.