gsutil: gsutil rsync hangs on large files

Hi! We are using gsutil rsync to upload our backups to coldline storage. After our files become larger (more than 30GB) rsync starts hanging on each such file. The same situation is for gsutil cp. I’ve already sent a log file with gsutil -D rsync to gs-team@google.com

gsutil version: 4.25
boto version: 2.42.0
python version: 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
OS: Linux 4.4.0-77-generic
multiprocessing available: True
using cloud sdk: True
config path(s): /etc/boto.cfg
gsutil path: /usr/lib/google-cloud-sdk/platform/gsutil/gsutil
compiled crcmod: True
installed via package manager: False
editable install: False

About this issue

Original URL
State: open
Created 7 years ago
Reactions: 5
Comments: 15

Most upvoted comments

Hello, I’m trying to rsync from gcs to s3 buckets but some files are bigger than 5 GiB.

I receive this error: “exceeds the maximum gsutil-supported size for an S3 upload. S3 objects greater than 5 GiB in size require multipart uploads, which gsutil does not support.”

As you can see here https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html there is a limitation in S3 where you have to use multipart upload for files bigger than 5 GiB.

Please help me, how can I fix that?

Any help will be really appreciated! thanks @houglum for your time

Best Regards Fabio Rigato

fabio-rigato on Feb 20, 2020