gsutil: gsutil rsync is broken in 4.39
Command: gsutil rsync -r -j -n /source/path/ gs://$GS_STATIC_BUCKET/DIR/
Here is the error encountered on running it:
Building synchronization state… Starting synchronization… Traceback (most recent call last): File “/usr/local/bin/gsutil”, line 10, in <module> sys.exit(main()) File “/usr/local/lib/python2.7/site-packages/gslib/main.py”, line 431, in main user_project=user_project) File “/usr/local/lib/python2.7/site-packages/gslib/main.py”, line 760, in _RunNamedCommandAndHandleExceptions _HandleUnknownFailure(e) File “/usr/local/lib/python2.7/site-packages/gslib/main.py”, line 626, in _RunNamedCommandAndHandleExceptions user_project=user_project) File “/usr/local/lib/python2.7/site-packages/gslib/command_runner.py”, line 411, in RunNamedCommand return_code = command_inst.RunCommand() File “/usr/local/lib/python2.7/site-packages/gslib/commands/rsync.py”, line 1658, in RunCommand seek_ahead_iterator=seek_ahead_iterator) File “/usr/local/lib/python2.7/site-packages/gslib/command.py”, line 1515, in Apply arg_checker, should_return_results, fail_on_error) File “/usr/local/lib/python2.7/site-packages/gslib/command.py”, line 1561, in _SequentialApply args = next(args_iterator) File “/usr/local/lib/python2.7/site-packages/gslib/commands/rsync.py”, line 1188, in iter src_md5) = (self._ParseTmpFileLine(next(self.sorted_src_urls_it))) File “/usr/local/lib/python2.7/site-packages/gslib/commands/rsync.py”, line 1049, in _ParseTmpFileLine md5) = line.split() ValueError: too many values to unpack
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 10
- Comments: 24 (1 by maintainers)
Commits related to this issue
- Fix broken _UrlEncode(), improve docs, add tests. Fixes #806 Fixes #833 — committed to houglum/gsutil by houglum 5 years ago
- Fix broken _UrlEncode(), improve docs, add tests. Fixes #806 Fixes #833 — committed to houglum/gsutil by houglum 5 years ago
- Fix broken _UrlEncode(), improve docs, add tests. (#835) Fixes #806 Fixes #833 — committed to GoogleCloudPlatform/gsutil by houglum 5 years ago
Thanks @catleeball for solving the issue, any update on a ETA for the next release?
Just released 4.40 an hour or so ago. The gcloud-installed version will be a bit behind, as we have to get our release bundled into their next release cycle, but the pypi and tarball installations of 4.40 are available now.
We’re aiming to do a release early this week. Apologies for the delay - we’ve been working to finish up Python 3 support as well.
As for your question, I’d guess that you need to pull in the submodules. Run
git submodule update --init --recursiveafter cloning the repo andcding into it.I would be very happy if you could consider pushing this fix (or revert) soon, as we rely on this for our nightly backups and more, which have been failing since the 27th because of this issue.
I’m not terribly familiar with Cloud Builder, but it looks like the dockerfile for it that uses the “gcloud-slim” image? https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/gsutil/Dockerfile
If it’s possible to update the gcloud version while running that image, you could revert gcloud back to the 251.0.0 version, which was the most recent one up until they added v4.39 of gsutil (in 252.0.0), by running something like
gcloud components update --version 251.0.0. If that’s not possible, maybe roll your own temporary version of the gcloud slim dockerfile that downloads a specific version of the Cloud SDK? Looks like the gcloud slim dockerfile is here: https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/gcloud/Dockerfile.slimSorry if these are poor suggestions; I don’t really use Docker or Cloud Build in my day-to-day workflow 🙃 Other than the suggestions above, I can’t think of any alternatives that use the (current version of the) gsutil Cloud Builder image.
And apologies again for the breakage - I understand this is a pretty big pain to deal with. We’ve added some unit and integration tests to make sure object/file URIs are being properly
quote_plus-escaped to prevent this from recurring in the future.If it helps anyone - downgrading sdk works for me.
gcloud components update --version 251.0.0 --quietIn case this helps anyone;
I solved it temporarily by cloning this repository along with its submodules to a shared path on the network:
git clone --recursive …
Then I symlinked the “gsutil” file from this shared path to each place where it needs to be able to execute.
I feel 20+ days to deploy a fix (even if it is a rollback per se) for such a crucial feature as this, is completely unacceptable.
@ain the issue is fixed, but hasn’t been released with gcloud. So you need to either build up your custom image using 4.38 or below or wait until gcloud releases a new version which could take a week. Or use the solution mentioned in the comment above. Read my comment on that ticket as that is something I think I will do by default going forward.
@houglum appreciate your feedback! Thankfully there was a CloudSDK repo out there already on github which I have been able to use that allows versioned images, so didn’t have to create a custom image.
I got it to work by following this https://github.com/GoogleCloudPlatform/cloud-builders/issues/508#issuecomment-507646005 . So the solution is basically to stop using this repo for now, and use the gsuitl bundled with the google/cloud-sdk:251.0.0-slim package
@houglum What do you recommend in terms of Cloud Build? Our builds are all failing because we use the default gsutil image provided by Cloud Build and it is using the latest and greatest. Ideally, I wouldn’t want to create my own new image with the older gsutil tool as a dependency. Is there a way to specify a version of gsutil builder in cloudbuild so I could use 4.38 until the gcloud tool catches up?
Something like this could very likely happen again and this sorts of puts us in a hostage situation if we can’t use specific versions (unless we can (I am not aware)). Appreciate your feedback!
Thanks!