gsutil: Cannot download file from the bucket with colon (:) in the name

The command I ran:

gsutil -m cp -r "gs://path/to/bucket" .

Problematic file name:

2022-04-08T17:13:08.136Z-735ec64eb2fb7a61c37a26112e920abe52cdf36f.json
_____________^

Notice the colon in the name that delimits hours:minutes:seconds. Windows file system doesn’t allow files with this character in the filename.

The issue seems to exist for a long time so Windows users need to apply a patch described here.

Is there a chance this patch to be applied in the next release of gsutil?

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 4
  • Comments: 15 (7 by maintainers)

Most upvoted comments

Substituting invalid Windows characters should be the default behavior in the next release of gcloud alpha storage (probably going public around next Wednesday).

P.S. We replaced “�” with “$” for terminals that don’t support Unicode.

Hi, I’ve added this to our backlog. We plan on replacing colons with a character like “�”.

This is to avoid leaving people with two sets of correct-looking file names (one with colons and one with hyphens), which could lead to confusion and doesn’t indicate lack of Windows support for colons.

Ideally, this is implemented somewhere like FileUrl to support the character conversion in all places on Windows.

We also plan on introducing this to gcloud storage 😃

It seems like you could have at least failed fast when you encountered : instead of letting gsutil waste huge gobs of api calls for millions of files (loki filenames all have many :s in them).

Actually, it isn’t failing, it’s just generating a message that looks like a fatal error but only relating to the gstmp file… which means the data was actually retrieved.

This is a lousy user experience. Failing fast and suggesting gcloud storage rsync would have been a much better experience.

@thomasmaclean wisely suggested expanding the conversions to more banned characters for Windows file names "*/:<>?\| We might have to do something like " -> �1, * -> �2, … to prevent naming conflicts. For example, we don’t want a file name collision if we attempt to convert both foo<.txt and foo>.txt to foo�.txt.

@goodwin64 , since this is growing into a slightly larger project, would you care if we implemented this in only gcloud storage? It’s the future storage CLI: http://cloud/sdk/gcloud/reference/alpha/storage