azure-storage-azcopy: azcopy command hangs

Which version of the AzCopy was used?

AzCopy 10.2.1

Which platform are you using? (ex: Windows, Mac, Linux)

Linux Server release 6.10

What command did you run?

azcopy ( without any argument) same thing happens for other success command , failed command errors out and doesnt get hung .

What problem was encountered?

azcopy commands are getting hung .

How can we reproduce the problem in the simplest way?

Download the mentioned version and run the command in Linux 6 or Linux 7

Have you found a mitigation/solution?

I have to press control C to cancel azcopy command to come in main prompt

# azcopy
AzCopy 10.2.1
Project URL: github.com/Azure/azure-storage-azcopy

AzCopy is a command line tool that moves data into/out of Azure Storage.
To report issues or to learn more about the tool, go to github.com/Azure/azure-storage-azcopy

The general format of the commands is: 'azcopy [command] [arguments] --[flag-name]=[flag-value]'.

Usage:
  azcopy [command]

Available Commands:
  copy        Copies source data to a destination location
  env         Shows the environment variables that can configure AzCopy's behavior
  help        Help about any command
  jobs        Sub-commands related to managing jobs
  list        List the entities in a given resource
  login       Log in to Azure Active Directory to access Azure Storage resources.
  logout      Log out to terminate access to Azure Storage resources.
  make        Create a container/share/filesystem
  remove      Delete entities from Azure Storage Blob/File/ADLS Gen2
  sync        Replicate source to the destination location

Flags:
      --cap-mbps uint32      caps the transfer rate, in Mega bits per second. Moment-by-moment throughput may vary slightly from the cap. If zero or omitted, throughput is not capped.
  -h, --help                 help for azcopy
      --output-type string   format of the command's output, the choices include: text, json. (default "text")
      --version              version for azcopy

Use "azcopy [command] --help" for more information about a command.
^C

I am attaching strace output of the this command ( just removed domain names from the output) (thanks - I’ve downloaded and saved the strace - JohnRusk)

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 54 (29 by maintainers)

Most upvoted comments

I got the same issue on ubuntu18.04(docker) with azcopy10.6.0: azcopy cp just hang with INFO: Scanning...

I solved this problem by running wget https://azcopyvnext.azureedge.net/releasemetadata/latest_version.txt beforehand. Looks like wget did some works and fixed things.

cdpath on Nov 30, 2020

This issue is fixed now, thanks for your relentless help.

For everyone benefit I had to add install root cert of ZSCALER in my linux host by adding the cert in files like /etc/pki/tls/certs/ca-bundle.trust.crt /etc/pki/tls/certs/ca-bundle.crt

And azcopy is working fine now.

imlight on Aug 8, 2019

Great, thanks. Those two pieces of information are enough to confirm what’s going on. AzCopy’s error message contains this text

certificate signed by unknown authority

I think that’s the real error when AzCopy tries to access the aka.ms link. (I don’t know why that gets wrapped in a “timeout” error, maybe because the cert error causes it to never complete).

Also, when you do wget locally, as you mentiond you are supressing the cert error with --no-check-certificate, which confirms that your local machine does not trust the cert that is being provided for the two version check URLs (the aka.ms one, and the link it redirects to).

Why doesn’t it trust that cert? Because the traffic is being intercepted and re-signed by a ZScaler device/service. (We can see that in the cert name). So the cert it sees is not the original one provided by Azure (which it would trust).

Why isn’t this a problem for the main traffic, to the login URL when you log in? I don’t know for sure, but I’m guessing that there may be an exception in ZScaler to let the login URL through un-encrypted.

But what about AzCopy traffic to your Storage account? (I.e. after you log in, and you want to transfer some data). Will it be re-signed by ZScaler. I’m not sure. That depends on how the exceptions are set up in ZScaler.

How can you make everything work?

Login. For now, the only way to get login to work is to either
1. install on your AzCopy machine a root cert that will allow that machine to verify the ZScaler cert. The ZScaler doc on this is brief, and I think you’ll probably need to get your company’s ZScaler admin person to help, if you want to install that certificate on your machine. I don’t know how to install a cert on your OS. You’ll need to look that up, or get your ZScaler admin person to help. OR
2. Get your company’s ZScaler admin person to add the two version check URLs as exceptions, that ZScaler will not intercept.
What about after you log in, how can you make real traffic to Azure storage work? As noted above, we don’t know yet whether you’ll also have the same cert problem with your actual Storage traffic. One way to test this is to do a login manually, then hit CTRL-C when it freezes (after it displays the message INFO: SPN Auth via secret succeeded.). Then you’ll be logged in. Try doing a test AzCopy transfer - e.g. move just one file, to a test container in Blob Storage. If that works, then you know that there’s no certificate problem with the main traffic, and the problem only affects the two version check URLs. But, if that fails, then you know there’s a problem for the main traffic too, and you need to either install the ZScaler cert (as above) or get your Storage URL added as an exception (as above) so that ZScaler will not intercept it. If you have the choice, getting it added as an exception is probably best for performance. When you’ve finished this test, do AzCopy logout and hit CTRL-C after it hangs, to log back out.

Hope this helps.

JohnRusk on Aug 6, 2019

OK, I’ve run some tests and found the cause of the problem. The problem only happens on help commands (i.e. AzCopy with no parameters or with --help on the command line). It happens when the domain name aka.ms cannot be resolved - because, when running a help command, AzCopy checks a specific aka.ms URL to see if a newer version has been released. Unfortunately, with help commands, that check is done synchronously - i.e. AzCopy waits until the URL is loaded. And that never happens if the domain name can’t be resolved. (E.g. your DNS or doesn’t support it, or your machine can only see Azure and not the public internet)

Fortunately, the commands that actually do stuff, e.g. copy data, all use an asynchronous version of that check (i.e. they don’t wait for the answer). So all of those commands will run fine. It is only the help commands that are affected.

I’ll make sure we fix this in a future release, but for now, you can safely use CTRL-C to exit the help commands, as you have been doing, and you won’t see the problem on the copy and sync commands.

JohnRusk on Jul 26, 2019

I am interested in latest version as it has authentication with service principal, which is not available in older versions.

imlight on Jul 26, 2019

No, This machine have 2 vcpu , and I have tried azcopy in other machine which has more vcpu . same story.

imlight on Jul 26, 2019