azure-storage-azcopy: azcopy command hangs
Which version of the AzCopy was used?
AzCopy 10.2.1
Which platform are you using? (ex: Windows, Mac, Linux)
Linux Server release 6.10
What command did you run?
azcopy ( without any argument) same thing happens for other success command , failed command errors out and doesnt get hung .
What problem was encountered?
azcopy commands are getting hung .
How can we reproduce the problem in the simplest way?
Download the mentioned version and run the command in Linux 6 or Linux 7
Have you found a mitigation/solution?
NO
I have to press control C to cancel azcopy command to come in main prompt
# azcopy
AzCopy 10.2.1
Project URL: github.com/Azure/azure-storage-azcopy
AzCopy is a command line tool that moves data into/out of Azure Storage.
To report issues or to learn more about the tool, go to github.com/Azure/azure-storage-azcopy
The general format of the commands is: 'azcopy [command] [arguments] --[flag-name]=[flag-value]'.
Usage:
azcopy [command]
Available Commands:
copy Copies source data to a destination location
env Shows the environment variables that can configure AzCopy's behavior
help Help about any command
jobs Sub-commands related to managing jobs
list List the entities in a given resource
login Log in to Azure Active Directory to access Azure Storage resources.
logout Log out to terminate access to Azure Storage resources.
make Create a container/share/filesystem
remove Delete entities from Azure Storage Blob/File/ADLS Gen2
sync Replicate source to the destination location
Flags:
--cap-mbps uint32 caps the transfer rate, in Mega bits per second. Moment-by-moment throughput may vary slightly from the cap. If zero or omitted, throughput is not capped.
-h, --help help for azcopy
--output-type string format of the command's output, the choices include: text, json. (default "text")
--version version for azcopy
Use "azcopy [command] --help" for more information about a command.
^C
I am attaching strace output of the this command ( just removed domain names from the output) (thanks - I’ve downloaded and saved the strace - JohnRusk)
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 54 (29 by maintainers)
I got the same issue on ubuntu18.04(docker) with azcopy10.6.0:
azcopy cp
just hang withINFO: Scanning...
I solved this problem by running
wget https://azcopyvnext.azureedge.net/releasemetadata/latest_version.txt
beforehand. Looks likewget
did some works and fixed things.This issue is fixed now, thanks for your relentless help.
For everyone benefit I had to add install root cert of ZSCALER in my linux host by adding the cert in files like /etc/pki/tls/certs/ca-bundle.trust.crt /etc/pki/tls/certs/ca-bundle.crt
And azcopy is working fine now.
Great, thanks. Those two pieces of information are enough to confirm what’s going on. AzCopy’s error message contains this text
I think that’s the real error when AzCopy tries to access the aka.ms link. (I don’t know why that gets wrapped in a “timeout” error, maybe because the cert error causes it to never complete).
Also, when you do wget locally, as you mentiond you are supressing the cert error with
--no-check-certificate
, which confirms that your local machine does not trust the cert that is being provided for the two version check URLs (the aka.ms one, and the link it redirects to).Why doesn’t it trust that cert? Because the traffic is being intercepted and re-signed by a ZScaler device/service. (We can see that in the cert name). So the cert it sees is not the original one provided by Azure (which it would trust).
Why isn’t this a problem for the main traffic, to the login URL when you log in? I don’t know for sure, but I’m guessing that there may be an exception in ZScaler to let the login URL through un-encrypted.
But what about AzCopy traffic to your Storage account? (I.e. after you log in, and you want to transfer some data). Will it be re-signed by ZScaler. I’m not sure. That depends on how the exceptions are set up in ZScaler.
How can you make everything work?
Login. For now, the only way to get login to work is to either
What about after you log in, how can you make real traffic to Azure storage work? As noted above, we don’t know yet whether you’ll also have the same cert problem with your actual Storage traffic. One way to test this is to do a login manually, then hit CTRL-C when it freezes (after it displays the message
INFO: SPN Auth via secret succeeded.
). Then you’ll be logged in. Try doing a test AzCopy transfer - e.g. move just one file, to a test container in Blob Storage. If that works, then you know that there’s no certificate problem with the main traffic, and the problem only affects the two version check URLs. But, if that fails, then you know there’s a problem for the main traffic too, and you need to either install the ZScaler cert (as above) or get your Storage URL added as an exception (as above) so that ZScaler will not intercept it. If you have the choice, getting it added as an exception is probably best for performance. When you’ve finished this test, doAzCopy logout
and hit CTRL-C after it hangs, to log back out.Hope this helps.
OK, I’ve run some tests and found the cause of the problem. The problem only happens on help commands (i.e.
AzCopy
with no parameters or with --help on the command line). It happens when the domain name aka.ms cannot be resolved - because, when running a help command, AzCopy checks a specific aka.ms URL to see if a newer version has been released. Unfortunately, with help commands, that check is done synchronously - i.e. AzCopy waits until the URL is loaded. And that never happens if the domain name can’t be resolved. (E.g. your DNS or doesn’t support it, or your machine can only see Azure and not the public internet)Fortunately, the commands that actually do stuff, e.g. copy data, all use an asynchronous version of that check (i.e. they don’t wait for the answer). So all of those commands will run fine. It is only the help commands that are affected.
I’ll make sure we fix this in a future release, but for now, you can safely use CTRL-C to exit the help commands, as you have been doing, and you won’t see the problem on the copy and sync commands.
I am interested in latest version as it has authentication with service principal, which is not available in older versions.
No, This machine have 2 vcpu , and I have tried azcopy in other machine which has more vcpu . same story.