azure-pipelines-tasks: [BUG]: AzureFileCopy to VM fails
New issue checklist
- I searched for existing GitHub issues
- I read pipeline troubleshooting guide
- I checked how to collect logs
Task name
AzureFilecopy
Task version
5.237.1
Issue Description
The AzureFileCopy command started failing last week after presumably being updated. The error being reported is:
2024-03-21T10:42:55.7596977Z -ExecutionPolicy Unrestricted -Command “. 2024-03-21T10:42:55.7597475Z ‘C:\Users\adminuser\AppData\Local\Temp\0a0b6c85-26a3-4c18-803e-537ce07dce30.ps1’” 2024-03-21T10:42:55.7597735Z 2024-03-21T10:42:55.7739358Z VERBOSE: Using default AzCopy arguments for dowloading to VM 2024-03-21T10:42:55.7740270Z 2024-03-21T10:42:55.7787839Z VERBOSE: ##[command] & azcopy copy 2024-03-21T10:42:55.7788454Z 2024-03-21T10:42:56.4219789Z “https://XXXdevconfigstorage.blob.core.windows.net/e576feb6-e8bb-47a2-8806-55dc5fea1c13*****” 2024-03-21T10:42:56.4220051Z 2024-03-21T10:42:56.4220291Z "c:\temp" --recursive --log-level=INFO 2024-03-21T10:42:56.4220397Z 2024-03-21T10:42:56.4220445Z 2024-03-21T10:42:56.4220489Z 2024-03-21T10:42:56.4220763Z failed to parse user input due to error: cannot use wildcards in the path section of the URL except in trailing “/*”. If you wish to use * in your URL, manually encode it to %2A 2024-03-21T10:42:56.4220980Z
Apparently the SAS token is being added to the URL but it is missing the ? separator.
We have implemented a workaround as proposed by another user using two tasks, one to upload to the blob and another PowerShell script to copy from the blob to the VM but this has started to fail this week.
The problem with the workaround is that it appears that the azcopy command is not being found on the Hosted Agent so the workaround no longer works for us. The error reported is:
2024-03-25T16:45:00.5157506Z azcopy : The term ‘azcopy’ is not recognized as the name of a cmdlet, function, script file, or
2024-03-25T16:45:00.5157839Z At line:1 char:1
2024-03-25T16:45:00.5158421Z + & ‘C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe’ -NoLogo …
2024-03-25T16:45:00.5159704Z + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2024-03-25T16:45:00.5160428Z + CategoryInfo : NotSpecified: (azcopy : The te…cript file, or :String) [], RemoteException
2024-03-25T16:45:00.5160843Z + FullyQualifiedErrorId : NativeCommandError
2024-03-25T16:45:00.5161132Z
2024-03-25T16:45:00.5161307Z
2024-03-25T16:45:00.5264081Z operable program. Check the spelling of the name, or if a path was included, verify that the path
2024-03-25T16:45:00.5264566Z
2024-03-25T16:45:00.5344297Z is correct and try again.
Currently we can’t release any of our software so development and testing teams are blocked.
Environment type (Please select at least one enviroment where you face this issue)
- Self-Hosted
- Microsoft Hosted
- VMSS Pool
- Container
Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
Windows Server 2019/2022
Operation system
Windows Server 2019/2022
Relevant log output
Errors are inline above.
Full task logs with system.debug enabled
[REPLACE THIS WITH YOUR INFORMATION]
Repro steps
An AzureFileCopy task to copy a file onto a VM was working correctly great before last week but started to fail. We noted that the build version of the task changed from 5.236.1 (worked okay) to 5.237.1 (fails).
About this issue
- Original URL
- State: closed
- Created 3 months ago
- Comments: 24 (5 by maintainers)
We still have version 5.237.11 in our deployments. We use Azure Hosted agents in region West Europe. When is the fix in version 5.237.12 available in West Europe?
I have been informed that the new task deployment cycle was started yesterday, and latest version (i.e 5.237.12) is already out on few regions. It might take the next 2-4 working days to be rolled out on all the regions.
I have reviewed the PR and testing the same in our env as well.
@v-mohithgc I can confirm that your changes are correct, we used the AzureFileCopy version 5 and issues on the failure on that stage are resolved, We need to merge this PR asap guys.
@v-mohithgc Hi, versioning of the tasks seems odd to me. When you complete an update rather than adding a new task version you seem to update all versions. This means that if we see a failure we don’t have any option to use another version. Logically I’d expect new versions to be added so AzureFileCopy@7 which has the new code in. We’ve seen several failures with AzureFileCopy over the past few months and it breaks our pipelines and we don’t have a way to revert unless you do so; we’ve lost hours of dev/test through the breakage. Just a thought…
My agents are also azure hosted, i can give this a try to use a self hosted agent as well, But can’t we use the updated task from the org. on the azure hosted agents as well.
Can confirm the same behaviour for our pipeline as well, Attaching logs for reference (removed the names for files and machines),
Re-ran the pipelines
Re-ran the deployment pipelines, are the Azure file copy task’s revert didn’t helped with the issue, We have used the AzureFileCopy version 3,4 and 5 as well The problem we are facing lies in the Azure PS 11.0.0+ . More specifically Az.Storage 6.0.0+. There was a breaking change in New-AzStorageContainerSASToken – the return value is no longer prefixed with a question mark. The code of the tasks relies on that, though.
https://learn.microsoft.com/en-us/powershell/azure/release-notes-azureps#azstorage-600 https://learn.microsoft.com/en-us/powershell/azure/migrate-az-11.0.0#azstorage
The task log is not very helpful as it displays the source URI with five asterisks appended, while in reality, a slash, an asterisk, and the SAS token are appended.
The New-AzStorageContainerSASToken cmdlet is called in Generate-AzureStorageContainerSASToken defined in AzureUtilityAz1.0.ps1. That one is called from Copy-FilesToAzureVMsFromStorageContainer defined in Utility.ps1, which is called from AzureFileCopy.ps1 itself. There, it can be seen that Copy-FilesToAzureVMsFromStorageContainer gets the “fileCopyJobScript” scriptblock from AzureFileCopyRemoteJob.ps1. At the bottom, see the uses of containerSasToken: “$containerURL/*$containerSasToken”
We are also facing the same issue with the AzureFileCopy task, trying to use workaround with azcopy to copy the builds to the VMs, Those are also failing as well, all of the deployments have been blocked for the team.