azure-pipelines-tasks: [BUG]: AzureFileCopy to VM fails

We still have version 5.237.11 in our deployments. We use Azure Hosted agents in region West Europe. When is the fix in version 5.237.12 available in West Europe?

I see that this has been merged a week ago, however it has not been released yet (we have 5.237.11 btw.)

When can we expect a release of the fix as all deployments are currently dead?

I have been informed that the new task deployment cycle was started yesterday, and latest version (i.e 5.237.12) is already out on few regions. It might take the next 2-4 working days to be rolled out on all the regions.

Hi @AndrewBates666 @gusAntonis , I have created a PR #19700 for the proposed changes. I will try to get in touch with task owners to get to approved and merged ASAP.

Meanwhile, since i am unable to repro the issue from my end, I might need anyone of you to assist here, to follow the below simple procedure to test if it fixes the issue. If the changes do work, same can be followed to unblock your case here immediately without need to wait for regular task deployment (which might take upto weeks)

Local task test Have code changes ready (checkout to this branch), navigate to task root path ie C:…\azure-pipelines-tasks Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task

Step 1: run “npm i”

Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login Service URL : [your task test org url] PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\ AzureFileCopyV5

New task version will be uploaded to your org, please test the changes by running the pipeline.

Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

I have reviewed the PR and testing the same in our env as well.

@v-mohithgc I can confirm that your changes are correct, we used the AzureFileCopy version 5 and issues on the failure on that stage are resolved, We need to merge this PR asap guys.

@v-mohithgc Hi, versioning of the tasks seems odd to me. When you complete an update rather than adding a new task version you seem to update all versions. This means that if we see a failure we don’t have any option to use another version. Logically I’d expect new versions to be added so AzureFileCopy@7 which has the new code in. We’ve seen several failures with AzureFileCopy over the past few months and it breaks our pipelines and we don’t have a way to revert unless you do so; we’ve lost hours of dev/test through the breakage. Just a thought…

Hi @AndrewBates666 @gusAntonis , I have created a PR #19700 for the proposed changes. I will try to get in touch with task owners to get to approved and merged ASAP.

Meanwhile, since i am unable to repro the issue from my end, I might need anyone of you to assist here, to follow the below simple procedure to test if it fixes the issue. If the changes do work, same can be followed to unblock your case here immediately without need to wait for regular task deployment (which might take upto weeks)

Local task test Have code changes ready (checkout to this branch), navigate to task root path ie C:…\azure-pipelines-tasks Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task

Step 1: run “npm i”

Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login Service URL : [your task test org url] PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\ AzureFileCopyV5

New task version will be uploaded to your org, please test the changes by running the pipeline.

Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

Hi @AndrewBates666 @gusAntonis , I have created a PR #19700 for the proposed changes. I will try to get in touch with task owners to get to approved and merged ASAP. Meanwhile, since i am unable to repro the issue from my end, I might need anyone of you to assist here, to follow the below simple procedure to test if it fixes the issue. If the changes do work, same can be followed to unblock your case here immediately without need to wait for regular task deployment (which might take upto weeks) Local task test Have code changes ready (checkout to this branch), navigate to task root path ie C:…\azure-pipelines-tasks Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task Step 1: run “npm i” Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login Service URL : [your task test org url] PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\ AzureFileCopyV5

New task version will be uploaded to your org, please test the changes by running the pipeline. Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

I have reviewed the PR and testing the same in our env as well.

We can only test in the Hosted Agents so let us know when it will be released and we will run tests on it. Best A

once @gusAntonis confirms the changes to be working fine, I will proceed to get in touch with relevant task owners to be get it merged.

My agents are also azure hosted, i can give this a try to use a self hosted agent as well, But can’t we use the updated task from the org. on the azure hosted agents as well.

Transfer to the blob appeared to work okay but it’s the azcopy to the VM that is raising the error. Confirmed that 5.236.1 was the last successful version run.

Log file for the failure is:

2024-03-26T08:48:40.4356045Z INFO: Logout succeeded. 2024-03-26T08:48:41.0590923Z INFO: AzCopy.exe: A newer version 10.23.0 is available to download 2024-03-26T08:48:41.0591071Z 2024-03-26T08:48:41.0600519Z INFO: AzCopy.exe: A newer version 10.23.0 is available to download 2024-03-26T08:48:41.0601575Z 2024-03-26T08:48:50.0901423Z PSSession created for Machines:‘utcbldevsol.uksouth.cloudapp.azure.com’ 2024-03-26T08:48:50.4821322Z ================================================ utcbldevsol.uksouth.cloudapp.azure.com ================================================ 2024-03-26T08:48:55.4789167Z ================================================ utcbldevsol.uksouth.cloudapp.azure.com ================================================ 2024-03-26T08:48:55.4879442Z ##[command]& ‘C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe’ -NoLogo -NoProfile -NonInteractive 2024-03-26T08:48:55.4879760Z -ExecutionPolicy Unrestricted -Command “. 2024-03-26T08:48:55.4880058Z ‘C:\Users\adminuser\AppData\Local\Temp\17c14967-f0bd-4737-a6b1-a9fa1354e0a1.ps1’” 2024-03-26T08:48:55.4930051Z Directory: C: 2024-03-26T08:48:55.4962444Z Mode LastWriteTime Length Name 2024-03-26T08:48:55.4973726Z ---- ------------- ------ ---- 2024-03-26T08:48:55.4983831Z d----- 3/26/2024 8:48 AM ADO_AzCopyV10 2024-03-26T08:48:55.4994545Z VERBOSE: Preparing to expand… 2024-03-26T08:48:55.5004869Z VERBOSE: Created ‘C:\ADO_AzCopyV10\AzCopy\AzCopy.exe’. 2024-03-26T08:49:00.4981637Z ================================================ utcbldevsol.uksouth.cloudapp.azure.com ================================================ 2024-03-26T08:49:00.5007325Z failed to parse user input due to error: cannot use wildcards in the path section of the URL except in trailing “/*”. If you wish to use * in your URL, manually encode it to %2A

Can confirm the same behaviour for our pipeline as well, Attaching logs for reference (removed the names for files and machines),

##[debug]Leaving Invoke-RemoteScript.
##[debug]Parallel file copy: Invoke-RemoteScript threw exception
##[debug]Microsoft.PowerShell.Commands.WriteErrorException: Non-Zero exit code: '1' for ComputerName: 'VIMBC001'
##[debug]Processed: ##vso[task.logissue type=error;code={"Task_Internal_Error":"CopyingToAzureVMFailed"};]
##[debug][Azure Call]Deleting container: 074e42-9fdd-a9289a670df7 in storage account: pstas001
##[debug]Removed container '07a402-9fdd-a9289670df7' successfully.
##[debug][Azure Call]Deleted container: 07a4e42-9fdd-a9289a670df7 in storage account: stosa001
##[debug]Completed Azure File Copy Task for Azure VMs Destination
##[debug]Leaving D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.237.1\AzureFileCopy.ps1.
##[debug]Trying to disconnect from Azure and clear context at process scope
##[debug]Cannot verify the Microsoft .NET Framework version 4.7.2 because it is not included in the list of permitted versions.
##[debug]Populating RepositorySourceLocation property for module Az.Accounts.
##[debug]Loading module from path 'C:\Modules\az_11.3.1\Az.Accounts\2.16.0\Az.Accounts.psm1'.
Disconnect-AzAccount -Scope Process -ErrorAction Stop
Clear-AzContext -Scope Process -ErrorAction Stop
##[debug]Caught exception from task script.
##[debug]Error record:
##[debug]Set-TaskResult : Non-Zero exit code: '1' for ComputerName: 'IMBC001'
##[debug]At D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.237.1\ps_modules\RemoteDeployer\RemoteDeployer.psm1:134 char:9
##[debug]+         Set-TaskResult -jobResults $jobResults -machinesCount $target ...
##[debug]+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
##[debug]    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
##[debug]    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Set-TaskResult
##[debug] 
##[debug]Script stack trace:
##[debug]at Set-TaskResult, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.237.1\ps_modules\RemoteDeployer\RemoteDeployer.Utility.ps1: line 138
##[debug]at Invoke-RemoteScript, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.237.1\ps_modules\RemoteDeployer\RemoteDeployer.psm1: line 134
##[debug]at Copy-FilesParallellyToAzureVMs, D:\a\_tasks\AzureFileCopy_eb72cb01427b-a8a1-1b31ccac8a43\4.237.1\Utility.ps1: line 871
##[debug]at Copy-FilesToAzureVMsFromStorageContainer, D:\a\_tasks\AzureFileCopye5-427b-a8a1-1b31ccac8a43\4.237.1\Utility.ps1: line 1015
##[debug]at <ScriptBlock>, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.237.1\AzureFileCopy.ps1: line 260
##[debug]at <ScriptBlock>, <No file>: line 1
##[debug]at <ScriptBlock>, <No file>: line 22
##[debug]at <ScriptBlock>, <No file>: line 18
##[debug]at <ScriptBlock>, <No file>: line 1
##[debug]Exception:
##[debug]Microsoft.PowerShell.Commands.WriteErrorException: Non-Zero exit code: '1' for ComputerName: 'IMBC001'
##[error]Non-Zero exit code: '1' for ComputerName: 'PVIMC001'
##[debug]Processed: ##vso[task.logissue type=error]Non-Zero exit code: '1' for ComputerName: 'VIMBC001'
##[debug]Processed: ##vso[task.complete result=Failed]

Re-ran the pipelines

Hi all, the task version is now reverted to 5.236.1. can you please rerun your pipeline and confirm back with the result. Thanks

Re-ran the deployment pipelines, are the Azure file copy task’s revert didn’t helped with the issue, We have used the AzureFileCopy version 3,4 and 5 as well The problem we are facing lies in the Azure PS 11.0.0+ . More specifically Az.Storage 6.0.0+. There was a breaking change in New-AzStorageContainerSASToken – the return value is no longer prefixed with a question mark. The code of the tasks relies on that, though.

https://learn.microsoft.com/en-us/powershell/azure/release-notes-azureps#azstorage-600 https://learn.microsoft.com/en-us/powershell/azure/migrate-az-11.0.0#azstorage

The task log is not very helpful as it displays the source URI with five asterisks appended, while in reality, a slash, an asterisk, and the SAS token are appended.

The New-AzStorageContainerSASToken cmdlet is called in Generate-AzureStorageContainerSASToken defined in AzureUtilityAz1.0.ps1. That one is called from Copy-FilesToAzureVMsFromStorageContainer defined in Utility.ps1, which is called from AzureFileCopy.ps1 itself. There, it can be seen that Copy-FilesToAzureVMsFromStorageContainer gets the “fileCopyJobScript” scriptblock from AzureFileCopyRemoteJob.ps1. At the bottom, see the uses of containerSasToken: “$containerURL/*$containerSasToken”

We are also facing the same issue with the AzureFileCopy task, trying to use workaround with azcopy to copy the builds to the VMs, Those are also failing as well, all of the deployments have been blocked for the team.