azure-pipelines-tasks: [BUG]: AzureResourceManagerTemplateDeployment@3 failing with: Error in setting up subscription: %s

Task name

AzureResourceManagerTemplateDeployment

Task version

3

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu 22.04

Task log

2023-08-08T12:05:16.8660029Z ##[section]Starting: Deploy Retention manager user assigned managed identity
2023-08-08T12:05:16.8668540Z ==============================================================================
2023-08-08T12:05:16.8668744Z Task         : ARM template deployment
2023-08-08T12:05:16.8668854Z Description  : Deploy an Azure Resource Manager (ARM) template to all the deployment scopes
2023-08-08T12:05:16.8669038Z Version      : 3.225.3
2023-08-08T12:05:16.8669143Z Author       : Microsoft Corporation
2023-08-08T12:05:16.8669250Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment
2023-08-08T12:05:16.8669455Z ==============================================================================
2023-08-08T12:05:20.4738632Z ARM Service Connection deployment scope - Subscription
2023-08-08T12:05:20.9223257Z Checking if the following resource group exists: 'redacted'-as-solutions-identity-rg.
2023-08-08T12:05:21.0640576Z Resource group exists: true.
2023-08-08T12:05:21.0647988Z Creating deployment parameters.
2023-08-08T12:05:24.6335625Z [command]/usr/bin/az login --service-principal -u *** --password=*** --tenant 'redacted' --allow-no-subscriptions
2023-08-08T12:05:28.8972405Z [
2023-08-08T12:05:28.9052103Z   {
2023-08-08T12:05:28.9052863Z     "cloudName": "AzureCloud",
2023-08-08T12:05:28.9055165Z     "homeTenantId": "redacted",
2023-08-08T12:05:28.9055999Z     "id": "redacted",
2023-08-08T12:05:28.9056452Z     "isDefault": true,
2023-08-08T12:05:28.9056950Z     "managedByTenants": [
2023-08-08T12:05:28.9057451Z       {
2023-08-08T12:05:28.9058319Z         "tenantId": "redacted"
2023-08-08T12:05:28.9058662Z       }
2023-08-08T12:05:28.9058820Z     ],
2023-08-08T12:05:28.9059117Z     "name": "redacted",
2023-08-08T12:05:28.9059328Z     "state": "Enabled",
2023-08-08T12:05:28.9059680Z     "tenantId": "redacted",
2023-08-08T12:05:28.9060072Z     "user": {
2023-08-08T12:05:28.9060580Z       "name": "***",
2023-08-08T12:05:28.9060793Z       "type": "servicePrincipal"
2023-08-08T12:05:28.9060976Z     }
2023-08-08T12:05:28.9061142Z   },
2023-08-08T12:05:28.9061292Z   {
2023-08-08T12:05:28.9061480Z     "cloudName": "AzureCloud",
2023-08-08T12:05:28.9061858Z     "homeTenantId": "redacted",
2023-08-08T12:05:28.9062233Z     "id": "redacted",
2023-08-08T12:05:28.9062471Z     "isDefault": false,
2023-08-08T12:05:28.9062667Z     "managedByTenants": [
2023-08-08T12:05:28.9062857Z       {
2023-08-08T12:05:28.9063180Z         "tenantId": "redacted"
2023-08-08T12:05:28.9063406Z       }
2023-08-08T12:05:28.9063556Z     ],
2023-08-08T12:05:28.9063866Z     "name": "redacted",
2023-08-08T12:05:28.9064091Z     "state": "Enabled",
2023-08-08T12:05:28.9064431Z     "tenantId": "redacted",
2023-08-08T12:05:28.9064657Z     "user": {
2023-08-08T12:05:28.9065067Z       "name": "***",
2023-08-08T12:05:28.9065257Z       "type": "servicePrincipal"
2023-08-08T12:05:28.9065451Z     }
2023-08-08T12:05:28.9065599Z   },
2023-08-08T12:05:28.9065759Z   {
2023-08-08T12:05:28.9065921Z     "cloudName": "AzureCloud",
2023-08-08T12:05:28.9066278Z     "homeTenantId": "redacted",
2023-08-08T12:05:28.9066637Z     "id": "redacted",
2023-08-08T12:05:28.9066861Z     "isDefault": false,
2023-08-08T12:05:28.9067044Z     "managedByTenants": [
2023-08-08T12:05:28.9067216Z       {
2023-08-08T12:05:28.9067784Z         "tenantId": "redacted"
2023-08-08T12:05:28.9067970Z       },
2023-08-08T12:05:28.9068111Z       {
2023-08-08T12:05:28.9068401Z         "tenantId": "redacted"
2023-08-08T12:05:28.9068621Z       }
2023-08-08T12:05:28.9068769Z     ],
2023-08-08T12:05:28.9069060Z     "name": "redacted",
2023-08-08T12:05:28.9069284Z     "state": "Enabled",
2023-08-08T12:05:28.9069811Z     "tenantId": "redacted",
2023-08-08T12:05:28.9070052Z     "user": {
2023-08-08T12:05:28.9070405Z       "name": "***",
2023-08-08T12:05:28.9070616Z       "type": "servicePrincipal"
2023-08-08T12:05:28.9070773Z     }
2023-08-08T12:05:28.9070916Z   }
2023-08-08T12:05:28.9071059Z ]
2023-08-08T12:05:28.9076545Z [command]/usr/bin/az account set --subscription redacted
2023-08-08T12:05:31.9172865Z ERROR: The subscription of 'redacted' doesn't exist in cloud 'AzureCloud'.
2023-08-08T12:05:31.9290373Z ##[error]Error Code: [1]
2023-08-08T12:05:31.9585148Z ##[error]Error: Error in setting up subscription: %s
2023-08-08T12:05:31.9586696Z ##[error]Check out the troubleshooting guide to see if your issue is addressed: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?view=azure-devops#troubleshooting
2023-08-08T12:05:31.9588187Z ##[error][object Object]
2023-08-08T12:05:32.0435492Z ##[section]Finishing: Deploy Retention manager user assigned managed identity

Relevant log output

/usr/bin/az account set --subscription 'redactedsubscriptionid'
ERROR: The subscription of 'redactedsubscriptionid' doesn't exist in cloud 'AzureCloud'.
##[error]Error Code: [1]
##[error]Error: Error in setting up subscription: %s
##[error]Check out the troubleshooting guide to see if your issue is addressed: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?view=azure-devops#troubleshooting
##[error][object Object]
Finishing: Deploy Retention manager user assigned managed identity

Aditional info

This issue started happening intermittently 08/08/2023 at ~11:00 CEST

The issue is affecting seemingly all AzureResourceManagerTemplateDeployment@3 steps in all our pipelines at random.

The results is not reproducible because different tasks fail each time, and sometimes all task succeeds.

No change has been made in the code involving the failing tasks, or dependencies to said tasks.