azure-cli: CONNECTIVITY_REFRESH_TOKEN_ERROR when using `az acr` with AzureCli@2

Related command

az acr repository update --name $TargetAcr `
    --image "$ArtifactToPromote" `
    --write-enabled false `
    --delete-enabled false

Describe the bug

We have a script that contains a bunch of az acr import and az acr repository update commands. It is a powershell core script, which itself is running in the Azure DevOps pipeline, inside AzureCli@2 task. The script is basically simple script to import images to ACR and lock them. It works fine until following error appears sometimes(!) near the end of the script:

WARNING: Unable to get AAD authorization tokens with message: 2022-12-12 16:05:11.564324 An error occurred: CONNECTIVITY_REFRESH_TOKEN_ERROR
Access to registry '<redacted>.azurecr.io' was denied. Response code: 429. Please try running 'az login' again to refresh permissions.

After this error, basically nothing can be done but running the build again. Retry does not help because az has lost the service principal context, which was given by AzureCli@2 task.

To Reproduce Execute a script that contains a bunch (about 20) of az acr import and az acr repository update commands ran sequentially.

Expected behavior Assuming that this error comes from ACR throttling (deducted from 429 status code) a retry of az acr repository update within the script should work but it doesn’t. The ‘az cli’ demands to run az login again, but that is not possible within AzureCli@2 task. Running of az login is not an option because the script is invoked inside of AzureCli@2 task which has a service principal context assigned through a service connection.

Environment summary Latest Azure Devops ubuntu agent AzureCli@2

Additional context

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 17 (9 by maintainers)

Commits related to this issue

{ACR} Fix #24886: `az acr`: Improve the 429 error handling for CONNECTIVITY_REFRESH_TOKEN_ERROR — committed to jasminetMSFT/azure-cli by jasminetMSFT a year ago
[ACR] Fix #24886: `az acr`: Improve the 429 error handling for CONNECTIVITY_REFRESH_TOKEN_ERROR (#25549) — committed to Azure/azure-cli by jasminetMSFT a year ago
[ACR] Fix #24886: `az acr`: Improve the 429 error handling for CONNECTIVITY_REFRESH_TOKEN_ERROR (#25549) — committed to avgale/azure-cli by jasminetMSFT a year ago

Most upvoted comments

It has a bug label, @msftbot should be closing this.

nosalan on Jan 31, 2023

We also received an internal email regarding a similar issue: Azure CLI + Azure Container Registry error

The error message in that email is:

This operation will delete the manifest 'sha256:xxx' and all the following images: 'xxx'
Unable to get AAD authorization tokens with message: 2022-12-25 16:54:47.327933 An error occurred: CONNECTIVITY_REFRESH_TOKEN_ERROR
Access to registry 'xxx.azurecr.io' was denied. Response code: 401. Please try running 'az login' again to refresh permissions.

but the response code is 401, instead of 429.

The email also mentions:

But the thing is that it was actually a transient error, my credentials were ok and when I retried the command (without running az login) it worked correctly.

so this seems to be some permission/propagation/throttling issue with ACR.

The error message Please try running 'az login' again to refresh permissions. is thrown by ACR, not Azure CLI core: https://github.com/Azure/azure-cli/blob/69e79fd69b81b28f077c365ed9502d27342ae7f7/src/azure-cli/azure/cli/command_modules/acr/_errors.py#L115-L118

Since retrying the command works without running az login, the solution would be

ACR command retries the HTTP request
The script containing az acr command retries the command

az login will only update the refresh token and access token for ARM, not ACR’s access token, so I am not sure running az login will help here. ACR command module may need to update the error message to make it more accurate and less misleading.

jiasli on Jan 4, 2023