azure-cli: az storage blob upload-batch returns ErrorCode:BlobAlreadyExists error v2.34.0 on upload
Describe the bug
Trying to upload a file to the ‘$web’ container for a static site with az storage blob upload-batch
from GitHub Actions is failing on the latest version of the cli v2.34.0 saying that the blob already exists. The errors says to use --overwrite
.
ERROR: The specified blob already exists.
RequestId:5a838987-601e-002b-6577-2d31d6000000
Time:2022-03-01T14:17:46.5388178Z
ErrorCode:BlobAlreadyExists
If you want to overwrite the existing one, please add --overwrite in your command.
Adding --overwrite command results in this error:
ERROR: argument --overwrite: expected one argument
To Reproduce Setup a GitHub Actions workflow to upload directory to a storage blob;
- name: Login to Azure CLI
uses: azure/login@v1
with:
creds: ${{ env.AZURE_CREDENTIALS }}
- name: Upload to blob storage
uses: azure/CLI@v1
env:
STORAGE_NAME: sadatastoresearchdev
with:
azcliversion: latest
inlineScript: |
az storage blob upload-batch --account-name ${{ env.STORAGE_NAME }} -d '$web' -s dist/
Expected behavior You should be able to upload the blob and overwrite as you could before.
Environment summary
https://github.com/Azure/cli in GitHub Actions
Additional context
Specifying an older azcliversion version v2.33.1 in the workflow allow you to upload the blob.
- name: Login to Azure CLI
uses: azure/login@v1
with:
creds: ${{ env.AZURE_CREDENTIALS }}
- name: Upload to blob storage
uses: azure/CLI@v1
env:
STORAGE_NAME: sadatastoresearchdev
with:
azcliversion: 2.33.1
inlineScript: |
az storage blob upload-batch --account-name ${{ env.STORAGE_NAME }} --auth-mode key -d '$web' -s dist/
Detailed logs from GitHub Actions:
022-03-01T14:17:06.8368067Z ##[group]Run azure/login@v1
2022-03-01T14:17:06.8368444Z with:
2022-03-01T14:17:06.8370124Z creds: ***
2022-03-01T14:17:06.8370507Z enable-AzPSSession: false
2022-03-01T14:17:06.8370913Z environment: azurecloud
2022-03-01T14:17:06.8371321Z allow-no-subscriptions: false
2022-03-01T14:17:06.8371762Z audience: api://AzureADTokenExchange
2022-03-01T14:17:06.8372170Z env:
2022-03-01T14:17:06.8373821Z AZURE_CREDENTIALS: ***
2022-03-01T14:17:06.8374471Z STORAGE_KEY: ***
2022-03-01T14:17:06.8374823Z ##[endgroup]
2022-03-01T14:17:15.8274876Z [command]/usr/bin/az cloud set -n azurecloud
2022-03-01T14:17:21.4767300Z Done setting cloud: "azurecloud"
2022-03-01T14:17:23.0710370Z Login successful.
2022-03-01T14:17:23.0894436Z ##[group]Run azure/CLI@v1
2022-03-01T14:17:23.0894808Z with:
2022-03-01T14:17:23.0895130Z azcliversion: latest
2022-03-01T14:17:23.0895667Z inlineScript: az storage blob upload-batch --account-name sadatastoresearchdev -d '$web' -s dist/
2022-03-01T14:17:23.0896146Z env:
2022-03-01T14:17:23.0897712Z AZURE_CREDENTIALS: ***
2022-03-01T14:17:23.0898317Z STORAGE_KEY: ***
2022-03-01T14:17:23.0898675Z AZURE_HTTP_USER_AGENT:
2022-03-01T14:17:23.0899158Z AZUREPS_HOST_ENVIRONMENT:
2022-03-01T14:17:23.0899547Z STORAGE_NAME: sadatastoresearchdev
2022-03-01T14:17:23.0899947Z ##[endgroup]
2022-03-01T14:17:23.1710590Z Starting script execution via docker image mcr.microsoft.com/azure-cli:latest
2022-03-01T14:17:45.3477491Z WARNING:
2022-03-01T14:17:45.3478765Z There are no credentials provided in your command and environment, we will query for account key for your storage account.
2022-03-01T14:17:45.3583025Z It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.
2022-03-01T14:17:45.3583675Z
2022-03-01T14:17:45.3598136Z You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
2022-03-01T14:17:45.3599424Z For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.
2022-03-01T14:17:45.3599847Z
2022-03-01T14:17:45.3600793Z In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
2022-03-01T14:17:46.6529259Z ERROR: The specified blob already exists.
2022-03-01T14:17:46.6530651Z RequestId:5a838987-601e-002b-6577-2d31d6000000
2022-03-01T14:17:46.6531594Z Time:2022-03-01T14:17:46.5388178Z
2022-03-01T14:17:46.6532370Z ErrorCode:BlobAlreadyExists
2022-03-01T14:17:46.6533513Z If you want to overwrite the existing one, please add --overwrite in your command.
2022-03-01T14:17:46.9476322Z ##[error]Error: az cli script failed.
2022-03-01T14:17:46.9487839Z cleaning up container...
2022-03-01T14:17:46.9851133Z MICROSOFT_AZURE_CLI_1646144243123_CONTAINER
2022-03-01T14:17:46.9851724Z
2022-03-01T14:17:46.9910045Z (node:1739) UnhandledPromiseRejectionWarning: Error: az cli script failed.
2022-03-01T14:17:46.9913655Z at /home/runner/work/_actions/azure/CLI/v1/dist/index.js:1:23964
2022-03-01T14:17:46.9914390Z at Generator.throw (<anonymous>)
2022-03-01T14:17:46.9915105Z at rejected (/home/runner/work/_actions/azure/CLI/v1/dist/index.js:1:20436)
2022-03-01T14:17:46.9915949Z at processTicksAndRejections (internal/process/task_queues.js:93:5)
2022-03-01T14:17:46.9917419Z (node:1739) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
2022-03-01T14:17:46.9919329Z (node:1739) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
2022-03-01T14:17:46.9996553Z Post job cleanup.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 12
- Comments: 16 (5 by maintainers)
Commits related to this issue
- Revert "add --overwrite flag" This is an upstream bug with the Azure CLI - see https://github.com/Azure/azure-cli/issues/21477 This reverts commit 44a7e794d7bc1146ab165f648fa6ccca18f8abae. — committed to fishworks/gofish by bacongobbler 2 years ago
- Fix production deploy to azure Ref: https://github.com/Azure/azure-cli/issues/21477 — committed to adobe/react-spectrum by devongovett 2 years ago
- Fix production deploy to azure (#2906) Ref: https://github.com/Azure/azure-cli/issues/21477 — committed to adobe/react-spectrum by devongovett 2 years ago
- specifiy --overwrite to overwrite existing blobs - az cli v2.34.0 introduced breaking change to prevent silent overwrites by default - https://github.com/Azure/azure-cli/issues/21477 Authored-by:... — committed to pivotal-cf/om by markstokan 2 years ago
- ci: add overwrite flag to az storage upload The storage command of azure cli needs an overwrite flag set to true since v2.34.0 see https://github.com/Azure/azure-cli/issues/21477 — committed to gravitee-io/gravitee-api-management by a-cordier 2 years ago
- ci: add overwrite flag to az storage upload The storage command of azure cli needs an overwrite flag set to true since v2.34.0 see https://github.com/Azure/azure-cli/issues/21477 — committed to gravitee-io/gravitee-api-management by a-cordier 2 years ago
- ci: add overwrite flag to az storage upload The storage command of azure cli needs an overwrite flag set to true since v2.34.0 see https://github.com/Azure/azure-cli/issues/21477 — committed to gravitee-io/gravitee-docs by phiz71 2 years ago
- ci: add overwrite flag to az storage upload (#715) The storage command of azure cli needs an overwrite flag set to true since v2.34.0 see https://github.com/Azure/azure-cli/issues/21477 — committed to gravitee-io/gravitee-docs by phiz71 2 years ago
- Add overwrite to Az CLI upload-batch There was a breaking change in a patch release: https://github.com/Azure/azure-cli/issues/21477. — committed to gabrielweyer/cors by gabrielweyer 2 years ago
- Add overwrite to Az CLI upload-batch There was a breaking change in a patch release: https://github.com/Azure/azure-cli/issues/21477. — committed to gabrielweyer/cors by gabrielweyer 2 years ago
- Add --overwrite flag as per breaking changes in az https://github.com/Azure/azure-cli/issues/21477 — committed to avinor/tau by BassOfLion 2 years ago
Yet more breaking changes marked with ‘patch’ numbers. Is it so hard to follow SemVer and mark it as new release and not cause customers workloads to fail which have been working for YEARS!
You need to do
--overwrite true
I’m curious why
true
is a required argument for the--overwrite
flag. Isn’t the presence of the--overwrite
flag enough to assert that the value is true, or is there an implied third value the CLI can provide?In other words…
Most CLI parsers work in this manner, so I’m curious why azure-cli chose to approach this differently.
Hotfix is getting merged today. You can specify --overwrite to overwrite existing blobs. This is a breaking change as we believe overwriting silently is not a good default. Thanks.
Can confirm - this is a recent change introduced in 2.34.0, breaking several of my pipelines using azure-blob-storage-upload which uses
az storage blob upload-batch
under the covers.In addition to the use case @nosvalds provided earlier, I use this action to upload nightly releases of my program to Azure. With this recent change, I have no way to work around this short of deleting the object before uploading the new asset.
Reverting to 2.33.1 resolves the issue for now.
The hotfix is on the way, both
--overwrite
and--overwrite true
will be accepted after hotfix“
latest
”, insofar as the azure-cli github action is concerned, is still 2.33.1 as of this post. It was 2.34.1 for at least a few hours last week, but was apparently reverted. That is, a github action such as the one in the OP shows something like this in the console:I have pinned the older 2.33.1 version in my workflows because the update to 2.34 breaks them in several different ways (newly requiring the
--overwrite
flag as in this issue, and the new content-type issue described here). This is instead of keeping the azcliversion argument undefined as suggested here.