rclone: 409 Error when RClone Sync --delete-before to Azure Blob Archive-Tier

What is the problem you are having with rclone?

I have files in Azure Blob storage that are stored in Archive-Tier. Azure Blob Archive-Tier files can be stored or deleted, have meta-data, and they CAN-NOT be read or modified. My local versions of these files have been updated and I now need to sync my local files to Azure. To do so, I essentially need to delete the Azure Blob version (in Archive-Tier, again: can not be read/modified outside of meta-data).

I’m using the RClone GUI but am setting the --delete-before flag on the sync.

When the sync occurs, I get a “RESPONSE Stats: 409 This operation is not permitted on an archived blob” error. The error code is many rows but those code do not provide any additional diagnostic information, just Azure, API, local machine, and etc basic information.

What is your rclone version (output from `rclone version`)

# rclone version
rclone v1.53.0
- os/arch: linux/amd64
- go version: go1.15

Using the GUI option

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Running in Docker/Kubernetes, linux/amd64

Which cloud storage system are you using? (e.g. Google Drive)

Azure Blob Storage

The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)

rclone sync --delete-before --size-only --verbose --no-update-modtime --transfers 4 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --delete-before --stats 1s --stats-file-name-length 0 --fast-list /directory:remote

Please forgive small formatting errors… This was manually copied from a VNC window

A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)

Deleted/modified the directory, remote name, and the GUIDs for privacy reasons

2020/11/30 01:21:25 ERROR : **MY-FILE**: Failed to copy: multipart upload failed to upload part: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=BlobArchived) =====
Description=This operation is not permitted on an archived blob.
RequestId:c97536a8-e01e-000b-50b7-c6f95f0000
Time:2020-11-30T01:21:25.4901576Z, Details:
   Code: BlobArchived
   PUT https://kubeomv.blob.core.windows.net/**MyRemote**?blockid=AQAAAAAAAAA%3D&comp=block&timeout=31536001
   Authorization: REDACTED
   Content-Length: [419434]
   Content-Md5: [km02rBUA9Ac7ArHhzRuw==]
   User-Agent: [rclone/v1.53.0]
   X-Ms-Client-Request-Id: [9409ff8f-f83f-4fae-53cc-556e67c74852]
   X-Ms-Date: [Mon, 30 Nov 2020 01:21:25 GMT]
   X-Ms-Version: [2019-02-02]
   --------------------------------------------------------------------------------
   RESPONSE Status: 409 This operation is not permitted on an archived blob.
   Content-Length: [233]
   Content-Type: [application/xml]
   Date: [Mon, 30 Nov 2020 01:21:24 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [9409ff8f-f83f-4fae-53cc-556e674852]
   X-Ms-Error-Code: [BlobArchived]
   X-Ms-Request-Id: [c97536a8-e01e-00-50b7-c6f95f000000]
   X-Ms-Version: [2019-02-02]

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 25 (11 by maintainers)

Commits related to this issue

azureblob: remove archive tier blobs before updating - fixes #4819 Before this change, attempting to update an archive tier blob failed with a 409 error message 409 This operation is not permitt... — committed to rclone/rclone by ncw 4 years ago
azureblob: remove archive tier blobs before updating - fixes #4819 Before this change, attempting to update an archive tier blob failed with a 409 error message 409 This operation is not permitt... — committed to rclone/rclone by ncw 4 years ago
azureblob: delete archive tier blobs before update if --azure-blob-archive-tier-delete Before this change, attempting to update an archive tier blob failed with a 409 error message: 409 This ope... — committed to rclone/rclone by ncw 4 years ago
azureblob: delete archive tier blobs before update if --azure-blob-archive-tier-delete Before this change, attempting to update an archive tier blob failed with a 409 error message: 409 This ope... — committed to rclone/rclone by ncw 4 years ago
azureblob: delete archive tier blobs before update if --azureblob-archive-tier-delete Before this change, attempting to update an archive tier blob failed with a 409 error message: 409 This oper... — committed to rclone/rclone by ncw 4 years ago
azureblob: delete archive tier blobs before update if --azureblob-archive-tier-delete Before this change, attempting to update an archive tier blob failed with a 409 error message: 409 This oper... — committed to negative0/rclone by ncw 4 years ago

Most upvoted comments

ivandeex on Dec 1, 2020

I wouldn’t necessarily use the word ‘update’. ‘Update’ involves updating the data which is not possible with a blob that is in Archive-Tier.

While I think it would be fun to think of this as a general flag for Azure Blobs that replaces the functionality of the RCLONE --delete-before with the DELETE from Azure CLI (instead of POST or whatever RClone uses naitively) so that RClone would delete the old data before beginning a copy/sync, we can scope this as something Azure Blob Achieve-Tier specific.

I’d recommend something laser specific for this:

--azure-blob-archive-tier-delete

For true, your idea is spot on. I’d also think about having it over-write any of the --delete-before, --delete-during, or --delete-after flags (or anything else that may propagate a conflict on deleting). No need for the CLI to pop a warning on billing cost, that kind of warning can be added to the RClone documentation.

For false, the error message idea is spot on… It’s be pretty bling if RCLONE could check the status of the blob (is archive?), and if it is achieve, pop the error and skip the file, else upload. It would be cool if the error provided a few words to reference the documentation but we don’t need the error to propagate a full essay.

Love it… Your ideas/fix are awesome

GoingOffRoading on Dec 1, 2020