velero: velero backup create fails to upload backup to s3 using aws plugin
Discussed in https://github.com/vmware-tanzu/velero/discussions/7542
<div type='discussions-op-text'>Originally posted by Wayne-H-Ha March 19, 2024 We used to be able to create backup using velero 1.12.2 and aws plugin 1.8.2.
We tried velero 1.13.0 and plugin 1.9.0 and it failed so we switched back to older version.
We tried again with velero 1.13.1 and plugin 1.9.1 and it still fails. Any configuration change we need to make in order to use the new version?
We tried to find the backup in s3 and it didn’t get uploaded there.
When we describe the backup, it returns:
velero-v1.13.1-linux-amd64/velero describe backup cp-20240319163110 | tail
Started: 2024-03-19 16:32:01 +0000 UTC
Completed: <n/a>
Expiration: 2024-04-18 16:32:01 +0000 UTC
Total items to be backed up: 2871
Items backed up: 2871
Backup Volumes:
<error getting backup volume info: DownloadRequest.velero.io "cp-20240319163110-63fd6028-b8fb-4c35-97e2-7fbfc44f74f3" is invalid: spec.target.kind: Unsupported value: "BackupVolumeInfos": supported values: "BackupLog", "BackupContents", "BackupVolumeSnapshots", "BackupItemOperations", "BackupResourceList", "BackupResults", "RestoreLog", "RestoreResults", "RestoreResourceList", "RestoreItemOperations", "CSIBackupVolumeSnapshots", "CSIBackupVolumeSnapshotContents">
We believe the problem is a suffix “@aws” is added to key id? For example, aws_access_key_id = “3…0” but “3…0@aws” is passed to s3? Is there a configuration we can use to not having this suffix added?
cat /credentials/cloud
[default]
aws_access_key_id = "3..0"
aws_secret_access_key = "a..b"
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 57 (12 by maintainers)
Commits related to this issue
- Update ibm-config.md As per https://github.com/vmware-tanzu/velero/issues/7543 setting checksumAlgorithm to avoid 403 errors Added plugins line as velero install failed without this option in vers... — committed to gjanders/velero by gjanders 2 years ago
- As per https://github.com/vmware-tanzu/velero/issues/7543 setting checksumAlgorithm to avoid 403 errors Added plugins line as velero install failed without this option in version 1.14.0 Removed the ... — committed to gjanders/velero by gjanders 2 years ago
- As per https://github.com/vmware-tanzu/velero/issues/7543 setting checksumAlgorithm to avoid 403 errors Added plugins line as velero install failed without this option in version 1.14.0 Removed the ... — committed to gjanders/velero by gjanders 2 years ago
- Updated IBM COS documentation Added option checksumAlgorith, this stops 403 errors as per https://github.com/vmware-tanzu/velero/issues/7543 Added plugins line as velero install failed without this o... — committed to gjanders/velero by gjanders 2 years ago
For me applying the non document
checksumAlgorithmflag fixes the issueThe s3 plugin is configured via the BackupStorageLocation. This field is included there. If you are unable to configure the BSL completely (including this parameter) via Replicated tools, then you may need to open an issue against Replicated. Since this is part of the BackupStorageLocation configuration, Velero takes configuration from there. If
checksumAlgorithmisn’t set, then Velero uses a default. The issue is that the default doesn’t work in your environment, so you need to set this when you create the BSL – and if Replicated is creating the BSL, then the Replicated tooling needs to be able to set this if they claim to be compatible with the latest velero-plugin-for-aws releases.@Wayne-H-Ha I also use IBM COS and my problem was fixed with empty checksum algorithm for Velero 1.14 and aws plugin 1.10
Here is how my BSL looks like
Reading your config, only thing I spotted different was s3Url, should it be
https://s3.direct.us-east-1.cloud-object-storage.appdomain.cloud?To grab the URL I am going to use with Velero, I do the following
https://prefix)Do you do the same steps?
Did you spot any difference in my BSL compared to yours?
We tried the following combinations:
Velero vs aws plugin 1.12.2 vs 1.8.2 works 1.12.2 vs 1.9.1 fails 1.13.1 vs 1.8.2 works 1.13.1 vs 1.9.1 fails
So we suspect aws plugin 1.9.1 is adding “@aws” to end of key id so velero fails to upload backup to IBM Cloud Object Storage?
Thanks for the link to the documentation. As I mentioned earlier, velero 1.12.2 and aws plugin 1.8.2 backup used to work for us. So not sure why it stopped working when we upgrade to 1.13.0 and 1.9.0 or 1.13.1 and 1.9.1? Here is the velero install command we use for many versions of velero including 1.11 and earlier versions: