velero: velero backup create fails to upload backup to s3 using aws plugin

Discussed in https://github.com/vmware-tanzu/velero/discussions/7542

<div type='discussions-op-text'>

Originally posted by Wayne-H-Ha March 19, 2024 We used to be able to create backup using velero 1.12.2 and aws plugin 1.8.2.

We tried velero 1.13.0 and plugin 1.9.0 and it failed so we switched back to older version.

We tried again with velero 1.13.1 and plugin 1.9.1 and it still fails. Any configuration change we need to make in order to use the new version?

We tried to find the backup in s3 and it didn’t get uploaded there.

When we describe the backup, it returns:

velero-v1.13.1-linux-amd64/velero describe backup cp-20240319163110 | tail
Started:    2024-03-19 16:32:01 +0000 UTC
Completed:  <n/a>
Expiration:  2024-04-18 16:32:01 +0000 UTC
Total items to be backed up:  2871
Items backed up:              2871
Backup Volumes:
  <error getting backup volume info: DownloadRequest.velero.io "cp-20240319163110-63fd6028-b8fb-4c35-97e2-7fbfc44f74f3" is invalid: spec.target.kind: Unsupported value: "BackupVolumeInfos": supported values: "BackupLog", "BackupContents", "BackupVolumeSnapshots", "BackupItemOperations", "BackupResourceList", "BackupResults", "RestoreLog", "RestoreResults", "RestoreResourceList", "RestoreItemOperations", "CSIBackupVolumeSnapshots", "CSIBackupVolumeSnapshotContents">

We believe the problem is a suffix “@aws” is added to key id? For example, aws_access_key_id = “3…0” but “3…0@aws” is passed to s3? Is there a configuration we can use to not having this suffix added?

cat /credentials/cloud
[default]
aws_access_key_id = "3..0"
aws_secret_access_key = "a..b"

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 57 (12 by maintainers)

Commits related to this issue

Most upvoted comments

For me applying the non document checksumAlgorithm flag fixes the issue

configuration:
  backupStorageLocation:
    - name: default
      provider: aws
      bucket: velero
      default: true
      config:
        region: de
        s3ForcePathStyle: "true"
        s3Url: ${minio_uri}
        checksumAlgorithm: ""

The s3 plugin is configured via the BackupStorageLocation. This field is included there. If you are unable to configure the BSL completely (including this parameter) via Replicated tools, then you may need to open an issue against Replicated. Since this is part of the BackupStorageLocation configuration, Velero takes configuration from there. If checksumAlgorithm isn’t set, then Velero uses a default. The issue is that the default doesn’t work in your environment, so you need to set this when you create the BSL – and if Replicated is creating the BSL, then the Replicated tooling needs to be able to set this if they claim to be compatible with the latest velero-plugin-for-aws releases.

@Wayne-H-Ha I also use IBM COS and my problem was fixed with empty checksum algorithm for Velero 1.14 and aws plugin 1.10

Here is how my BSL looks like

spec:
  config:
    checksumAlgorithm: ''
    profile: default
    region: <my-region>
    s3ForcePathStyle: 'true'
    s3Url: 'https://s3.direct.<my-region>.cloud-object-storage.appdomain.cloud'
  credential:
    key: cloud
    name: cloud-credentials
  default: true
  objectStorage:
    bucket: <my-bucket-name>
    prefix: velero
  provider: aws

Reading your config, only thing I spotted different was s3Url, should it be https://s3.direct.us-east-1.cloud-object-storage.appdomain.cloud?

To grab the URL I am going to use with Velero, I do the following

  • Go to my Cloud Object Storage (COS) instance page
  • Click on the Bucket I want to use
  • On the Bucket page, click on Configuration tab
  • Go to Endpoints section and select one of them (adding https:// prefix)

Do you do the same steps?

Did you spot any difference in my BSL compared to yours?

We tried the following combinations:

Velero vs aws plugin 1.12.2 vs 1.8.2 works 1.12.2 vs 1.9.1 fails 1.13.1 vs 1.8.2 works 1.13.1 vs 1.9.1 fails

So we suspect aws plugin 1.9.1 is adding “@aws” to end of key id so velero fails to upload backup to IBM Cloud Object Storage?

Thanks for the link to the documentation. As I mentioned earlier, velero 1.12.2 and aws plugin 1.8.2 backup used to work for us. So not sure why it stopped working when we upgrade to 1.13.0 and 1.9.0 or 1.13.1 and 1.9.1? Here is the velero install command we use for many versions of velero including 1.11 and earlier versions:

      /tmp/velero-${RELEASE}-linux-amd64/velero install \
      --image "${REGISTRY_PATH}/velero:${VELERO_IMAGE_TAG}"  \
      --provider aws \
      --plugins ${REGISTRY_PATH}/velero-plugin-for-aws-amd64:${VELERO_PLUGIN_IMAGE_TAG} \
      --bucket ${COS_BUCKET}  \
      --prefix ${COS_PREFIX} \
      --secret-file /tmp/cos-credentials \
      --use-volume-snapshots=false \
      --backup-location-config region=us-east-1,s3ForcePathStyle="true",s3Url=${COS_ENDPOINT} \
      --velero-pod-cpu-request "700m" \
      --velero-pod-mem-request "$MEM_REQUEST" \
      --velero-pod-cpu-limit "700m" \
      --velero-pod-mem-limit "$MEM_LIMIT"