VictoriaMetrics: vmbackup fails on non AWS implementations of S3

Describe the bug

Due to recent changes in the AWS SDK alternative implementations of the S3 API are no longer working with vmbackup 1.112.0 or later since they do not implement checksuming. So far we know that ceph and linode’s S3 implementations are affected.

To Reproduce

  1. run vmbackup with an s3 url pointed to ceph or linode
  2. receive the following output
cannot create backup: cannot upload part{path: "indexdb/182C61247B4E051A/parts.json", file_size: 590, offset: 0, size: 590} to S3{bucket: "backup-bucket-123", dir: "vmstorage-0/latest/"}: cannot upload data to "indexdb/182C61247B4E051A/parts.json" at S3{bucket: "backup-bucket-123", dir: "vmstorage-0/latest/"} (remote path "vmstorage-0/latest/indexdb/182C61247B4E051A/parts.json/000000000000024E_0000000000000000_000000000000024E"): operation error S3: PutObject, https response error StatusCode: 400, RequestID: tx000004f2728d16b3658d8-0067d81d11-abadb1fa-default, HostID: abadb1fa-default-default, api error XAmzContentSHA256Mismatch: UnknownError

Version

vmbackup 1.112.0 or later

Logs

No response

Screenshots

No response

Used command-line flags

No response

Additional information

To work around this issue the version of vmbackup should be downgraded to v1.111. For a more permanent fix the version of the AWS SDK should be pinned to version prior to the checksumming change being added.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 22 (9 by maintainers)

Commits related to this issue

Most upvoted comments

@wwojdanowski @raml7646cPYr6jOTeXEF Thank you for testing! The PR was included in the upstream and will be available in the next release.

Thanks for downgrading the aws sdk. I ran into this issue using vmbackup with Backblaze B2.

My bad, i’ve pulled the gh-8622-downgrade-aws-deps-0-g610bc64f17 and i’ve no erros.

`[root@clustervmtest sources]# ./sauvegarde_S3_horaire_new.sh 2025-04-04T09:11:45.190Z info VictoriaMetrics/lib/logger/flag.go:12 build version: vmbackup-20250403-091937-heads-gh-8622-downgrade-aws-deps-0-g610bc64f17 2025-04-04T09:11:45.191Z info VictoriaMetrics/lib/logger/flag.go:13 command-line flags

0.546 seconds; server-side deleted 24770903 bytes; server-side copied 0 bytes; uploaded 37478556 bytes 2025-04-04T09:11:46.168Z info VictoriaMetrics/lib/snapshot/snapshot.go:77 Deleting snapshot 20250404091145-1832BFF763B4ADF7 2025-04-04T09:11:46.186Z info VictoriaMetrics/lib/snapshot/snapshot.go:111 Snapshot 20250404091145-1832BFF763B4ADF7 deleted 2025-04-04T09:11:46.186Z info VictoriaMetrics/app/vmbackup/main.go:109 gracefully shutting down http server for metrics at “[:8420]” 2025-04-04T09:11:46.186Z info VictoriaMetrics/app/vmbackup/main.go:113 successfully shut down http server for metrics in 0.000 seconds`

`[root@clustervmtest sources]# ./sauvegarde_S3_journ_new.sh 2025-04-04T09:15:03.111Z info VictoriaMetrics/lib/logger/flag.go:12 build version: vmbackup-20250403-091937-heads-gh-8622-downgrade-aws-deps-0-g610bc64f17 2025-04-04T09:15:03.111Z info VictoriaMetrics/lib/logger/flag.go:13 command-line flags

2025-04-04T09:15:03.796Z info VictoriaMetrics/lib/backup/actions/backup.go:238 server-side copied 99 out of 99 parts from S3{bucket: “int-obsd-backup-victoriametrics”, dir: “clustervmtest/latest/”} to S3{bucket: “int-obsd-backup-victoriametrics”, dir: “clustervmtest/20250404/”} in 350.72076ms 2025-04-04T09:15:03.796Z info VictoriaMetrics/lib/backup/actions/copy.go:92 remote backup copy from S3{bucket: “int-obsd-backup-victoriametrics”, dir: “clustervmtest/latest/”} to S3{bucket: “int-obsd-backup-victoriametrics”, dir: “clustervmtest/20250404/”} is complete; backed up 3810437817 bytes in 0.636 seconds; server-side deleted 102171629 bytes; server-side copied 95651066 bytes 2025-04-04T09:15:03.817Z info VictoriaMetrics/app/vmbackup/main.go:109 gracefully shutting down http server for metrics at “[:8420]” 2025-04-04T09:15:03.817Z info VictoriaMetrics/app/vmbackup/main.go:113 successfully shut down http server for metrics in 0.000 seconds`

@zekker6 it worked for me with this image.

S3 Ceph with the following limitations: Paquet Max supported Version AWS CLI v1 V1.36.40 AWS CLI v2 V2.22.35 AWS SDK for Python V1.35.99 AWS SDK for GO Release 2025-01-14