velero: Fetching logs does not work when using Quobyte s3 object storage

What steps did you take and what happened: When creating a backup with using OpenStack s3 object storage, backup and restore work just fine, but fetching the logs fails:

$ ark backup logs backup-test
An error occurred: request failed: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The resource you requested does not exist</Message><Resource>/metakube-cluster-backup-fhgbvx65xg-ark/backup-test/backup-test-logs.gz</Resource><RequestId>2925491272881701579</RequestId></Error>

$ ark restore logs backup-test-20180830163119
An error occurred: request failed: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><Message>The resource you requested does not exist</Message><Resource>/metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz</Resource><RequestId>1135366011835801288</RequestId></Error>

The files exist though and also have the correct valid content:

$ s3cmd la --recursive
2018-08-30 14:48         0   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/
2018-08-30 14:13      2446   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/backup-test-logs.gz
2018-08-30 14:13      5907   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/backup-test.tar.gz
2018-08-30 14:31       647   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz
2018-08-30 14:31        82   s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-results.gz

$ s3cmd get s3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz
download: 's3://metakube-cluster-backup-fhgbvx65xg-ark/backup-test/restore-backup-test-20180830163119-logs.gz' -> './restore-backup-test-20180830163119-logs.gz'  [1 of 1]
 647 of 647   100% in    0s   110.17 kB/s  done

$ gunzip restore-backup-test-20180830163119-logs.gz

$ cat restore-backup-test-20180830163119-logs
time="2018-08-30T14:31:21Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:124"
...

What did you expect to happen:

Logs are displayed correctly

The output of the following commands will help us better understand what’s going on: (Pasting long output into a GitHub gist or other pastebin is fine.)

  • kubectl logs deployment/ark -n heptio-ark No errors.
  • ark backup describe <backupname> or kubectl get backup/<backupname> -n heptio-ark -o yaml No errors.
  • ark backup logs <backupname> Doesn’t work, see above. No errors when manually downloading and viewing them.
  • ark restore describe <restorename> or kubectl get restore/<restorename> -n heptio-ark -o yaml No errors.
  • ark restore logs <restorename> Doesn’t work, see above. No errors when manually downloading and viewing them.

Environment:

  • Ark version (use ark version): 0.9.3
  • Kubernetes version (use kubectl version): 1.11.2 on server and client
  • Cloud provider or hardware configuration: OpenStack
  • OS (e.g. from /etc/os-release): Ubuntu Xenial

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 19 (19 by maintainers)

Commits related to this issue

Most upvoted comments

Ah, that’s unfortunate. I’ll talk with the team about whether or not we want to own a v1 signing algorithm in the Ark code base and get back to you.