longhorn: [BUG] Backup - S3 Timeout
Describe the bug In the backup UI, I get an error and the backups are not listed.
error listing backups: error listing backup volumes: Timeout executing: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.0.0/longhorn [backup ls --volume-only s3://PATH/], output , stderr, , error <nil>
Number of Volumes: 39 Number of Volumes with backup enabled: 32
S3 Bucket Size: 2TB
Expected behavior Backups of Volumes are showed
Log
time="2020-11-05T12:55:15Z" level=warning msg="backup store monitor: failed to list backup volumes in s3://S3PATH/: error listing backup volumes: Timeout executing: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.0.0/longhorn [backup ls --volume-only s3://longhorn-production@ch-dk-2/], output , stderr, , error <nil>"
Environment:
- Longhorn version: 1.0.0
- Kubernetes version: 1.18.3
- Node OS type and version: Centos 7.7
Additional context On an empty bucket the backups are listed normally.
S3 Provider https://exoscale.com
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 40 (22 by maintainers)
@jenting Upgrade to 1.1.2 worked. Backups are available all the time now. Thank you! Keep up the good work!
@jenting unfortunately still the same after upgrade to 1.1.1
8 CPUs.
Btw, you can use @longhorn/maintainer to ping the maintainers.
real 0m0.739s user 0m0.528s sys 0m0.077s
From the AWS SDK, the default time is 30 seconds https://github.com/longhorn/backupstore/blob/18aa87a/vendor/github.com/aws/aws-sdk-go/aws/session/cabundle_transport.go#L17 From the issue reported, the longhorn version is
1.0.0which means that the list operation can’t be fulfilled with 30 seconds.At longhorn release
1.1.0, we add retry count to3https://github.com/longhorn/longhorn-engine/blob/v1.1.0/vendor/github.com/longhorn/backupstore/s3/s3_service.go#L28.@timmy59100 Have you tried to upgrade your longhorn cluster to version
1.1.0to see the error is still happening?