restic: crash on unlock
Output of restic version
restic 0.12.1 (v0.12.1-41-g10b39d75) compiled with go1.15.9 on linux/amd64
How did you run restic exactly?
export B2_ACCOUNT_ID=
export B2_ACCOUNT_KEY=
export RESTIC_PASSWORD=
export RESTIC_REPOSITORY=b2:bucket:repo
export RESTIC_CACHE_DIR=/a/dir/restic_cache
restic unlock
What backend/server/service did you use to store the repository?
B2
Expected behavior
# restic unlock
repository 7fcde7e2 opened successfully, password is correct
successfully removed locks
Actual behavior
# restic unlock
repository 7fcde7e2 opened successfully, password is correct
Remove(<lock/f288a00798>) returned error, retrying after 440.328595ms: Delete: b2_delete_file_version: 400: File not present: tm/locks/f288a0079883217b22d3005fb3cb4b250b7cecef50ddb7472b22083698d0f4ba 4_z56b124abe83975857ab70118_f11658776f00f68f9_d20211007_m005928_c000_v0001401_t0055
Remove(<lock/f288a00798>) returned error, retrying after 587.275613ms: Delete: b2_download_file_by_name: 404: File with such name does not exist.
Remove(<lock/f288a00798>) returned error, retrying after 892.239589ms: Delete: b2_download_file_by_name: 404: File with such name does not exist.
Remove(<lock/f288a00798>) returned error, retrying after 1.989705391s: Delete: b2_download_file_by_name: 404: File with such name does not exist.
Remove(<lock/f288a00798>) returned error, retrying after 1.818837414s: Delete: b2_download_file_by_name: 404: File with such name does not exist.
Remove(<lock/f288a00798>) returned error, retrying after 2.669912672s: Delete: b2_download_file_by_name: 404: File with such name does not exist.
Remove(<lock/f288a00798>) returned error, retrying after 4.902931741s: Delete: b2_download_file_by_name: 404: File with such name does not exist.
Remove(<lock/f288a00798>) returned error, retrying after 9.146728339s: Delete: b2_download_file_by_name: 404: File with such name does not exist.
Remove(<lock/f288a00798>) returned error, retrying after 10.214177147s: Delete: b2_download_file_by_name: 404: File with such name does not exist.
Remove(<lock/f288a00798>) returned error, retrying after 26.244032286s: Delete: b2_download_file_by_name: 404: File with such name does not exist.
b2_download_file_by_name: 404: File with such name does not exist.
Delete
github.com/restic/restic/internal/backend/b2.(*b2Backend).Remove
github.com/restic/restic/internal/backend/b2/b2.go:268
github.com/restic/restic/internal/backend.(*RetryBackend).Remove.func1
github.com/restic/restic/internal/backend/backend_retry.go:108
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer
github.com/cenkalti/backoff/v4@v4.1.1/retry.go:55
github.com/cenkalti/backoff/v4.RetryNotify
github.com/cenkalti/backoff/v4@v4.1.1/retry.go:34
github.com/restic/restic/internal/backend.(*RetryBackend).retry
github.com/restic/restic/internal/backend/backend_retry.go:46
github.com/restic/restic/internal/backend.(*RetryBackend).Remove
github.com/restic/restic/internal/backend/backend_retry.go:107
github.com/restic/restic/internal/cache.(*Backend).Remove
github.com/restic/restic/internal/cache/backend.go:38
github.com/restic/restic/internal/restic.RemoveStaleLocks.func1
github.com/restic/restic/internal/restic/lock.go:282
github.com/restic/restic/internal/restic.ForAllLocks.func2
github.com/restic/restic/internal/restic/lock.go:336
golang.org/x/sync/errgroup.(*Group).Go.func1
golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
runtime.goexit
runtime/asm_amd64.s:1374
Steps to reproduce the behavior
Backup TBs of data to B2. Get lots of unlock errors during backup, try to clean up with unlock.
Do you have any idea what may have caused this?
Brittle B2 implementation.
Do you have an idea how to solve the issue?
Keep running unlock over and over.
Did restic help you today? Did it make you happy in any way?
Not today.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 19 (3 by maintainers)
It’s possible to blame both restic and B2 here: restic for not gracefully handling the failed delete requests and B2 for the increase of 500 errors. That said it’s essentially impossible to guarantee that all requests are successful or that the client (restic) will receive a reply in time. As we can’t ask B2 whether a previous delete request was successful, we can do the next best thing: accept it when restic learns that the file to remove does not exist. That would allow restic to successfully “delete” the file when retrying the request.
I’ve opened PR #3544 to address this.
I can also confirm that this problem has existed for about 2 weeks for me, and there has been no change on the restic configurations.
Here are some sample outputs from different commands.
restic 0.12.1 compiled with go1.16.6 on linux/arm64restic --verbose=1 check --read-data-subset=0.5%restic --verbose=3 rebuild-indexrestic --verbose=3 prune