restic: Panic during SaveFile

Output of restic version

restic 0.6.1
compiled with go1.8.3 on linux/amd64

How did you start restic exactly? (Include the complete command line)

#!/bin/bash
set -e
set -u
set -o pipefail
source ~/.restic.auth
source ~/.restic.conf
restic list --no-lock locks 2>/dev/null || restic init
restic backup / --exclude-file $HOME/.restic.excluded
restic forget --prune --host $(hostname --fqdn) --keep-last 10 --keep-daily 183 --keep-tag persist

I’m 99.9% sure this is during the backup step.

What backend/server/service did you use?

S3 (Frankfurt)

Expected behavior

  • Attempt one or more retry (possibly with some sensible exponential backoff timing)
  • If retry fails, bail out cleanly and release the lock

Actual behavior

Yesterday, on a DigitalOcean VM: ContentLength=5064749 with Body length 0

panic: client.PutObject: Put https://backup-restic-7f89ac695d.s3-eu-central-1.amazonaws.com/hosts/com/adhoc-gti/gitlab/data/6a106c93fffe06d182f77a754bb87541bc5b6cfef526a6645db0d6e55ce3c4cd: http: ContentLength=5064749 with Body length 0

goroutine 170142 [running]:
restic/archiver.(*Archiver).saveChunk(0xc420323bd0, 0x9443887b5, 0x1e20d6, 0x7b72107f00000, 0xc431214000, 0x1e20d6, 0x1ee000, 0xc4200aa2c0, 0xb76400, 0xc4217b4ae8, ...)
	src/restic/archiver/archiver.go:163 +0x270
created by restic/archiver.(*Archiver).SaveFile
	src/restic/archiver/archiver.go:237 +0x381

Two days ago, on another DigitalOcean VM: ContentLength=5842361 with Body length 2795554

panic: client.PutObject: Put https://backup-restic-7f89ac695d.s3-eu-central-1.amazonaws.com/hosts/com/adhoc-gti/geminabox/data/3c148ee192368982b0f86a714cb2be5e96f7f49ffc49a536e531703330af52ff: http: ContentLength=5842361 with Body length 2795554

goroutine 7051 [running]:
restic/archiver.(*Archiver).saveChunk(0xc4202e78b0, 0x0, 0x1e79a1, 0x10556e57400000, 0xc42b45c000, 0x1e79a1, 0x3c0000, 0xc42007c370, 0xb78400, 0xc42000c2c0, ...)
    /tmp/restic-build-252735916/src/restic/archiver/archiver.go:163 +0x270
created by restic/archiver.(*Archiver).SaveFile
    /tmp/restic-build-252735916/src/restic/archiver/archiver.go:237 +0x381

Steps to reproduce the behavior

During a backup.

Do you have any idea what may have caused this?

No idea, maybe temporary a network problem, but S3 bucket and DO machines are both hosted in Frankfurt and have close to ideal link with very low latency and high throughput.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 1
  • Comments: 30 (16 by maintainers)

Most upvoted comments

It’s been ages since the last time, 0.7.0 really did help a lot! But I do have one new occurence today:

panic: client.PutObject: Put https://backup-restic-7f89ac695d.s3-eu-central-1.amazonaws.com/hosts/com/courtier-en-ligne/acs/data/5da83b5cd123fad9025d1d5a1a39648e76f919d82b5e7129f46ff8ff0a1152bf: write tcp 46.101.246.177:43935->52.219.74.5:443: write: connection reset by peer

goroutine 19929 [running]:
restic/archiver.(*Archiver).saveChunk(0xc42d2feaf0, 0xc1d4e0, 0xc4200123d8, 0x0, 0x3648e4, 0x2f50618f9f970, 0xc4368b8000, 0x3648e4, 0x3c4000, 0xc4200822c0, ...)
    src/restic/archiver/archiver.go:165 +0x395
created by restic/archiver.(*Archiver).SaveFile
    src/restic/archiver/archiver.go:239 +0x3a4