restic: panic: runtime error: invalid memory address or nil pointer dereference

Output of `restic version`

restic 0.8.1 compiled with go1.9.2 on linux/amd64

How did you run restic exactly?

RESTIC_PASSWORD=$RESTIC_PASSWORD B2_ACCOUNT_ID=$B2_ACCOUNT_ID B2_ACCOUNT_KEY=$B2_ACCOUNT_KEY /usr/bin/restic --cache-dir=$cache_dir -r b2:$b2_bucket:/ backup $bkp_dir -e $bkp_dir_exclude -o b2.connections=10

And on the dataset in which it panics, I get this output: panic: runtime error: invalid memory address or nil pointer dereference11649 items 0 errors ETA 41:46 [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x30]

goroutine 2380372 [running]: panic(0xa36200, 0xe71070) /usr/lib64/go/src/runtime/panic.go:540 +0x45e runtime.panicmem() /usr/lib64/go/src/runtime/panic.go:63 +0x5e runtime.sigpanic() /usr/lib64/go/src/runtime/signal_unix.go:367 +0x17c created by net.(*Resolver).goLookupIPCNAMEOrder /usr/lib64/go/src/net/dnsclient_unix.go:482 +0x21c password is correct unable to create lock in backend: repository is already locked by PID 12213 on nfs1 by root (UID 0, GID 0) lock was created at 2017-12-29 18:47:25 (35h59m1.245739467s ago) storage ID 2919f7d7

The execution trace (at least on 0.8.0) tends to be different and not always the same. The panic part is consistent.

I’ve ran it on VM’s on different hosts. I have high confidence a hardware issue isn’t involved. The dataset resides on a btrfs filesystem. I’ve ran it with different kernels (grsec+4.9 branch, and 4.14 branch). I get no errors logged by the kernel elsewhere. (It’s not going OOM or anything, I’ve increased VM memory to 16 GB). The OS running is gentoo amd64 hardened.

What backend/server/service did you use to store the repository?

B2 Storage Cloud

Expected behavior

I expected all the directories to backup successfully.

Actual behavior

Out of 4 directories, restic backups 3 which are < 200gb each, and always panics on one which is > 750gb in size.

Steps to reproduce the behavior

I’m not sure. It always fails for me on that data set. At least since 0.8.0. Possibly also used 0.7.3 but not sure.

Do you have any idea what may have caused this?

No.

Do you have an idea how to solve the issue?

Sorry, I don’t.

Did restic help you or made you happy in any way?

Not yet but I’m hoping 😃

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 21 (10 by maintainers)

Most upvoted comments

restic_0.8.1_go1.10_linux_amd64 finished the prune successfully on a 16GB RAM+12GB Swap VM.

Normal output:

password is correct
counting files in repo
building new index for repo
[3:17:38] 100.00%  49153 / 49153 packs
repository contains 49153 packs (2582677 blobs) with 226.142 GiB
processed 2582677 blobs: 1240254 duplicate blobs, 28.274 GiB duplicate
load all snapshots
find data that is still in use for 268 snapshots
[2:15:13] 100.00%  268 / 268 snapshots
found 1287180 of 2582677 data blobs still in use, removing 1295497 blobs
will remove 0 invalid files
will delete 937 packs and rewrite 9519 packs, this frees 36.041 GiB
[5:23:12] 100.00%  9519 / 9519 packs rewritten
counting files in repo
[2:56:25] 100.00%  41060 / 41060 packs
finding old index files
saved new index as 007084e8
remove 502 old index files
[2:16:34] 100.00%  10456 / 10456 packs deleted
done

I should note that it’s unclear if the Go version is what fixed it, or if there’s something different in your compiler options than Gentoo’s. (Although the Go version is the more likely of the two.)

Philippe23 on Feb 21, 2018

@rawtaz

At least for me, my VM’s are AWS EC2 instances located in different zones so they’re definitely on different machines. Seems like a stretch that we’re seeing this on 3 different machines with such similar setups.

Also, I haven’t had any other memory issues on the primary/original VM, so it seems odd that it only hits restic’s DNS requests and not any of the thousands of others that happen on the machine each day.

Restic is the only go application I have installed, so it seems to me that it’s far more likely a restic or go bug (or some interaction with Gentoo’s hardened kernel settings) than bad memory.

That said, I have no idea how to I’d run memtest on one of these EC2 VMs because it would require a different AMI to boot into a different image, which would most likely place me on different hardware.

Philippe23 on Feb 16, 2018