restic: restic fatal error: runtime: out of memory

This was from trying to run a first time backup from my laptop via sftp to a backup repository which is 5 TiB in size (from backing up a server & workstation).

Output of `restic version`

restic 0.1.0 (v0.1.0-548-g795e3d5) compiled at 2016-05-24 22:26:48 with go1.5.4

Actual behavior

fatal error: runtime: out of memory

runtime stack: runtime.throw(0xa37550, 0x16) /usr/lib/golang/src/runtime/panic.go:527 +0x90 runtime.sysMap(0xc98a740000, 0x5a000000, 0x118f00, 0xc94d58) /usr/lib/golang/src/runtime/mem_linux.go:203 +0x9b runtime.mHeap_SysAlloc(0xc75b00, 0x5a000000, 0xc820001c80) /usr/lib/golang/src/runtime/malloc.go:426 +0x160 runtime.mHeap_Grow(0xc75b00, 0x2d000, 0x0) /usr/lib/golang/src/runtime/mheap.go:628 +0x63 runtime.mHeap_AllocSpanLocked(0xc75b00, 0x2d000, 0x100) /usr/lib/golang/src/runtime/mheap.go:532 +0x5f1 runtime.mHeap_Alloc_m(0xc75b00, 0x2d000, 0x100000000, 0xc800000001) /usr/lib/golang/src/runtime/mheap.go:425 +0x1ac runtime.mHeap_Alloc.func1() /usr/lib/golang/src/runtime/mheap.go:484 +0x41 runtime.systemstack(0x7fc0f294fe28) /usr/lib/golang/src/runtime/asm_amd64.s:278 +0xab runtime.mHeap_Alloc(0xc75b00, 0x2d000, 0x10100000000, 0x4357af) /usr/lib/golang/src/runtime/mheap.go:485 +0x63 runtime.largeAlloc(0x5a000000, 0x7fc000000001, 0xc80000001d) /usr/lib/golang/src/runtime/malloc.go:748 +0xb3 runtime.mallocgc.func3() /usr/lib/golang/src/runtime/malloc.go:637 +0x33 runtime.systemstack(0xc820020a00) /usr/lib/golang/src/runtime/asm_amd64.s:262 +0x79 runtime.mstart() /usr/lib/golang/src/runtime/proc1.go:668

goroutine 82 [running]: runtime.systemstack_switch() /usr/lib/golang/src/runtime/asm_amd64.s:216 fp=0xc824197608 sp=0xc824197600 runtime.mallocgc(0x5a000000, 0x94a5c0, 0x7fc000000001, 0x429622) /usr/lib/golang/src/runtime/malloc.go:638 +0x9c4 fp=0xc8241976d8 sp=0xc824197608 runtime.newarray(0x94a5c0, 0x200000, 0x4bd4693f) /usr/lib/golang/src/runtime/malloc.go:780 +0xc9 fp=0xc824197718 sp=0xc8241976d8 runtime.hashGrow(0x84f740, 0xc822d9a000) /usr/lib/golang/src/runtime/hashmap.go:761 +0x82 fp=0xc824197748 sp=0xc824197718 runtime.mapassign1(0x84f740, 0xc822d9a000, 0xc824197900, 0xc8241978b8) /usr/lib/golang/src/runtime/hashmap.go:481 +0x55b fp=0xc8241977f0 sp=0xc824197748 restic/repository.DecodeIndex(0x7fc0f2951530, 0xc822d9a030, 0xc8200fe070, 0x0, 0x0) /tmp/restic-build-781614583/src/restic/repository/index.go:519 +0x5ba fp=0xc824197a68 sp=0xc8241977f0 restic/repository.LoadIndexWithDecoder(0xc8200942a0, 0xe579cf34c883a906, 0x375522f06072b6de, 0x7a12e8d1a4c630ce, 0x14e7bc67aaec6f38, 0xac7120, 0x40, 0x0, 0x0) /tmp/restic-build-781614583/src/restic/repository/index.go:568 +0x2d2 fp=0xc824197bb0 sp=0xc824197a68 restic/repository.LoadIndex(0xc8200942a0, 0xe579cf34c883a906, 0x375522f06072b6de, 0x7a12e8d1a4c630ce, 0x14e7bc67aaec6f38, 0x375522f06072b6de, 0x0, 0x0) /tmp/restic-build-781614583/src/restic/repository/repository.go:380 +0x60 fp=0xc824197c78 sp=0xc824197bb0 restic/repository.(*Repository).LoadIndex.func1(0xe579cf34c883a906, 0x375522f06072b6de, 0x7a12e8d1a4c630ce, 0x14e7bc67aaec6f38, 0xc820094c00, 0x0, 0x0) /tmp/restic-build-781614583/src/restic/repository/repository.go:348 +0x59 fp=0xc824197d78 sp=0xc824197c78 restic/repository.ParallelWorkFuncParseID.func1(0xc820099080, 0x40, 0xc820094c00, 0x0, 0x0) /tmp/restic-build-781614583/src/restic/repository/parallel.go:89 +0x269 fp=0xc824197e60 sp=0xc824197d78 restic/repository.FilesInParallel.func1(0xc8200f4770, 0xc820094c60, 0xc8200f4760, 0xc820094c00, 0xc820094cc0) /tmp/restic-build-781614583/src/restic/repository/parallel.go:53 +0x114 fp=0xc824197f68 sp=0xc824197e60 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc824197f70 sp=0xc824197f68 created by restic/repository.FilesInParallel /tmp/restic-build-781614583/src/restic/repository/parallel.go:63 +0x185

goroutine 1 [chan receive, 4 minutes]: restic/repository.(_Repository).LoadIndex(0xc8200942a0, 0x0, 0x0) /tmp/restic-build-781614583/src/restic/repository/repository.go:367 +0x17a main.CmdBackup.Execute(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7ffe97eed4e9, 0x1d, 0x0, 0xa943e2, …) /tmp/restic-build-781614583/src/cmds/restic/cmd_backup.go:302 +0x59c main.(_CmdBackup).Execute(0xc82001c240, 0xc820013dd0, 0x1, 0x3, 0x0, 0x0) <autogenerated>:14 +0xe0 github.com/jessevdk/go-flags.(_Parser).ParseArgs(0xc820078870, 0xc82000a250, 0x3, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0) /tmp/restic-build-781614583/src/github.com/jessevdk/go-flags/parser.go:278 +0x8e3 github.com/jessevdk/go-flags.(_Parser).Parse(0xc820078870, 0x0, 0x0, 0x0, 0x0, 0x0) /tmp/restic-build-781614583/src/github.com/jessevdk/go-flags/parser.go:154 +0x9b main.main() /tmp/restic-build-781614583/src/cmds/restic/main.go:26 +0x1d1

goroutine 17 [syscall, 4 minutes, locked to thread]: runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1721 +0x1

goroutine 5 [syscall, 4 minutes]: os/signal.loop() /usr/lib/golang/src/os/signal/signal_unix.go:22 +0x18 created by os/signal.init.1 /usr/lib/golang/src/os/signal/signal_unix.go:28 +0x37

goroutine 6 [chan receive, 4 minutes]: restic.init.1.func1.1() /tmp/restic-build-781614583/src/restic/lock.go:262 +0x16f created by restic.init.1.func1 /tmp/restic-build-781614583/src/restic/lock.go:265 +0x2b

goroutine 18 [select, 4 minutes, locked to thread]: runtime.gopark(0xac7450, 0xc820026728, 0x9d4da8, 0x6, 0x44ed18, 0x2) /usr/lib/golang/src/runtime/proc.go:185 +0x163 runtime.selectgoImpl(0xc820026728, 0x0, 0x18) /usr/lib/golang/src/runtime/select.go:392 +0xa64 runtime.selectgo(0xc820026728) /usr/lib/golang/src/runtime/select.go:212 +0x12 runtime.ensureSigM.func1() /usr/lib/golang/src/runtime/signal1_unix.go:227 +0x353 runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1721 +0x1

goroutine 7 [chan receive, 4 minutes]: main.CleanupHandler(0xc82001c1e0) /tmp/restic-build-781614583/src/cmds/restic/cleanup.go:58 +0x7b created by main.init.1 /tmp/restic-build-781614583/src/cmds/restic/cleanup.go:25 +0x12a

goroutine 8 [syscall, 1 minutes]: syscall.Syscall(0x0, 0x5, 0xc823bbbca0, 0x4, 0x18, 0x18, 0x0) /usr/lib/golang/src/syscall/asm_linux_amd64.s:18 +0x5 syscall.read(0x5, 0xc823bbbca0, 0x4, 0x4, 0x18, 0x0, 0x0) /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:783 +0x5f syscall.Read(0x5, 0xc823bbbca0, 0x4, 0x4, 0x18, 0x0, 0x0) /usr/lib/golang/src/syscall/syscall_unix.go:160 +0x4d os.(_File).read(0xc82002e240, 0xc823bbbca0, 0x4, 0x4, 0x18, 0x0, 0x0) /usr/lib/golang/src/os/file_unix.go:211 +0x53 os.(_File).Read(0xc82002e240, 0xc823bbbca0, 0x4, 0x4, 0x18, 0x0, 0x0) /usr/lib/golang/src/os/file.go:95 +0x8a io.ReadAtLeast(0x7fc0f4999568, 0xc82002e240, 0xc823bbbca0, 0x4, 0x4, 0x4, 0x0, 0x0, 0x0) /usr/lib/golang/src/io/io.go:298 +0xe6 io.ReadFull(0x7fc0f4999568, 0xc82002e240, 0xc823bbbca0, 0x4, 0x4, 0x18, 0x0, 0x0) /usr/lib/golang/src/io/io.go:316 +0x62 github.com/pkg/sftp.recvPacket(0x7fc0f4999568, 0xc82002e240, 0xc82003af70, 0x0, 0x0, 0x0, 0x0, 0x0) /tmp/restic-build-781614583/src/github.com/pkg/sftp/packet.go:143 +0xde github.com/pkg/sftp.(*Client).recv(0xc820078960) /tmp/restic-build-781614583/src/github.com/pkg/sftp/client.go:160 +0x7f created by github.com/pkg/sftp.NewClientPipe /tmp/restic-build-781614583/src/github.com/pkg/sftp/client.go:77 +0x2d6

goroutine 11 [select, 4 minutes]: main.refreshLocks(0xc6d760, 0xc820094ae0) /tmp/restic-build-781614583/src/cmds/restic/lock.go:69 +0x434 created by main.lockRepository /tmp/restic-build-781614583/src/cmds/restic/lock.go:46 +0x196

goroutine 12 [semacquire, 4 minutes]: sync.runtime_Semacquire(0xc8200f477c) /usr/lib/golang/src/runtime/sema.go:43 +0x26 sync.(_WaitGroup).Wait(0xc8200f4770) /usr/lib/golang/src/sync/waitgroup.go:126 +0xb4 restic/repository.FilesInParallel(0x7fc0f4999f20, 0xc82012c000, 0x9d37b0, 0x5, 0x14, 0xc8200f4760, 0x0, 0x0) /tmp/restic-build-781614583/src/restic/repository/parallel.go:66 +0x1ad restic/repository.(_Repository).LoadIndex.func2(0xc820094ba0, 0xc820094b40, 0xc8200942a0, 0xc820142760) /tmp/restic-build-781614583/src/restic/repository/repository.go:364 +0xc4 created by restic/repository.(*Repository).LoadIndex /tmp/restic-build-781614583/src/restic/repository/repository.go:365 +0x143

Steps to reproduce the behavior

Simply try to run the first time backup.

I believe the problem is that the memory requirements for reading the index files is too large. I recently did a rebuild index on another machine. It was super memory intensive and required the machine to have 8GiB + swap to complete the job.

On the backup drive:

du -sh index

939M index

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 29 (15 by maintainers)

Most upvoted comments

I’ll chime in and add a bit of background information. Restic is still a young project, so there are several rough edges that we’re aware of. Memory usage is one of them.

Restic mainly uses too much memory in two areas:

Index handling: At the start, restic reads all the index files from a repository to learn what data is already stored there (and where). This is then kept in memory. Since these index files can be quite large, and restic processes them in parallel, it’ll use much memory in the beginning for loading and decrypting the index files. Later, only the containing information is stored for further use, but even that’s sometimes too big (depending on the amount of data stored in the repo).
Encrypting/decrypting blobs during backup: Restic at the moment uses way too much concurrency during backup for small boxes. I’m currently reworking the archiver code allowing this to be tweaked, and I also plan to add a check for how much concurrency makes sense for the current machine.

So, maybe this helps to understand a bit better what our challenges (as a project) are at the moment. Restic isn’t suited for machines without much memory (yet).

fd0 on Nov 9, 2017

@fd0 running in debug mode now. It will take some time to run and fail. I’ll gather the full output when it fails + the restic.pprof file.

@cfcs the task is already running, will check it after this run.

johansmitsnl on Feb 21, 2017