kopia: Regression: kopia maintenance failed on v0.11.2 due to out of memory

kopia maintenance failing on v0.11.2 (compiled with go1.18.3 for i386 architecture) After switching back to v0.10.7 maintenance completed without any errors

Issue found on computer with 4Gb of RAM and according to top, kopia didn’t allocated more than 2Gb

panic log

# kopia maintenance run --full
2022-07-08T05:45:55.510220-04:00  Running full maintenance...
2022-07-08T05:45:55.527128-04:00  Looking for active contents...
runtime: out of memory: cannot allocate 67108864-byte block (2062876672 in use)
fatal error: out of memory

goroutine 164 [running]:
runtime.throw({0x8e9aaf4, 0xd})
        runtime/panic.go:992 +0x64 fp=0x43dafa8c sp=0x43dafa78 pc=0x80802a4
runtime.(*mcache).allocLarge(0x29c9e2e0, 0x3fffe00, 0x1)
        runtime/mcache.go:215 +0x1d3 fp=0x43dafab4 sp=0x43dafa8c pc=0x805c243
runtime.mallocgc(0x3fffe00, 0x8ce7760, 0x1)
        runtime/malloc.go:1096 +0x434 fp=0x43dafafc sp=0x43dafab4 pc=0x8053534
runtime.makeslice(0x8ce7760, 0x1fffe00, 0x3fffe00)
        runtime/slice.go:103 +0x49 fp=0x43dafb10 sp=0x43dafafc pc=0x8096ea9
encoding/json.(*Decoder).refill(0x4e7340b0)
        encoding/json/stream.go:159 +0xd9 fp=0x43dafb40 sp=0x43dafb10 pc=0x83de439
encoding/json.(*Decoder).readValue(0x4e7340b0)
        encoding/json/stream.go:140 +0x90 fp=0x43dafb68 sp=0x43dafb40 pc=0x83de0e0
encoding/json.(*Decoder).Decode(0x4e7340b0, {0x8c76a00, 0x4015e768})
        encoding/json/stream.go:63 +0x68 fp=0x43dafb90 sp=0x43dafb68 pc=0x83dde78
github.com/kopia/kopia/snapshot/snapshotfs.readDirEntries({0x5fff0cf8, 0x447f5540})
        github.com/kopia/kopia/snapshot/snapshotfs/dir_reader.go:19 +0x7e fp=0x43dafbb4 sp=0x43dafb90 pc=0x8720eae
github.com/kopia/kopia/snapshot/snapshotfs.(*repositoryDirectory).loadLocked(0x7313a240, {0x9525ddc, 0x3a49ab58})
        github.com/kopia/kopia/snapshot/snapshotfs/repofs.go:179 +0x12e fp=0x43dafc60 sp=0x43dafbb4 pc=0x872709e
github.com/kopia/kopia/snapshot/snapshotfs.(*repositoryDirectory).ensureDirEntriesLoaded(0x7313a240, {0x9525ddc, 0x3a49ab58})
        github.com/kopia/kopia/snapshot/snapshotfs/repofs.go:158 +0xb9 fp=0x43dafc8c sp=0x43dafc60 pc=0x8726dc9
github.com/kopia/kopia/snapshot/snapshotfs.(*repositoryDirectory).IterateEntries(0x7313a240, {0x9525ddc, 0x3a49ab58}, 0x7313a3c0)
        github.com/kopia/kopia/snapshot/snapshotfs/repofs.go:137 +0x2f fp=0x43dafcf4 sp=0x43dafc8c pc=0x87269ef
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x952a0ac, 0x7313a240}, {0x81315dd0, 0x30})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:115 +0x132 fp=0x43dafd34 sp=0x43dafcf4 pc=0x8729e72
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x5ff02f90, 0x7313a240}, {0x81315dd0, 0x30})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:103 +0x170 fp=0x43dafdc8 sp=0x43dafd34 pc=0x8729ce0
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry.func1({0x9525ddc, 0x3a49ab58}, {0x5ff02f90, 0x7313a240})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:131 +0x2d7 fp=0x43dafe24 sp=0x43dafdc8 pc=0x872a247
github.com/kopia/kopia/snapshot/snapshotfs.(*repositoryDirectory).IterateEntries(0x74bdecc0, {0x9525ddc, 0x3a49ab58}, 0x74bdef00)
        github.com/kopia/kopia/snapshot/snapshotfs/repofs.go:142 +0x306 fp=0x43dafe8c sp=0x43dafe24 pc=0x8726cc6
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x952a0ac, 0x74bdecc0}, {0x53c2db30, 0x2c})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:115 +0x132 fp=0x43dafecc sp=0x43dafe8c pc=0x8729e72
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x5ff02f90, 0x74bdecc0}, {0x53c2db30, 0x2c})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:103 +0x170 fp=0x43daff60 sp=0x43dafecc pc=0x8729ce0
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry.func1.1(0x3a269980, {0x0, 0x0})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:128 +0x44 fp=0x43daff80 sp=0x43daff60 pc=0x872a2c4
github.com/kopia/kopia/internal/workshare.NewPool.func1()
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:62 +0xb0 fp=0x43dafff0 sp=0x43daff80 pc=0x86fc880
runtime.goexit()
        runtime/asm_386.s:1326 +0x1 fp=0x43dafff4 sp=0x43dafff0 pc=0x80b0181
created by github.com/kopia/kopia/internal/workshare.NewPool
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:55 +0xe6

goroutine 1 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0x461fe738)
        runtime/sema.go:56 +0x30
sync.(*WaitGroup).Wait(0x461fe730)
        sync/waitgroup.go:136 +0x81
github.com/kopia/kopia/internal/workshare.(*AsyncGroup).Wait(0x68668b58)
        github.com/kopia/kopia/internal/workshare/workshare_waitgroup.go:76 +0x39
github.com/kopia/kopia/internal/workshare.(*AsyncGroup).Close(0x68668b58)
        github.com/kopia/kopia/internal/workshare/workshare_waitgroup.go:88 +0x2b
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x952a0ac, 0x3a268700}, {0x0, 0x0})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:141 +0x20f
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x5ff02f90, 0x3a268700}, {0x0, 0x0})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:103 +0x170
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).Process(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x5ff02f90, 0x3a268700}, {0x0, 0x0})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:153 +0x146
github.com/kopia/kopia/snapshot/snapshotgc.findInUseContentIDs({0x9525ddc, 0x3a49ab58}, {0x39fcb490, 0x3a2a6340}, 0x3a0c5470)
        github.com/kopia/kopia/snapshot/snapshotgc/gc.go:61 +0x2c1
github.com/kopia/kopia/snapshot/snapshotgc.runInternal({0x9525ddc, 0x3a49ab58}, {0x952ca38, 0x3a2a6340}, 0x1, {0x68c61714000, 0x4e94914f0000, 0xd18c2e28000, 0x1, 0x0, ...}, ...)
        github.com/kopia/kopia/snapshot/snapshotgc/gc.go:102 +0xd1
github.com/kopia/kopia/snapshot/snapshotgc.Run.func1()
        github.com/kopia/kopia/snapshot/snapshotgc/gc.go:74 +0x87
github.com/kopia/kopia/repo/maintenance.ReportRun({0x9525ddc, 0x3a49ab58}, {0x952ca38, 0x3a2a6340}, {0x8e98502, 0xb}, 0x0, 0x3a3df75c)
        github.com/kopia/kopia/repo/maintenance/maintenance_schedule.go:192 +0xc8
github.com/kopia/kopia/snapshot/snapshotgc.Run({0x9525ddc, 0x3a49ab58}, {0x952ca38, 0x3a2a6340}, 0x1, {0x68c61714000, 0x4e94914f0000, 0xd18c2e28000, 0x1, 0x0, ...}, ...)
        github.com/kopia/kopia/snapshot/snapshotgc/gc.go:73 +0xee
github.com/kopia/kopia/snapshot/snapshotmaintenance.Run.func1({0x9525ddc, 0x3a49ab58}, {{0x952ca38, 0x3a2a6340}, {0x8e8f906, 0x4}, 0x3a451ac0, {0x0, 0xeda59f3d3, 0x9d1aae0}})
        github.com/kopia/kopia/snapshot/snapshotmaintenance/snapshotmaintenance.go:21 +0x8b
github.com/kopia/kopia/repo/maintenance.RunExclusive({0x9525ddc, 0x3a49ab10}, {0x952ca38, 0x3a2a6340}, {0x8e8f906, 0x4}, 0x0, 0x3a3dfa78)
        github.com/kopia/kopia/repo/maintenance/maintenance_run.go:215 +0x8ea
github.com/kopia/kopia/snapshot/snapshotmaintenance.Run({0x9525ddc, 0x3a49ab10}, {0x952ca38, 0x3a2a6340}, {0x8e8f906, 0x4}, 0x0, {0x68c61714000, 0x4e94914f0000, 0xd18c2e28000, ...})
        github.com/kopia/kopia/snapshot/snapshotmaintenance/snapshotmaintenance.go:17 +0x95
github.com/kopia/kopia/cli.(*commandMaintenanceRun).run(0x3a45413c, {0x9525ddc, 0x3a49ab10}, {0x952ca38, 0x3a2a6340})
        github.com/kopia/kopia/cli/command_maintenance_run.go:35 +0xd7
github.com/kopia/kopia/cli.(*App).directRepositoryWriteAction.func1.1({0x9525ddc, 0x3a49ab10}, {0x952ca38, 0x3a2a6340})
        github.com/kopia/kopia/cli/app.go:386 +0x35
github.com/kopia/kopia/repo.DirectWriteSession({0x9525ddc, 0x3a0c4168}, {0x952c268, 0x3a2a61a0}, {{0x3a09e018, 0x13}, 0x0, 0x3a08c398}, 0x3a3dfc44)
        github.com/kopia/kopia/repo/repository.go:381 +0x1d1
github.com/kopia/kopia/cli.(*App).directRepositoryWriteAction.func1({0x9525ddc, 0x3a0c4168}, {0x952c268, 0x3a2a61a0})
        github.com/kopia/kopia/cli/app.go:383 +0x108
github.com/kopia/kopia/cli.assertDirectRepository.func1({0x9525ddc, 0x3a0c4168}, {0x39fcb490, 0x3a2a61a0})
        github.com/kopia/kopia/cli/app.go:376 +0x71
github.com/kopia/kopia/cli.(*App).maybeRepositoryAction.func1({0x9525ddc, 0x3a0c4168})
        github.com/kopia/kopia/cli/app.go:497 +0x144
github.com/kopia/kopia/cli.(*App).baseActionWithContext.func1.1.1()
        github.com/kopia/kopia/cli/app.go:480 +0xb7
github.com/kopia/kopia/cli.(*profileFlags).withProfiling(0x3a453304, 0x3a51fd9c)
        github.com/kopia/kopia/cli/profile.go:45 +0x325
github.com/kopia/kopia/cli.(*App).baseActionWithContext.func1.1({0x9525ddc, 0x3a49aab0})
        github.com/kopia/kopia/cli/app.go:474 +0x5f
github.com/kopia/kopia/cli.(*App).runAppWithContext.func1({0x9525ddc, 0x3a49aa98}, 0x3a4878f0, 0x3a453300, 0x3a51fe98)
        github.com/kopia/kopia/cli/app.go:445 +0x186
github.com/kopia/kopia/cli.(*App).runAppWithContext(0x3a453300, 0x3a4878f0, 0x3a51fe98)
        github.com/kopia/kopia/cli/app.go:446 +0x152
github.com/kopia/kopia/cli.(*App).baseActionWithContext.func1(0x3a0be370)
        github.com/kopia/kopia/cli/app.go:473 +0x59
github.com/alecthomas/kingpin.(*actionMixin).applyActions(0x3a4878fc, 0x3a0be370)
        github.com/alecthomas/kingpin@v1.3.8-0.20220615105907-eae6867f4166/actions.go:28 +0x4b
github.com/alecthomas/kingpin.(*Application).applyActions(0x3a2c4780, 0x3a0be370)
        github.com/alecthomas/kingpin@v1.3.8-0.20220615105907-eae6867f4166/app.go:560 +0xdd
github.com/alecthomas/kingpin.(*Application).execute(0x3a2c4780, 0x3a0be370, {0x3a260110, 0x2, 0x2})
        github.com/alecthomas/kingpin@v1.3.8-0.20220615105907-eae6867f4166/app.go:398 +0x77
github.com/alecthomas/kingpin.(*Application).Parse(0x3a2c4780, {0x3a090068, 0x3, 0x3})
        github.com/alecthomas/kingpin@v1.3.8-0.20220615105907-eae6867f4166/app.go:230 +0x17f
main.main()
        github.com/kopia/kopia/main.go:77 +0x1f4

goroutine 10 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x9d1ace0)
        github.com/golang/glog@v1.0.0/glog.go:882 +0x65
created by github.com/golang/glog.init.0
        github.com/golang/glog@v1.0.0/glog.go:410 +0x22e

goroutine 11 [select]:
go.opencensus.io/stats/view.(*worker).start(0x3a3802a0)
        go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0x9e
created by go.opencensus.io/stats/view.init.0
        go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x93

goroutine 28 [select]:
github.com/kopia/kopia/internal/cache.(*PersistentCache).sweepDirectoryPeriodically(0x3a0be960, {0x95265fc, 0x3a260530})
        github.com/kopia/kopia/internal/cache/persistent_lru_cache.go:208 +0xd7
created by github.com/kopia/kopia/internal/cache.NewPersistentCache
        github.com/kopia/kopia/internal/cache/persistent_lru_cache.go:359 +0x3a7

goroutine 29 [select]:
github.com/kopia/kopia/internal/cache.(*PersistentCache).sweepDirectoryPeriodically(0x3a0be9b0, {0x95265fc, 0x3a2605d0})
        github.com/kopia/kopia/internal/cache/persistent_lru_cache.go:208 +0xd7
created by github.com/kopia/kopia/internal/cache.NewPersistentCache
        github.com/kopia/kopia/internal/cache/persistent_lru_cache.go:359 +0x3a7

goroutine 30 [select]:
github.com/kopia/kopia/internal/cache.(*PersistentCache).sweepDirectoryPeriodically(0x3a0bea00, {0x95265fc, 0x3a260670})
        github.com/kopia/kopia/internal/cache/persistent_lru_cache.go:208 +0xd7
created by github.com/kopia/kopia/internal/cache.NewPersistentCache
        github.com/kopia/kopia/internal/cache/persistent_lru_cache.go:359 +0x3a7

goroutine 168 [semacquire]:
sync.runtime_Semacquire(0x4993bf68)
        runtime/sema.go:56 +0x30
sync.(*WaitGroup).Wait(0x4993bf60)
        sync/waitgroup.go:136 +0x81
github.com/kopia/kopia/internal/workshare.(*AsyncGroup).Wait(0x68669620)
        github.com/kopia/kopia/internal/workshare/workshare_waitgroup.go:76 +0x39
github.com/kopia/kopia/internal/workshare.(*AsyncGroup).Close(0x68669620)
        github.com/kopia/kopia/internal/workshare/workshare_waitgroup.go:88 +0x2b
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x952a0ac, 0x3a145a80}, {0x441238a8, 0x11})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:141 +0x20f
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x5ff02f90, 0x3a145a80}, {0x441238a8, 0x11})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:103 +0x170
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry.func1.1(0x3a269980, {0x0, 0x0})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:128 +0x44
github.com/kopia/kopia/internal/workshare.NewPool.func1()
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:62 +0xb0
created by github.com/kopia/kopia/internal/workshare.NewPool
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:55 +0xe6

goroutine 167 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0x4752cdf8)
        runtime/sema.go:56 +0x30
sync.(*WaitGroup).Wait(0x4752cdf0)
        sync/waitgroup.go:136 +0x81
github.com/kopia/kopia/internal/workshare.(*AsyncGroup).Wait(0x40772780)
        github.com/kopia/kopia/internal/workshare/workshare_waitgroup.go:76 +0x39
github.com/kopia/kopia/internal/workshare.(*AsyncGroup).Close(0x40772780)
        github.com/kopia/kopia/internal/workshare/workshare_waitgroup.go:88 +0x2b
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x952a0ac, 0x3a268880}, {0x461fe720, 0x3})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:141 +0x20f
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x5ff02f90, 0x3a268880}, {0x461fe720, 0x3})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:103 +0x170
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry.func1.1(0x3a269980, {0x0, 0x0})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:128 +0x44
github.com/kopia/kopia/internal/workshare.NewPool.func1()
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:62 +0xb0
created by github.com/kopia/kopia/internal/workshare.NewPool
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:55 +0xe6

goroutine 166 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0x4752d198)
        runtime/sema.go:56 +0x30
sync.(*WaitGroup).Wait(0x4752d190)
        sync/waitgroup.go:136 +0x81
github.com/kopia/kopia/internal/workshare.(*AsyncGroup).Wait(0x407728a0)
        github.com/kopia/kopia/internal/workshare/workshare_waitgroup.go:76 +0x39
github.com/kopia/kopia/internal/workshare.(*AsyncGroup).Close(0x407728a0)
        github.com/kopia/kopia/internal/workshare/workshare_waitgroup.go:88 +0x2b
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x952a0ac, 0x3a145680}, {0x4752cde0, 0x9})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:141 +0x20f
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x5ff02f90, 0x3a145680}, {0x4752cde0, 0x9})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:103 +0x170
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry.func1.1(0x3a269980, {0x0, 0x0})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:128 +0x44
github.com/kopia/kopia/internal/workshare.NewPool.func1()
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:62 +0xb0
created by github.com/kopia/kopia/internal/workshare.NewPool
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:55 +0xe6

goroutine 165 [select]:
github.com/kopia/kopia/internal/workshare.NewPool.func1()
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:59 +0x17b
created by github.com/kopia/kopia/internal/workshare.NewPool
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:55 +0xe6

goroutine 163 [select]:
github.com/kopia/kopia/internal/workshare.NewPool.func1()
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:59 +0x17b
created by github.com/kopia/kopia/internal/workshare.NewPool
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:55 +0xe6

goroutine 162 [semacquire]:
sync.runtime_Semacquire(0x3c9a56c8)
        runtime/sema.go:56 +0x30
sync.(*WaitGroup).Wait(0x3c9a56c0)
        sync/waitgroup.go:136 +0x81
github.com/kopia/kopia/internal/workshare.(*AsyncGroup).Wait(0x60aabb30)
        github.com/kopia/kopia/internal/workshare/workshare_waitgroup.go:76 +0x39
github.com/kopia/kopia/internal/workshare.(*AsyncGroup).Close(0x60aabb30)
        github.com/kopia/kopia/internal/workshare/workshare_waitgroup.go:88 +0x2b
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x952a0ac, 0x4e3acc60}, {0x74f94cc0, 0x23})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:141 +0x20f
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processEntry(0x3a16e600, {0x9525ddc, 0x3a49ab58}, {0x5ff02f90, 0x4e3acc60}, {0x74f94cc0, 0x23})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:103 +0x170
github.com/kopia/kopia/snapshot/snapshotfs.(*TreeWalker).processDirEntry.func1.1(0x3a269980, {0x0, 0x0})
        github.com/kopia/kopia/snapshot/snapshotfs/snapshot_tree_walker.go:128 +0x44
github.com/kopia/kopia/internal/workshare.NewPool.func1()
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:62 +0xb0
created by github.com/kopia/kopia/internal/workshare.NewPool
        github.com/kopia/kopia/internal/workshare/workshare_pool.go:55 +0xe6

About this issue

Original URL
State: open
Created 2 years ago
Comments: 23 (22 by maintainers)

Most upvoted comments

The walker is generally reused when processing N snapshots of the same source, because they are very likely to contain common subtrees, there are big savings when memoizing. This is just a performance optimization.

jkowalski on Aug 24, 2022

BTW I like the idea of having custom set implementation for hashes. Here’s one more idea:

We maintain one regular map (for quick add/lookup) + N sorted arrays with interpolation search.

On insert we add to the map only, and when the map becomes too large - (say above 1-10M items), we dump its contents into an array, sort it and clear the map. We’d keep a maximum number of these sorted arrays, If we end up producing too many, we’d compact them using merge-sort them into fewer, larger arrays. Alternatively, we let them grow and start doing parallel reads using goroutines.

All lookups would consult a map (O(1))+ all sorted arrays (O(log(log(N)). There’s some cost to growing the set, but if should be well amortized over its lifetime.

jkowalski on Aug 24, 2022

You’re right, we can’t use bloom filter for enqueued because it wouldn’t be correct. For used it’s perfectly fine for three reasons:

contentID only maps to a bounded amount of space (4MB-8MB), depending on the splitter.
false positives will possibly keep content from being released, which only affects efficiency not correctness
we can randomize hash functions used on each run of maintenance so that they won’t be giving false positives on the same content IDs over time, thus each content will eventually be released.

Because of this, even high false positive rate (say 1-in-100 or more) should be acceptable for used.

jkowalski on Aug 24, 2022