restic: Performance improvement for content checksum
I think there a can be get a 300% performance gain by using the Blake2b checksum instead of SHA-256 (which is a improved version of the Blake algorithm used in SHA-3)
Benchmark from github.com/codahale/blake2:
My NAS Intel® Atom™ CPU C2550 @ 2.41GHz, 4 cores, 1MByte L1 cache
BenchmarkBlake2B-4 200 8354838 ns/op 125.51 MB/s
BenchmarkMD5-4 500 2568903 ns/op 408.18 MB/s
BenchmarkSHA1-4 300 5342146 ns/op 196.28 MB/s
BenchmarkSHA256-4 100 13622543 ns/op 76.97 MB/s
BenchmarkSHA512-4 200 8909665 ns/op 117.69 MB/s
KVM VPS (depends on host os scheduling) Westmere E56xx/L56xx/X56xx (Nehalem-C) @ 2294.686 MHz, 4MByte L1 cache
BenchmarkBlake2B 1000 2633524 ns/op 398.16 MB/s
BenchmarkMD5 1000 2095968 ns/op 500.28 MB/s
BenchmarkSHA1 1000 2677232 ns/op 391.66 MB/s
BenchmarkSHA256 500 7270033 ns/op 144.23 MB/s
BenchmarkSHA512 500 4834102 ns/op 216.91 MB/s
MacBookPro11,1 2.4 GHz (i5-4258U) Intel Core i5 Haswell
BenchmarkBlake2B-4 1000 2130623 ns/op 492.15 MB/s
BenchmarkMD5-4 1000 1865851 ns/op 561.98 MB/s
BenchmarkSHA1-4 500 2346633 ns/op 446.84 MB/s
BenchmarkSHA256-4 200 6472990 ns/op 161.99 MB/s
BenchmarkSHA512-4 300 3949461 ns/op 265.50 MB/s
The specification can be read here: https://blake2.net/ Wikipedia: https://en.wikipedia.org/wiki/BLAKE_(hash_function)
On my macbook this is a performance gain of 300% 👍
If seen bazil filesystem using Blake2 for chunking: https://github.com/bazil/bazil/blob/master/cas/chunks/chunkutil/hash.go Architecture: https://bazil.org/doc/architecture/#cas
Just an idea to use newer and faster technologies 😃
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 1
- Comments: 21 (10 by maintainers)
In case anybody missed the Blake3 hype:
Maybe worth considering instead of blake2b.