restic: Performance improvement for content checksum
I think there a can be get a 300% performance gain by using the Blake2b checksum instead of SHA-256 (which is a improved version of the Blake algorithm used in SHA-3)
Benchmark from github.com/codahale/blake2:
My NAS Intel® Atom™ CPU C2550 @ 2.41GHz, 4 cores, 1MByte L1 cache
BenchmarkBlake2B-4       200       8354838 ns/op     125.51 MB/s
BenchmarkMD5-4           500       2568903 ns/op     408.18 MB/s
BenchmarkSHA1-4          300       5342146 ns/op     196.28 MB/s
BenchmarkSHA256-4        100      13622543 ns/op      76.97 MB/s
BenchmarkSHA512-4        200       8909665 ns/op     117.69 MB/s
KVM VPS (depends on host os scheduling) Westmere E56xx/L56xx/X56xx (Nehalem-C) @ 2294.686 MHz, 4MByte L1 cache
BenchmarkBlake2B        1000       2633524 ns/op     398.16 MB/s
BenchmarkMD5        1000       2095968 ns/op     500.28 MB/s
BenchmarkSHA1       1000       2677232 ns/op     391.66 MB/s
BenchmarkSHA256      500       7270033 ns/op     144.23 MB/s
BenchmarkSHA512      500       4834102 ns/op     216.91 MB/s
MacBookPro11,1 2.4 GHz (i5-4258U) Intel Core i5 Haswell
BenchmarkBlake2B-4      1000       2130623 ns/op     492.15 MB/s
BenchmarkMD5-4          1000       1865851 ns/op     561.98 MB/s
BenchmarkSHA1-4          500       2346633 ns/op     446.84 MB/s
BenchmarkSHA256-4        200       6472990 ns/op     161.99 MB/s
BenchmarkSHA512-4        300       3949461 ns/op     265.50 MB/s
The specification can be read here: https://blake2.net/ Wikipedia: https://en.wikipedia.org/wiki/BLAKE_(hash_function)
On my macbook this is a performance gain of 300% 👍
If seen bazil filesystem using Blake2 for chunking: https://github.com/bazil/bazil/blob/master/cas/chunks/chunkutil/hash.go Architecture: https://bazil.org/doc/architecture/#cas
Just an idea to use newer and faster technologies 😃
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Reactions: 1
- Comments: 21 (10 by maintainers)
In case anybody missed the Blake3 hype:
Maybe worth considering instead of blake2b.