murmur3: Flaky "unsafe pointer conversion"-panics with -race and go1.14rc1
I cannot reproduce the panic below with a minimal example and it occurs very often, but neither all the time nor in the same test - so this report might be entirely unhelpful, but I am reporting it anyway in case it makes any sense to you:
When running the Syncthing database testsuite on go1.14rc1 with race detection enabled on linux/amd64 I get the following panic:
$ go test -v -race -short -count 1 github.com/syncthing/syncthing/lib/db/ 2>&1 | tee test.out
fatal error: checkptr: unsafe pointer conversion
goroutine 148 [running]:
runtime.throw(0xd106c0, 0x23)
/media/ext4_data/Linux/source/go/src/runtime/panic.go:1112 +0x72 fp=0xc00022ba50 sp=0xc00022ba20 pc=0x4630d2
runtime.checkptrAlignment(0xc00011e3f1, 0xc40880, 0x1)
/media/ext4_data/Linux/source/go/src/runtime/checkptr.go:13 +0xd0 fp=0xc00022ba80 sp=0xc00022ba50 pc=0x4348a0
github.com/spaolacci/murmur3.(*digest128).bmix(0xc000123c80, 0xc00011e3f1, 0x20, 0x2f, 0x0, 0x48, 0x58)
/media/ext4_data/Coding/go/pkg/mod/github.com/spaolacci/murmur3@v1.1.0/murmur128.go:67 +0xc0 fp=0xc00022bae8 sp=0xc00022ba80 pc=0xb649f0
github.com/spaolacci/murmur3.(*digest).Write(0xc000123c80, 0xc00011e3f1, 0x20, 0x2f, 0xb64474, 0xc000123c80, 0xc000123c80)
/media/ext4_data/Coding/go/pkg/mod/github.com/spaolacci/murmur3@v1.1.0/murmur.go:51 +0x127 fp=0xc00022bb98 sp=0xc00022bae8 pc=0xb63dd7
github.com/spaolacci/murmur3.(*digest128).Write(0xc000123c80, 0xc00011e3f1, 0x20, 0x2f, 0x10, 0x10, 0x21)
<autogenerated>:1 +0x6a fp=0xc00022bbf8 sp=0xc00022bb98 pc=0xb6558a
github.com/willf/bloom.baseHashes(0xc00011e3f1, 0x20, 0x2f, 0x0, 0x0, 0x0, 0x0)
/media/ext4_data/Coding/go/pkg/mod/github.com/willf/bloom@v2.0.3+incompatible/bloom.go:97 +0xb9 fp=0xc00022bc68 sp=0xc00022bbf8 pc=0xb6d179
github.com/willf/bloom.(*BloomFilter).Test(0xc000134700, 0xc00011e3f1, 0x20, 0x2f, 0xdf6fe0)
/media/ext4_data/Coding/go/pkg/mod/github.com/willf/bloom@v2.0.3+incompatible/bloom.go:183 +0x6b fp=0xc00022bd20 sp=0xc00022bc68 pc=0xb6deab
github.com/syncthing/syncthing/lib/db.(*Lowlevel).gcBlocks(0xc000098870, 0x0, 0x0)
/media/ext4_data/Coding/go/src/github.com/syncthing/syncthing/lib/db/lowlevel.go:570 +0x4d4 fp=0xc00022beb8 sp=0xc00022bd20 pc=0xb798b4
github.com/syncthing/syncthing/lib/db.(*Lowlevel).gcRunner(0xc000098870)
/media/ext4_data/Coding/go/src/github.com/syncthing/syncthing/lib/db/lowlevel.go:483 +0x279 fp=0xc00022bfd8 sp=0xc00022beb8 pc=0xb78fb9
runtime.goexit()
/media/ext4_data/Linux/source/go/src/runtime/asm_amd64.s:1375 +0x1 fp=0xc00022bfe0 sp=0xc00022bfd8 pc=0x4969c1
created by github.com/syncthing/syncthing/lib/db.NewLowlevel
/media/ext4_data/Coding/go/src/github.com/syncthing/syncthing/lib/db/lowlevel.go:65 +0x36a
(rest of the backtrace does not include murmur3 nor bloom).
Do you have any ideas what this might be about? Thanks in advance for any help.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 8
- Comments: 15
Commits related to this issue
- Use encoding/binary instead of unsafe (fixes #29) This removes the incorrect unsafe usage, instead using the marshalling operations from encoding/binary. Obviously, it's a bit slower: benchmark ... — committed to calmh/murmur3 by calmh 4 years ago
- Use encoding/binary instead of unsafe (fixes #29) This removes the incorrect unsafe usage, instead using the marshalling operations from encoding/binary. Obviously, it's a bit slower: benchmark ... — committed to calmh/murmur3 by calmh 4 years ago
The fork of @twmb seems quite nice: https://github.com/twmb/murmur3
Fundamentally, the unsafe code needs changing. Feel free to take my patches: https://github.com/twmb/murmur3/compare/b0c3ada...HEAD
Note that using binary.LittleEndian (or my path) will resolve this repos tickets around issues across architectures, as well.
@sheregeda did you read the thread? Especially https://github.com/spaolacci/murmur3/issues/29#issuecomment-607429181