VictoriaMetrics: vmstorage: unexpected fault address 0x7ef816d0ad4f
Describe the bug
vmstorage stoped with log:
unexpected fault address 0x7ef816d0ad4f
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x7ef816d0ad4f pc=0x46a0da]
goroutine 73314 [running]:
runtime.throw(0xa4b142, 0x5)
runtime/panic.go:1116 +0x72 fp=0xce641053e0 sp=0xce641053b0 pc=0x436112
runtime.sigpanic()
runtime/signal_unix.go:692 +0x443 fp=0xce64105410 sp=0xce641053e0 pc=0x44ccb3
runtime.memmove(0xc1a66f2000, 0x7ef816d0ad4f, 0x5fc)
runtime/memmove_amd64.s:354 +0x40a fp=0xce64105418 sp=0xce64105410 pc=0x46a0da
github.com/VictoriaMetrics/VictoriaMetrics/lib/fs.(*ReaderAt).MustReadAt(0xca84ede550, 0xc1a66f2000, 0x5fc, 0xa000, 0x362ed4f)
github.com/VictoriaMetrics/VictoriaMetrics/lib/fs/reader_at.go:79 +0x15d fp=0xce64105560 sp=0xce64105418 pc=0x6fb4ad
github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*partSearch).readInmemoryBlock(0xc9df7601e0, 0xc266dc3ac0, 0x0, 0x362ed4f, 0x0)
github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset/part_search.go:336 +0x12d fp=0xce64105638 sp=0xce64105560 pc=0x73ab4d
github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*partSearch).getInmemoryBlock(0xc9df7601e0, 0xc266dc3ac0, 0xbc, 0x100, 0xbd, 0xbc)
github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset/part_search.go:324 +0xae fp=0xce64105698 sp=0xce64105638 pc=0x73a85e
github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*partSearch).nextBlock(0xc9df7601e0, 0xce64105750, 0xbd)
github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset/part_search.go:258 +0x99 fp=0xce641056e0 sp=0xce64105698 pc=0x739ef9
github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*partSearch).Seek(0xc9df7601e0, 0xc82eba9980, 0x87, 0x1980)
github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset/part_search.go:148 +0x45f fp=0xce641057c8 sp=0xce641056e0 pc=0x7395df
github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*TableSearch).Seek(0xcba68c6148, 0xc82eba9980, 0x87, 0x1980)
github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset/table_search.go:96 +0x168 fp=0xce64105890 sp=0xce641057c8 pc=0x745e18
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).getTSIDByMetricName(0xcba68c6140, 0xc74752be68, 0xc40638186f, 0x85, 0x76791, 0xc00024e210, 0xc74752be68)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/index_db.go:1385 +0x1c5 fp=0xce64105970 sp=0xce64105890 pc=0x76cf15
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*indexSearch).GetOrCreateTSIDByName(0xcba68c6140, 0xc74752be68, 0xc40638186f, 0x85, 0x76791, 0x0, 0x20)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/index_db.go:554 +0x1da fp=0xce641059f8 sp=0xce64105970 pc=0x764caa
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*Storage).add(0xc000346000, 0xc74750e000, 0x0, 0x1000, 0xc4b905e000, 0x8ad, 0x1c00, 0x40, 0x120, 0x10000, ...)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/storage.go:1205 +0xc3b fp=0xce64105bd0 sp=0xce641059f8 pc=0x7aa9bb
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage.(*Storage).AddRows(0xc000346000, 0xc4b905e000, 0x8ad, 0x1c00, 0xcde375a940, 0x0, 0x0)
github.com/VictoriaMetrics/VictoriaMetrics/lib/storage/storage.go:1088 +0x11c fp=0xce64105d20 sp=0xce64105bd0 pc=0x7a9a1c
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/transport.(*Server).processVMInsertConn(0xc019cfe580, 0xc378446480, 0x4, 0xa5b8d9)
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/transport/server.go:356 +0x45e fp=0xce64105eb0 sp=0xce64105d20 pc=0x7c1bfe
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/transport.(*Server).RunVMInsert.func1(0xc019cfe580, 0xb0e0e0, 0xc3389cabc0)
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/transport/server.go:161 +0x364 fp=0xce64105fc8 sp=0xce64105eb0 pc=0x7c7f44
runtime.goexit()
runtime/asm_amd64.s:1373 +0x1 fp=0xce64105fd0 sp=0xce64105fc8 pc=0x468b21
created by github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/transport.(*Server).RunVMInsert
github.com/VictoriaMetrics/VictoriaMetrics/app/vmstorage/transport/server.go:133 +0x21e
detail log: panic.log
Version
The line returned when passing --version command line flag to binary. For example:
$ ./vmstorage-prod -version
vmstorage-20200622-062829-heads-cluster-0-gf227799c
$ ./vminsert-prod -version
vminsert-20200610-061147-heads-cluster-0-gd71b6e65
Used command-line flags
./vmstorage-prod -storageDataPath /data1/vmdata -retentionPeriod 6 -search.maxUniqueTimeseries 5000000
Additional context
the vmstorage instance which panic, even though i start it, it didn’t accept sample any more(rate(vm_vminsert_metrics_read_total) == 0).
and vminsert instance exposed lots of log like
2020-06-22T21:40:30.343Z warn VictoriaMetrics/app/vminsert/main.go:180 error in "/insert/0/influx/write": cannot handle more than 32 concurrent inserts during 10s; possible solutions: increase `-insert.maxQueueDuration`, increase `-maxConcurrentInserts`, increase server capacity
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (8 by maintainers)
Commits related to this issue
- lib/fs: an attempt to fix SIGBUS error by rounding mmap`ed region to multiple of 4KB pages Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581 — committed to VictoriaMetrics/VictoriaMetrics by valyala 4 years ago
- lib/fs: an attempt to fix SIGBUS error by rounding mmap`ed region to multiple of 4KB pages Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581 — committed to VictoriaMetrics/VictoriaMetrics by valyala 4 years ago
- lib/fs: fall back to cgo copy for copying the last 4KB of mmaped data This probably should fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/581 — committed to VictoriaMetrics/VictoriaMetrics by valyala 4 years ago
@valyala
after upgrade, it’s panic again
@valyala
it’s panic just now panic2.log
version