VictoriaLogs: victorialogs panic
Describe the bug
Panic occurs after running for a period of time.
To Reproduce
Even if there is no access operation in the middle of the night, panic will occur.
Version
victorialogs version from v0.11.0 to v0.17.0
Logs
{"ts":"2024-06-05T20:40:17.034Z","level":"info","caller":"VictoriaMetrics/lib/logstorage/datadb.go:599","msg":"merged (12 parts, 40671346 rows, 15787 blocks, 2384297208 bytes) into (1 part, 40671346 rows, 14815 blocks, 2380922535 bytes) in 104.231 seconds at 390203 rows/sec to \"/storage/partitions/20240605/datadb/17D62E170E935969\""}
SIGSEGV: segmentation violation
PC=0x415ef7 m=33 sigcode=1 addr=0x20
goroutine 0 gp=0xc00e0748c0 m=33 mp=0xc001fe6808 [idle]:
runtime.(*mspan).typePointersOfUnchecked(0xc030dcc000?, 0x7f039f82f948?)
runtime/mbitmap_allocheaders.go:202 +0x37 fp=0x7f039f82f8f8 sp=0x7f039f82f8d8 pc=0x415ef7
runtime.scanobject(0xc000052668?, 0xc000052668)
runtime/mgcmark.go:1446 +0xb5 fp=0x7f039f82f988 sp=0x7f039f82f8f8 pc=0x421dd5
runtime.gcDrain(0xc000052668, 0x7)
runtime/mgcmark.go:1242 +0x1f4 fp=0x7f039f82f9f0 sp=0x7f039f82f988 pc=0x421734
runtime.gcDrainMarkWorkerIdle(...)
runtime/mgcmark.go:1114
runtime.gcBgMarkWorker.func2()
runtime/mgc.go:1406 +0x6f fp=0x7f039f82fa40 sp=0x7f039f82f9f0 pc=0x41dd6f
runtime.systemstack(0x20000)
runtime/asm_amd64.s:509 +0x4a fp=0x7f039f82fa50 sp=0x7f039f82fa40 pc=0x472aca
goroutine 36 gp=0xc00029c380 m=33 mp=0xc001fe6808 [GC worker (active)]:
runtime.systemstack_switch()
runtime/asm_amd64.s:474 +0x8 fp=0xc0002a2f50 sp=0xc0002a2f40 pc=0x472a68
runtime.gcBgMarkWorker()
runtime/mgc.go:1370 +0x1f2 fp=0xc0002a2fe0 sp=0xc0002a2f50 pc=0x41da32
runtime.goexit({})
runtime/asm_amd64.s:1695 +0x1 fp=0xc0002a2fe8 sp=0xc0002a2fe0 pc=0x474a81
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1234 +0x1c
goroutine 1 gp=0xc0000061c0 m=nil [chan receive, 156 minutes]:
runtime.gopark(0x84ebc5?, 0x94d160?, 0x1?, 0x3?, 0xc00052bdc0?)
runtime/proc.go:402 +0xce fp=0xc000285d30 sp=0xc000285d10 pc=0x43d4ae
runtime.chanrecv(0xc000d10240, 0xc00052be08, 0x1)
runtime/chan.go:583 +0x3bf fp=0xc000285da8 sp=0xc000285d30 pc=0x40759f
runtime.chanrecv1(0xc000d10240?, 0xc00052be18?)
runtime/chan.go:442 +0x12 fp=0xc000285dd0 sp=0xc000285da8 pc=0x4071b2
github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil.WaitForSigterm()
github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil/signal.go:23 +0xa5 fp=0xc000285e58 sp=0xc000285dd0 pc=0x84f4c5
main.main()
github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-logs/main.go:53 +0x317 fp=0xc000285f50 sp=0xc000285e58 pc=0x84fd17
runtime.main()
runtime/proc.go:271 +0x29d fp=0xc000285fe0 sp=0xc000285f50 pc=0x43d05d
runtime.goexit({})
runtime/asm_amd64.s:1695 +0x1 fp=0xc000285fe8 sp=0xc000285fe0 pc=0x474a81
goroutine 2 gp=0xc000006c40 m=nil [force gc (idle), 156 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:402 +0xce fp=0xc000068fa8 sp=0xc000068f88 pc=0x43d4ae
runtime.goparkunlock(...)
runtime/proc.go:408
runtime.forcegchelper()
runtime/proc.go:326 +0xb3 fp=0xc000068fe0 sp=0xc000068fa8 pc=0x43d313
runtime.goexit({})
runtime/asm_amd64.s:1695 +0x1 fp=0xc000068fe8 sp=0xc000068fe0 pc=0x474a81
created by runtime.init.6 in goroutine 1
runtime/proc.go:314 +0x1a
goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
runtime.gopark(0x1083a01?, 0x1083a60?, 0xc?, 0x9?, 0x1?)
runtime/proc.go:402 +0xce fp=0xc000069780 sp=0xc000069760 pc=0x43d4ae
runtime.goparkunlock(...)
runtime/proc.go:408
runtime.bgsweep(0xc000092000)
runtime/mgcsweep.go:318 +0xdf fp=0xc0000697c8 sp=0xc000069780 pc=0x426fdf
runtime.gcenable.gowrap1()
runtime/mgc.go:203 +0x25 fp=0xc0000697e0 sp=0xc0000697c8 pc=0x41b845
Screenshots
No response
Used command-line flags
No response
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 40 (19 by maintainers)
Commits related to this issue
- lib/logstorage/consts.go: document that it isn't recommended setting maxColumnsPerBlock constant to too big values This should help avoiding cases like this one - https://github.com/VictoriaMetrics/V... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage/consts.go: document that it isn't recommended setting maxColumnsPerBlock constant to too big values This should help avoiding cases like this one - https://github.com/VictoriaMetrics/V... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: increase the the maximum number of columns per block from 1000 to 2000 This will allow storing wide events with up to 2K fields per event into VictoriaLogs. While at it, remove the mi... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: increase the the maximum number of columns per block from 1000 to 2000 This will allow storing wide events with up to 2K fields per event into VictoriaLogs. While at it, remove the mi... — committed to VictoriaMetrics/VictoriaMetrics by valyala 2 years ago
- lib/logstorage/consts.go: document that it isn't recommended setting maxColumnsPerBlock constant to too big values This should help avoiding cases like this one - https://github.com/VictoriaMetrics/V... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: increase the the maximum number of columns per block from 1000 to 2000 This will allow storing wide events with up to 2K fields per event into VictoriaLogs. While at it, remove the mi... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage/consts.go: document that it isn't recommended setting maxColumnsPerBlock constant to too big values This should help avoiding cases like this one - https://github.com/VictoriaMetrics/V... — committed to truepele/VictoriaMetrics by valyala 2 years ago
- lib/logstorage: increase the the maximum number of columns per block from 1000 to 2000 This will allow storing wide events with up to 2K fields per event into VictoriaLogs. While at it, remove the mi... — committed to truepele/VictoriaMetrics by valyala 2 years ago
Thanks for the info @Haleygo! I just tried and can no longer reproduce the problem. We changed more things so not entirely sure what caused it or what solved it, but the operator runs fine again now. Thanks 👍🏻
I just updated to the latest version 0.29.0 two hours ago. Everything is working fine now. Thank you very much!
@valyala It has been running normally for 5 days without panic.
@valyala LGTM! I have been running the image compiled by go1.21.11 for 25 hours now, and it has been running normally without panic.
You can query the information about columns with the following pipes:
field_namesblock_stats.For example, the following query returns field names across all the logs for the last 5 minutes:
The following query returns the number of unique fields during the last 5 minutes:
@valyala Just FYI. When I start it directly using the official Docker image, the Pod will panic shortly after. The log is as follows:
I’m very sorry for the late reply, because we modified maxColumnsPerBlock to 100000 and then recompiled the image, so we waited until the new version v0.29.0 came out before updating the production environment version. Now everything is normal.
@valyala It has been running normally for 20 hours without panic.
Now that the new image compiled with go1.21.11 is used to run, I will observe whether panic occurs in the next day. Previous versions including v0.19.0 will panic after running for a few hours.
@kk-kwok , I prepared Go1.21.11 builds of VictoriaLogs from the branch victorialogs-go1.21-issue-6425. There is a Docker image
docker.io/victoriametrics/victoria-logs:heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615at DockerHub and the amd64-linux binary attached here. Could you try these builds and verify whether they are free from the panic? If yes, then the issue is definitely in Go1.22, and in this case we need to help Go core developers narrowing down the issue at https://github.com/golang/go/issues/66383 , so they could fix it quickly.victoria-logs-linux-amd64-heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615.tar.gz
Yesterday, it was updated to v0.18.0, and panic often occurred.