VictoriaLogs: victorialogs panic

Describe the bug

Panic occurs after running for a period of time.

To Reproduce

Even if there is no access operation in the middle of the night, panic will occur.

Version

victorialogs version from v0.11.0 to v0.17.0

Logs

{"ts":"2024-06-05T20:40:17.034Z","level":"info","caller":"VictoriaMetrics/lib/logstorage/datadb.go:599","msg":"merged (12 parts, 40671346 rows, 15787 blocks, 2384297208 bytes) into (1 part, 40671346 rows, 14815 blocks, 2380922535 bytes) in 104.231 seconds at 390203 rows/sec to \"/storage/partitions/20240605/datadb/17D62E170E935969\""}
SIGSEGV: segmentation violation
PC=0x415ef7 m=33 sigcode=1 addr=0x20

goroutine 0 gp=0xc00e0748c0 m=33 mp=0xc001fe6808 [idle]:
runtime.(*mspan).typePointersOfUnchecked(0xc030dcc000?, 0x7f039f82f948?)
	runtime/mbitmap_allocheaders.go:202 +0x37 fp=0x7f039f82f8f8 sp=0x7f039f82f8d8 pc=0x415ef7
runtime.scanobject(0xc000052668?, 0xc000052668)
	runtime/mgcmark.go:1446 +0xb5 fp=0x7f039f82f988 sp=0x7f039f82f8f8 pc=0x421dd5
runtime.gcDrain(0xc000052668, 0x7)
	runtime/mgcmark.go:1242 +0x1f4 fp=0x7f039f82f9f0 sp=0x7f039f82f988 pc=0x421734
runtime.gcDrainMarkWorkerIdle(...)
	runtime/mgcmark.go:1114
runtime.gcBgMarkWorker.func2()
	runtime/mgc.go:1406 +0x6f fp=0x7f039f82fa40 sp=0x7f039f82f9f0 pc=0x41dd6f
runtime.systemstack(0x20000)
	runtime/asm_amd64.s:509 +0x4a fp=0x7f039f82fa50 sp=0x7f039f82fa40 pc=0x472aca

goroutine 36 gp=0xc00029c380 m=33 mp=0xc001fe6808 [GC worker (active)]:
runtime.systemstack_switch()
	runtime/asm_amd64.s:474 +0x8 fp=0xc0002a2f50 sp=0xc0002a2f40 pc=0x472a68
runtime.gcBgMarkWorker()
	runtime/mgc.go:1370 +0x1f2 fp=0xc0002a2fe0 sp=0xc0002a2f50 pc=0x41da32
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc0002a2fe8 sp=0xc0002a2fe0 pc=0x474a81
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1234 +0x1c

goroutine 1 gp=0xc0000061c0 m=nil [chan receive, 156 minutes]:
runtime.gopark(0x84ebc5?, 0x94d160?, 0x1?, 0x3?, 0xc00052bdc0?)
	runtime/proc.go:402 +0xce fp=0xc000285d30 sp=0xc000285d10 pc=0x43d4ae
runtime.chanrecv(0xc000d10240, 0xc00052be08, 0x1)
	runtime/chan.go:583 +0x3bf fp=0xc000285da8 sp=0xc000285d30 pc=0x40759f
runtime.chanrecv1(0xc000d10240?, 0xc00052be18?)
	runtime/chan.go:442 +0x12 fp=0xc000285dd0 sp=0xc000285da8 pc=0x4071b2
github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil.WaitForSigterm()
	github.com/VictoriaMetrics/VictoriaMetrics/lib/procutil/signal.go:23 +0xa5 fp=0xc000285e58 sp=0xc000285dd0 pc=0x84f4c5
main.main()
	github.com/VictoriaMetrics/VictoriaMetrics/app/victoria-logs/main.go:53 +0x317 fp=0xc000285f50 sp=0xc000285e58 pc=0x84fd17
runtime.main()
	runtime/proc.go:271 +0x29d fp=0xc000285fe0 sp=0xc000285f50 pc=0x43d05d
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000285fe8 sp=0xc000285fe0 pc=0x474a81

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle), 156 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xce fp=0xc000068fa8 sp=0xc000068f88 pc=0x43d4ae
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.forcegchelper()
	runtime/proc.go:326 +0xb3 fp=0xc000068fe0 sp=0xc000068fa8 pc=0x43d313
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000068fe8 sp=0xc000068fe0 pc=0x474a81
created by runtime.init.6 in goroutine 1
	runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
runtime.gopark(0x1083a01?, 0x1083a60?, 0xc?, 0x9?, 0x1?)
	runtime/proc.go:402 +0xce fp=0xc000069780 sp=0xc000069760 pc=0x43d4ae
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.bgsweep(0xc000092000)
	runtime/mgcsweep.go:318 +0xdf fp=0xc0000697c8 sp=0xc000069780 pc=0x426fdf
runtime.gcenable.gowrap1()
	runtime/mgc.go:203 +0x25 fp=0xc0000697e0 sp=0xc0000697c8 pc=0x41b845

Screenshots

No response

Used command-line flags

No response

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 40 (19 by maintainers)

Commits related to this issue

Most upvoted comments

Thanks for the info @Haleygo! I just tried and can no longer reproduce the problem. We changed more things so not entirely sure what caused it or what solved it, but the operator runs fine again now. Thanks 👍🏻

@kk-kwok , could you check whether the panic exists in v0.29.0-victorialogs? This release has been built with Go1.23.1, so, hopefully, the panic shouldn’t be reproduced there.

I just updated to the latest version 0.29.0 two hours ago. Everything is working fine now. Thank you very much!

@kk-kwok , I prepared Go1.21.11 builds of VictoriaLogs from the branch victorialogs-go1.21-issue-6425. There is a Docker image docker.io/victoriametrics/victoria-logs:heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615 at DockerHub and the amd64-linux binary attached here. Could you try these builds and verify whether they are free from the panic? If yes, then the issue is definitely in Go1.22, and in this case we need to help Go core developers narrowing down the issue at golang/go#66383 , so they could fix it quickly. victoria-logs-linux-amd64-heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615.tar.gz

Now that the new image compiled with go1.21.11 is used to run, I will observe whether panic occurs in the next day. Previous versions including v0.19.0 will panic after running for a few hours.

@valyala LGTM! I have been running the image compiled by go1.21.11 for 25 hours now, and it has been running normally without panic.

@valyala It has been running normally for 5 days without panic.

@kk-kwok , I prepared Go1.21.11 builds of VictoriaLogs from the branch victorialogs-go1.21-issue-6425. There is a Docker image docker.io/victoriametrics/victoria-logs:heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615 at DockerHub and the amd64-linux binary attached here. Could you try these builds and verify whether they are free from the panic? If yes, then the issue is definitely in Go1.22, and in this case we need to help Go core developers narrowing down the issue at golang/go#66383 , so they could fix it quickly. victoria-logs-linux-amd64-heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615.tar.gz

Now that the new image compiled with go1.21.11 is used to run, I will observe whether panic occurs in the next day. Previous versions including v0.19.0 will panic after running for a few hours.

@valyala LGTM! I have been running the image compiled by go1.21.11 for 25 hours now, and it has been running normally without panic.

Are there any related metrics to query column headers?

You can query the information about columns with the following pipes:

For example, the following query returns field names across all the logs for the last 5 minutes:

_time:5m | field_names

The following query returns the number of unique fields during the last 5 minutes:

_time:5m | field_names | count()

@kk-kwok, thanks for the update!

BTW, increasing maxColumnsPerBlock to too big values isn’t recommended, since this may lead to too big memory usage during data ingestion. This also may result in significant query execution slowdown. See cf2e7d0 .

Why do you need increasing this constant from the the default 1_000 value to 100_000?

@valyala Just FYI. When I start it directly using the official Docker image, the Pod will panic shortly after. The log is as follows:

{"ts":"2024-10-16T17:58:14.078-0800","level":"panic","caller":"VictoriaMetrics/lib/logstorage/block_data.go:154","msg":"FATAL: /storage/partitions/20241017/datadb/17FF1463B37459F8/columns_header.bin: cannot unmarshal columnsHeader: too many column headers: 1392; mustn't exceed 1000"}

Hey @kk-kwok! Could you try the following image https://hub.docker.com/layers/victoriametrics/victoria-logs/heads-vlogs-go-1.23-0-g566c437b0e/images/sha256-42c22e99fa42a1a5459a84ff2e7f14987770447e2a383aee85a8fc1057d4f1db?context=explore ? It is built with Go v1.23.0.

I’m very sorry for the late reply, because we modified maxColumnsPerBlock to 100000 and then recompiled the image, so we waited until the new version v0.29.0 came out before updating the production environment version. Now everything is normal.

@kk-kwok , could you try running VictoriaLogs Docker image heads-victorialogs-go1.21-issue-6425-0-gaadb73740f? This image is built with go1.23rc1 according to the suggestion from @mknyszek at golang/go#66383 (comment) . This will help determining whether the issue is fixed in the upcoming Go1.23 release.

If you prefer running executable instead of Docker image, then please download and run the following executable - victoria-logs-linux-amd64-heads-victorialogs-go1.21-issue-6425-0-gaadb73740f.tar.gz.

Note that the name of the docker image and the executable contains go1.21, while in reality they are built with Go1.23rc1 - the go1.21 is inherited from the branch name - https://github.com/VictoriaMetrics/VictoriaMetrics/tree/victorialogs-go1.21-issue-6425 , which was used for building these binaries. You can build VictoriaLogs binaries from this branch on yourself according to these docs.

If the panic is still present in these binaries built with Go1.23rc1, then could you run them with GODEBUG=asyncpreemptoff=1 environment variable as suggested at golang/go#66383 (comment) and verify whether this helps avoiding the panic?

@valyala It has been running normally for 20 hours without panic.

@kk-kwok , I prepared Go1.21.11 builds of VictoriaLogs from the branch victorialogs-go1.21-issue-6425. There is a Docker image docker.io/victoriametrics/victoria-logs:heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615 at DockerHub and the amd64-linux binary attached here. Could you try these builds and verify whether they are free from the panic? If yes, then the issue is definitely in Go1.22, and in this case we need to help Go core developers narrowing down the issue at golang/go#66383 , so they could fix it quickly.

victoria-logs-linux-amd64-heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615.tar.gz

Now that the new image compiled with go1.21.11 is used to run, I will observe whether panic occurs in the next day. Previous versions including v0.19.0 will panic after running for a few hours.

@kk-kwok , I prepared Go1.21.11 builds of VictoriaLogs from the branch victorialogs-go1.21-issue-6425. There is a Docker image docker.io/victoriametrics/victoria-logs:heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615 at DockerHub and the amd64-linux binary attached here. Could you try these builds and verify whether they are free from the panic? If yes, then the issue is definitely in Go1.22, and in this case we need to help Go core developers narrowing down the issue at https://github.com/golang/go/issues/66383 , so they could fix it quickly.

victoria-logs-linux-amd64-heads-victorialogs-go1.21-issue-6425-0-gbeee1cc615.tar.gz

Also, try using VictoriaLogs v0.18.0. It is built with Go1.22.4, which may contain the fix for this issue. See the list of fixes in Go1.22.4

Yesterday, it was updated to v0.18.0, and panic often occurred.