VictoriaMetrics: vmstorage crashes when built with an old version of glibc

Describe the bug

We have been running vm-cluster version 1.91.3 for a few weeks. We have met this for serveral times: fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

To Reproduce

N/A Do not know the circumstance that will trigger this issue

Version

v1.91.3-cluster (is built by us)

Logs

crash.log

Screenshots

screencapture

Used command-line flags

./vmstorage-prod -search.maxUniqueTimeseries=2000000 -storageDataPath=/lingtian/opt/vmstorage-data -retentionPeriod=31d -bigMergeConcurrency=1

Additional information

uname -a 
 5.12.12-1.el7.elrepo.x86_64 #1 SMP Thu Jun 17 10:00:40 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

We have 40 vmstorage in two physically separated cluster; each have 20 vmstorage.

About this issue

Original URL
State: open
Created a year ago
Comments: 15 (6 by maintainers)

Most upvoted comments

@renchili Binaries built with commands which has -prod suffix will be statically linked. You can check this by running:

ldd ./bin/vminsert-prod
        not a dynamic executable

zekker6 on Aug 2, 2023