node_exporter: fatal error: all goroutines are asleep - deadlock!

Host operating system: output of uname -a

Linux vm 5.4.0-136-generic #153-Ubuntu SMP Thu Nov 24 15:56:58 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.6.0 (branch: HEAD, revision: ff7f9d69b645cb691dd3e84dc3afc88f5c006962) build user: root@f9c3ed0cfbd3 build date: 20230527-12:03:54 go version: go1.20.4 platform: linux/amd64 tags: netgo osusergo static_build

node_exporter command line flags

[Service] User=node_exporter Group=node_exporter Type=simple ExecStart=/usr/local/bin/node_exporter --collector.systemd

node_exporter log output

May 30 14:46:55 vm node_exporter[412095]: fatal error: all goroutines are asleep - deadlock! May 30 14:46:56 vm node_exporter[412095]: goroutine 1 [semacquire, 1495 minutes]: May 30 14:46:56 vm node_exporter[412095]: sync.runtime_Semacquire(0xc0000061a0?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/sema.go:62 +0x27 May 30 14:46:56 vm node_exporter[412095]: sync.(*WaitGroup).Wait(0x7f3462854401?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/sync/waitgroup.go:116 +0x4b May 30 14:46:56 vm node_exporter[412095]: golang.org/x/sync/errgroup.(*Group).Wait(0xc0001faa80) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:53 +0x27 May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/exporter-toolkit/web.ServeMultiple({0xc0002217b0, 0x1, 0xbcadc7?}, 0xc00016d770, 0xc0001642e8, {0xce3360?, 0xc0000419c0}) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:235 +0x151 May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/exporter-toolkit/web.ListenAndServe(0x11381a0?, 0xc0001642e8, {0xce3360, 0xc0000419c0}) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:268 +0x488 May 30 14:46:56 vm node_exporter[412095]: main.main() May 30 14:46:56 vm node_exporter[412095]: /app/node_exporter.go:210 +0x164e May 30 14:46:56 vm node_exporter[412095]: goroutine 16 [IO wait, 1491 minutes]: May 30 14:46:56 vm node_exporter[412095]: internal/poll.runtime_pollWait(0x7f34629ff898, 0x72) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/netpoll.go:306 +0x89 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).wait(0xc000225780?, 0x5?, 0x0) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).waitRead(…) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*FD).Accept(0xc000225780) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd May 30 14:46:56 vm node_exporter[412095]: net.(*netFD).accept(0xc000225780) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/fd_unix.go:172 +0x35 May 30 14:46:56 vm node_exporter[412095]: net.(*TCPListener).accept(0xc0001fc408) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 May 30 14:46:56 vm node_exporter[412095]: net.(*TCPListener).Accept(0xc0001fc408) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/tcpsock.go:297 +0x3d May 30 14:46:56 vm node_exporter[412095]: net/http.(*Server).Serve(0xc00016d770, {0xce8800, 0xc0001fc408}) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:3059 +0x385 May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/exporter-toolkit/web.Serve({0xce8800, 0xc0001fc408}, 0xc00016d770, 0xc0001642e8, {0xce3360?, 0xc0000419c0}) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:278 +0x35f May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1() May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:232 +0x31 May 30 14:46:56 vm node_exporter[412095]: golang.org/x/sync/errgroup.(*Group).Go.func1() May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75 +0x64 May 30 14:46:56 vm node_exporter[412095]: created by golang.org/x/sync/errgroup.(*Group).Go May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:72 +0xa5 May 30 14:46:56 vm node_exporter[412095]: goroutine 4236 [select]: May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc0002865a0) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:522 +0x9f1 May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus.Gatherers.Gather({0xc000231d60, 0x2, 0xc0000903c0?}) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:753 +0x166 May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus.(*noTransactionGatherer).Gather(0xee3?) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:1073 +0x22 May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1({0x7f34627c01b8, 0xc000228190}, 0xc00064a000) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/http.go:135 +0xfe May 30 14:46:56 vm node_exporter[412095]: net/http.HandlerFunc.ServeHTTP(0xce5480?, {0x7f34627c01b8?, 0xc000228190?}, 0xc000658180?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:2122 +0x2f May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7f34627c01b8, 0xc000228190}, 0xce8a00?) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:60 +0xd4 May 30 14:46:56 vm node_exporter[412095]: net/http.HandlerFunc.ServeHTTP(0xce8a10?, {0x7f34627c01b8?, 0xc000228190?}, 0x7f3489d88108?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:2122 +0x2f May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0xce8a10?, 0xc0001f8000?}, 0xc00064a000) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:147 +0xc5 May 30 14:46:56 vm node_exporter[412095]: net/http.HandlerFunc.ServeHTTP(0xc000228140?, {0xce8a10?, 0xc0001f8000?}, 0x2?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:2122 +0x2f May 30 14:46:56 vm node_exporter[412095]: main.(*handler).ServeHTTP(0xc000041b40, {0xce8a10, 0xc0001f8000}, 0xc00064a000) May 30 14:46:56 vm node_exporter[412095]: /app/node_exporter.go:82 +0x3f2 May 30 14:46:56 vm node_exporter[412095]: net/http.(*ServeMux).ServeHTTP(0x0?, {0xce8a10, 0xc0001f8000}, 0xc00064a000) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:2500 +0x149 May 30 14:46:56 vm node_exporter[412095]: net/http.serverHandler.ServeHTTP({0xc0002cc0f0?}, {0xce8a10, 0xc0001f8000}, 0xc00064a000) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:2936 +0x316 May 30 14:46:56 vm node_exporter[412095]: net/http.(*conn).serve(0xc0000ea1b0, {0xce8f70, 0xc0001ffa70}) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:1995 +0x612 May 30 14:46:56 vm node_exporter[412095]: created by net/http.(*Server).Serve May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:3089 +0x5ed May 30 14:46:56 vm node_exporter[412095]: goroutine 1664356 [chan receive]: May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Conn).send.func1() May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:560 +0x45 May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.(*Conn).send May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:559 +0x42c May 30 14:46:56 vm node_exporter[412095]: goroutine 1664245 [semacquire]: May 30 14:46:56 vm node_exporter[412095]: sync.runtime_Semacquire(0x0?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/sema.go:62 +0x27 May 30 14:46:56 vm node_exporter[412095]: sync.(*WaitGroup).Wait(0xc0003d5790?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/sync/waitgroup.go:116 +0x4b May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2() May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:472 +0x2f May 30 14:46:56 vm node_exporter[412095]: created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:471 +0x61b May 30 14:46:56 vm node_exporter[412095]: goroutine 1664305 [IO wait]: May 30 14:46:56 vm node_exporter[412095]: internal/poll.runtime_pollWait(0x7f34629ff5c8, 0x72) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/netpoll.go:306 +0x89 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).wait(0xc0005e7200?, 0xc0002b0480?, 0x0) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).waitRead(…) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*FD).ReadMsg(0xc0005e7200, {0xc0002b0480, 0x10, 0x10}, {0xc00001c620, 0x1000, 0x1000}, 0x70000c000066400?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_unix.go:304 +0x3aa May 30 14:46:56 vm node_exporter[412095]: net.(*netFD).readMsg(0xc0005e7200, {0xc0002b0480?, 0x10f5118?, 0xc0002b0388?}, {0xc00001c620?, 0xc0002b027e?, 0xc0002b0278?}, 0xc0002b0274?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/fd_posix.go:78 +0x37 May 30 14:46:56 vm node_exporter[412095]: net.(*UnixConn).readMsg(0xc0002124c8, {0xc0002b0480?, 0xc000310710?, 0xce2ec0?}, {0xc00001c620?, 0xcec200?, 0x11675f8?}) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/unixsock_posix.go:115 +0x4f May 30 14:46:56 vm node_exporter[412095]: net.(*UnixConn).ReadMsgUnix(0xc0002124c8, {0xc0002b0480?, 0xc000473d50?, 0x43bb60?}, {0xc00001c620?, 0x40dcea?, 0xc0001fd7b8?}) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/unixsock.go:143 +0x3c May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*oobReader).Read(0xc00001c600, {0xc0002b0480?, 0xc000473de8?, 0x40e0c7?}) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:21 +0x45 May 30 14:46:56 vm node_exporter[412095]: io.ReadAtLeast({0xce3480, 0xc00001c600}, {0xc0002b0480, 0x10, 0x10}, 0x10) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/io/io.go:332 +0x9a May 30 14:46:56 vm node_exporter[412095]: io.ReadFull(…) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/io/io.go:351 May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*unixTransport).ReadMessage(0xc00060cf60) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:91 +0x11e May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Conn).inWorker(0xc00016da40) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:390 +0x48 May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.(*Conn).Auth May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/auth.go:118 +0x99c May 30 14:46:56 vm node_exporter[412095]: goroutine 1664355 [chan receive]: May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Conn).send.func1() May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:560 +0x45 May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.(*Conn).send May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:559 +0x42c May 30 14:46:56 vm node_exporter[412095]: goroutine 1664314 [chan receive]: May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Object).CallWithContext(0xbca663?, {0xce8f00, 0xc000038048}, {0xbed119, 0x23}, 0x7?, {0xc000285ee0, 0x2, 0x2}) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/object.go:39 +0xdc May 30 14:46:56 vm node_exporter[412095]: github.com/coreos/go-systemd/v22/dbus.(*Conn).getProperty(0xc00027b880, {0xce8f00, 0xc000038048}, {0xc000400510, 0x10}, {0xc000320e00, 0x20}, {0xbca663, 0x4}) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/methods.go:310 +0x205 May 30 14:46:56 vm node_exporter[412095]: github.com/coreos/go-systemd/v22/dbus.(*Conn).GetUnitTypePropertyContext(…) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/methods.go:376 May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.(*systemdCollector).collectUnitStatusMetrics(0xc000133900, 0xc0003daf01?, 0x4fdf06?, {0xc00059a000?, 0xcb, 0x0?}) May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:274 +0x4d3 May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.(*systemdCollector).Update.func1() May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:219 +0x108 May 30 14:46:56 vm node_exporter[412095]: created by github.com/prometheus/node_exporter/collector.(*systemdCollector).Update May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:216 +0xcca May 30 14:46:56 vm node_exporter[412095]: goroutine 1664240 [IO wait]: May 30 14:46:56 vm node_exporter[412095]: internal/poll.runtime_pollWait(0x7f34629ff7a8, 0x72) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/netpoll.go:306 +0x89 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).wait(0xc00027a400?, 0xc0002cc101?, 0x0) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).waitRead(…) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*FD).Read(0xc00027a400, {0xc0002cc101, 0x1, 0x1}) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 May 30 14:46:56 vm node_exporter[412095]: net.(*netFD).Read(0xc00027a400, {0xc0002cc101?, 0xc0000419c0?, 0x10035a3c0?}) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/fd_posix.go:55 +0x29 May 30 14:46:56 vm node_exporter[412095]: net.(*conn).Read(0xc000212000, {0xc0002cc101?, 0x9e471d?, 0xbcf46c?}) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/net.go:183 +0x45 May 30 14:46:56 vm node_exporter[412095]: net/http.(*connReader).backgroundRead(0xc0002cc0f0) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:674 +0x3f May 30 14:46:56 vm node_exporter[412095]: created by net/http.(*connReader).startBackgroundRead May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/http/server.go:670 +0xca May 30 14:46:56 vm node_exporter[412095]: goroutine 1664304 [chan receive]: May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.newConn.func1() May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:303 +0x32 May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.newConn May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:302 +0x4f6 May 30 14:46:56 vm node_exporter[412095]: goroutine 1664246 [semacquire]: May 30 14:46:56 vm node_exporter[412095]: sync.runtime_Semacquire(0xc000659fb0?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/sema.go:62 +0x27 May 30 14:46:56 vm node_exporter[412095]: sync.(*WaitGroup).Wait(0xc0003d7e90?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/sync/waitgroup.go:116 +0x4b May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.NodeCollector.Collect({0xc000154c90?, {0xce3360?, 0xc0000419c0?}}, 0xc00042ee40) May 30 14:46:56 vm node_exporter[412095]: /app/collector/collector.go:156 +0x24f May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1() May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:455 +0x10d May 30 14:46:56 vm node_exporter[412095]: created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/registry.go:547 +0xc09 May 30 14:46:56 vm node_exporter[412095]: goroutine 1664307 [chan receive]: May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.newConn.func1() May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:303 +0x32 May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.newConn May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:302 +0x4f6 May 30 14:46:56 vm node_exporter[412095]: goroutine 1664308 [IO wait]: May 30 14:46:56 vm node_exporter[412095]: internal/poll.runtime_pollWait(0x7f34629ff4d8, 0x72) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/netpoll.go:306 +0x89 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).wait(0xc00027b280?, 0xc00026e5d0?, 0x0) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*pollDesc).waitRead(…) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:89 May 30 14:46:56 vm node_exporter[412095]: internal/poll.(*FD).ReadMsg(0xc00027b280, {0xc00026e5d0, 0x10, 0x10}, {0xc000426020, 0x1000, 0x1000}, 0x7000000000000c0?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/internal/poll/fd_unix.go:304 +0x3aa May 30 14:46:56 vm node_exporter[412095]: net.(*netFD).readMsg(0xc00027b280, {0xc00026e5d0?, 0x10f5118?, 0x11675f8?}, {0xc000426020?, 0xc00026e476?, 0xc00026e470?}, 0xc00026e46c?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/fd_posix.go:78 +0x37 May 30 14:46:56 vm node_exporter[412095]: net.(*UnixConn).readMsg(0xc000212d08, {0xc00026e5d0?, 0xc00038ad00?, 0x416d8b?}, {0xc000426020?, 0x423127?, 0xab7980?}) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/unixsock_posix.go:115 +0x4f May 30 14:46:56 vm node_exporter[412095]: net.(*UnixConn).ReadMsgUnix(0xc000212d08, {0xc00026e5d0?, 0xc000479d50?, 0x43bb60?}, {0xc000426020?, 0x40dcea?, 0xc000334e10?}) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/net/unixsock.go:143 +0x3c May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*oobReader).Read(0xc000426000, {0xc00026e5d0?, 0xc000479de8?, 0x40e0c7?}) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:21 +0x45 May 30 14:46:56 vm node_exporter[412095]: io.ReadAtLeast({0xce3480, 0xc000426000}, {0xc00026e5d0, 0x10, 0x10}, 0x10) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/io/io.go:332 +0x9a May 30 14:46:56 vm node_exporter[412095]: io.ReadFull(…) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/io/io.go:351 May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*unixTransport).ReadMessage(0xc000334228) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/transport_unix.go:91 +0x11e May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Conn).inWorker(0xc00016dc20) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/conn.go:390 +0x48 May 30 14:46:56 vm node_exporter[412095]: created by github.com/godbus/dbus/v5.(*Conn).Auth May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/auth.go:118 +0x99c May 30 14:46:56 vm node_exporter[412095]: goroutine 1664311 [chan receive]: May 30 14:46:56 vm node_exporter[412095]: github.com/coreos/go-systemd/v22/dbus.(*Conn).dispatch.func1() May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/subscription.go:55 +0x52 May 30 14:46:56 vm node_exporter[412095]: created by github.com/coreos/go-systemd/v22/dbus.(*Conn).dispatch May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/subscription.go:53 +0x9b May 30 14:46:56 vm node_exporter[412095]: goroutine 1664316 [chan receive]: May 30 14:46:56 vm node_exporter[412095]: github.com/godbus/dbus/v5.(*Object).CallWithContext(0xbd2627?, {0xce8f00, 0xc000038048}, {0xbed119, 0x23}, 0x6?, {0xc000285fc0, 0x2, 0x2}) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/godbus/dbus/v5@v5.1.0/object.go:39 +0xdc May 30 14:46:56 vm node_exporter[412095]: github.com/coreos/go-systemd/v22/dbus.(*Conn).getProperty(0xc00027b880, {0xce8f00, 0xc000038048}, {0xc000401780, 0xf}, {0xc000320e80, 0x1f}, {0xbd2627, 0xc}) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/methods.go:310 +0x205 May 30 14:46:56 vm node_exporter[412095]: github.com/coreos/go-systemd/v22/dbus.(*Conn).GetUnitTypePropertyContext(…) May 30 14:46:56 vm node_exporter[412095]: /go/pkg/mod/github.com/coreos/go-systemd/v22@v22.5.0/dbus/methods.go:376 May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.(*systemdCollector).collectSockets(0xc000133900, 0xce6c98?, 0xc00032f270?, {0xc00059a000?, 0xcb, 0xac6ce0?}) May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:326 +0x4ec May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.(*systemdCollector).Update.func5() May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:257 +0x108 May 30 14:46:56 vm node_exporter[412095]: created by github.com/prometheus/node_exporter/collector.(*systemdCollector).Update May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:254 +0x111b May 30 14:46:56 vm node_exporter[412095]: goroutine 1664270 [semacquire]: May 30 14:46:56 vm node_exporter[412095]: sync.runtime_Semacquire(0x8?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/runtime/sema.go:62 +0x27 May 30 14:46:56 vm node_exporter[412095]: sync.(*WaitGroup).Wait(0x1138300?) May 30 14:46:56 vm node_exporter[412095]: /usr/local/go/src/sync/waitgroup.go:116 +0x4b May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.(*systemdCollector).Update(0xc000133900, 0xc00042ee40) May 30 14:46:56 vm node_exporter[412095]: /app/collector/systemd_linux.go:267 +0x1369 May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.execute({0xbce40f, 0x7}, {0xce3e40, 0xc000133900}, 0xc000362fd0?, {0xce3360, 0xc0000419c0}) May 30 14:46:56 vm node_exporter[412095]: /app/collector/collector.go:161 +0x9c May 30 14:46:56 vm node_exporter[412095]: github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xbce40f?, 0xc0003016b0?}, {0xce3e40?, 0xc000133900?}) May 30 14:46:56 vm node_exporter[412095]: /app/collector/collector.go:152 +0x3d May 30 14:46:56 vm node_exporter[412095]: created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect May 30 14:46:56 vm node_exporter[412095]: /app/collector/collector.go:151 +0xd0 May 30 14:46:56 vm systemd[1]: node_exporter.service: Main process exited, code=exited, status=2/INVALIDARGUMENT May 30 14:46:56 vm systemd[1]: node_exporter.service: Failed with result ‘exit-code’.

Are you running node_exporter in Docker?

What did you do that produced an error?

Upgraded node_exporter yesterday from 1.5.0 > 1.6.0

What did you expect to see?

There was no fatal errors on 1.5.0 version

What did you see instead?