telegraf: Prometneus_client error in telegraf 0.2.3

Hello! I’ve updated to version 0.2.3 Telegraph, but the problem has remained. Telegraph falls as before

2015/12/01 09:34:19 Attempting connection to output: influxdb
2015/12/01 09:34:19 Successfully connected to output: influxdb
2015/12/01 09:34:19 Attempting connection to output: prometheus_client
2015/12/01 09:34:19 Successfully connected to output: prometheus_client
2015/12/01 09:34:19 Starting Telegraf (version 0.2.3)
2015/12/01 09:34:19 Loaded outputs: influxdb prometheus_client
2015/12/01 09:34:19 Loaded plugins: disk io mem swap system cpu
2015/12/01 09:34:19 Tags enabled: host=influxdb.uiscom.ru
2015/12/01 09:34:19 Agent Config: Interval:10s, Debug:true, Hostname:"influxdb", Flush Interval:12.762447502s
2015/12/01 09:34:30 Gathered metrics, (10s interval), from 6 plugins in 12.037977ms
panic: inconsistent label cardinality

goroutine 58 [running]:
github.com/prometheus/client_golang/prometheus.(*MetricVec).With(0xc8201c2370, 0xc8206d8c60, 0x0, 0x0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/Godeps/_workspace/src/github.com/prometheus/client_golang/prometheus/vec.go:137 +0x91
github.com/prometheus/client_golang/prometheus.(*UntypedVec).With(0xc8201c2370, 0xc8206d8c60, 0x0, 0x0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/Godeps/_workspace/src/github.com/prometheus/client_golang/prometheus/untyped.go:120 +0x42
github.com/influxdb/telegraf/outputs/prometheus_client.(*PrometheusClient).Write(0xc820232da0, 0xc820363000, 0x1ee, 0x200, 0x0, 0x0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/outputs/prometheus_client/prometheus_client.go:96 +0x978
github.com/influxdb/telegraf.(*Agent).writeOutput(0xc8201081a0, 0xc820363000, 0x1ee, 0x200, 0xc820232dc0, 0xc82023c2a0, 0xc820010ee0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:229 +0x11d
created by github.com/influxdb/telegraf.(*Agent).flush
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:269 +0x11a

goroutine 1 [select]:
github.com/influxdb/telegraf.(*Agent).Run(0xc8201081a0, 0xc82023c2a0, 0x0, 0x0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:385 +0x9d2
main.main()
        /home/telebuild/go/src/github.com/influxdb/telegraf/cmd/telegraf/telegraf.go:151 +0x1832

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall]:
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 38 [IO wait]:
net.runtime_pollWait(0x7f3af8346e88, 0x72, 0xc820010200)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8201e7db0, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201e7db0, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8201e7d50, 0x0, 0x7f3af8347130, 0xc82031f060)
        /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc8201081e0, 0x452c60, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc8201081e0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:2135 +0x41
net/http.(*Server).Serve(0xc82023c240, 0x7f3af83470f8, 0xc8201081e0, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1887 +0xb3
net/http.(*Server).ListenAndServe(0xc82023c240, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:1877 +0x136
created by github.com/influxdb/telegraf/outputs/prometheus_client.(*PrometheusClient).Start
        /home/telebuild/go/src/github.com/influxdb/telegraf/outputs/prometheus_client/prometheus_client.go:33 +0x14a

goroutine 57 [select]:
net/http.(*persistConn).roundTrip(0xc8201a0630, 0xc8203101b0, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc8201753b0, 0xc8202f41c0, 0xa8, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:235 +0x530
net/http.send(0xc8202f41c0, 0x7f3afa98c4d8, 0xc8201753b0, 0xc82017af00, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xc8201a3ec0, 0xc8202f41c0, 0x44, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xc8201a3ec0, 0xc8202f41c0, 0x1097a40, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0xc8201a3ec0, 0xc8202f41c0, 0x28, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:178 +0x188
github.com/influxdb/influxdb/client/v2.(*client).Write(0xc8201a7a40, 0x7f3af80f87b8, 0xc82031cf00, 0x0, 0x0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/Godeps/_workspace/src/github.com/influxdb/influxdb/client/v2/client.go:382 +0x8e5
github.com/influxdb/telegraf/outputs/influxdb.(*InfluxDB).Write(0xc820129cc0, 0xc820363000, 0x1ee, 0x200, 0x0, 0x0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/outputs/influxdb/influxdb.go:146 +0x3aa
github.com/influxdb/telegraf.(*Agent).writeOutput(0xc8201081a0, 0xc820363000, 0x1ee, 0x200, 0xc820232d80, 0xc82023c2a0, 0xc820010ee0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:229 +0x11d
created by github.com/influxdb/telegraf.(*Agent).flush
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:269 +0x11a

goroutine 24 [select]:
github.com/influxdb/telegraf.(*Agent).flusher(0xc8201081a0, 0xc82023c2a0, 0xc82023c480, 0x0, 0x0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:286 +0x44f
github.com/influxdb/telegraf.(*Agent).Run.func1(0xc8200e07d0, 0xc8201081a0, 0xc82023c2a0, 0xc82023c480)
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:346 +0x7f
created by github.com/influxdb/telegraf.(*Agent).Run
        /home/telebuild/go/src/github.com/influxdb/telegraf/agent.go:350 +0x469

goroutine 39 [select, locked to thread]:
runtime.gopark(0x1097e78, 0xc820296728, 0xeca0c8, 0x6, 0x18, 0x2)
        /usr/local/go/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc820296728, 0x0, 0x18)
        /usr/local/go/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc820296728)
        /usr/local/go/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 40 [chan receive]:
main.main.func1(0xc82023c300, 0xc82023c2a0)
        /home/telebuild/go/src/github.com/influxdb/telegraf/cmd/telegraf/telegraf.go:131 +0x36
created by main.main
        /home/telebuild/go/src/github.com/influxdb/telegraf/cmd/telegraf/telegraf.go:133 +0x1117

goroutine 94 [select]:
net/http.(*persistConn).writeLoop(0xc8201a0630)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 93 [IO wait]:
net.runtime_pollWait(0x7f3af8346dc8, 0x72, 0xc820010200)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82017d250, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82017d250, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82017d1f0, 0xc82022d000, 0x1000, 0x1000, 0x0, 0x7f3afa988050, 0xc820010200)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc82035a168, 0xc82022d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f3af8346f48, 0xc82035a168, 0xc8201a0688, 0xc82022d000, 0x1000, 0x1000, 0xc82030ca20, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc8201c1dc0, 0xc82022d000, 0x1000, 0x1000, 0xc82038e010, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc82030c900)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82030c900, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8201a0630)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

About this issue

Original URL
State: closed
Created 9 years ago
Comments: 27 (21 by maintainers)

Commits related to this issue

Don't use panic-happy prometheus client With() function Deals with part of #405, although it doesn't deal with the underlying label error that is occuring. — committed to influxdata/telegraf by sparrc 9 years ago
Don't use panic-happy prometheus client With() function Deals with part of #405, although it doesn't deal with the underlying label error that is occuring. — committed to influxdata/telegraf by sparrc 9 years ago
io plugin, add an 'unknown' tag when the serial number can't be found closes #405 — committed to influxdata/telegraf by sparrc 9 years ago
io plugin, add an 'unknown' tag when the serial number can't be found closes #405 — committed to influxdata/telegraf by sparrc 9 years ago
io plugin, add an 'unknown' tag when the serial number can't be found closes #405 — committed to influxdata/telegraf by sparrc 9 years ago

Most upvoted comments

This means that some metric has say two labels sometimes and three at other times, which is not allowed (it’s an anti-pattern). I’d guess that there’s a clash on some metric name.

brian-brazil on Dec 1, 2015

This actually appears to be a different problem. The previous panic was because of a NaN value trying to get sent to InfluxDB. This panic is coming from the prometheus client code here: https://github.com/prometheus/client_golang/blob/master/prometheus/vec.go#L143

It is getting an error on this function call: https://godoc.org/github.com/prometheus/client_golang/prometheus#MetricVec.GetMetricWith, which states:

An error is returned if the number and names of the Labels are inconsistent with those 
of the VariableLabels in Desc.

This doesn’t really mean anything to me as I’m not familiar with Prometheus, does it mean anything to you? Do you know why this function would be returning an error?

sparrc on Dec 1, 2015