smokeping_prober: panic: Value kind is Duration, not Time

We’ve got a smokeping_prober docker container that is dying with this error message. Here’s one recent example, with hostnames and IP addresses changed to protect the innocent:

2025-04-03T15:50:34.021208361Z time=2025-04-03T15:50:34.021Z level=INFO source=main.go:228 msg="Starting smokeping_prober" version="(version=0.9.0, branch=HEAD, revision=1c1bd4704bfa7254e898025907a7582c6df284e1)"
2025-04-03T15:50:34.021267761Z time=2025-04-03T15:50:34.021Z level=INFO source=main.go:229 msg="Build context" build_context="(go=go1.23.5, platform=linux/amd64, user=root@c3ae5db9db58, date=20250121-16:38:33, tags=unknown)"
2025-04-03T15:50:34.021596945Z time=2025-04-03T15:50:34.021Z level=INFO source=main.go:237 msg="Leaving GOMAXPROCS=8: CPU quota undefined" component=automaxprocs
2025-04-03T15:50:34.029490512Z time=2025-04-03T15:50:34.029Z level=INFO source=main.go:109 msg="Starting prober" address=localhost interval=1s size_bytes=56 source_address=127.0.0.1
2025-04-03T15:50:34.079939153Z time=2025-04-03T15:50:34.079Z level=INFO source=main.go:109 msg="Starting prober" address=192.168.10.20 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.130627987Z time=2025-04-03T15:50:34.130Z level=INFO source=main.go:109 msg="Starting prober" address=192.168.10.30 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.180875477Z time=2025-04-03T15:50:34.180Z level=INFO source=main.go:109 msg="Starting prober" address=host1.example.com interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.231518661Z time=2025-04-03T15:50:34.231Z level=INFO source=main.go:109 msg="Starting prober" address=172.16.10.20 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.281877042Z time=2025-04-03T15:50:34.281Z level=INFO source=main.go:109 msg="Starting prober" address=172.16.10.30 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.332441354Z time=2025-04-03T15:50:34.332Z level=INFO source=main.go:109 msg="Starting prober" address=host2.example.com interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.382761425Z time=2025-04-03T15:50:34.382Z level=INFO source=main.go:109 msg="Starting prober" address=192.168.100.200 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.432803963Z time=2025-04-03T15:50:34.432Z level=INFO source=main.go:109 msg="Starting prober" address=192.168.100.300 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.483868690Z time=2025-04-03T15:50:34.483Z level=INFO source=main.go:109 msg="Starting prober" address=host3.example.com interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.534916428Z time=2025-04-03T15:50:34.534Z level=INFO source=main.go:109 msg="Starting prober" address=172.16.20.10 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.585506941Z time=2025-04-03T15:50:34.585Z level=INFO source=main.go:109 msg="Starting prober" address=172.16.30.40 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.635850101Z time=2025-04-03T15:50:34.635Z level=INFO source=main.go:109 msg="Starting prober" address=172.16.40.10 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.686441674Z time=2025-04-03T15:50:34.686Z level=INFO source=main.go:109 msg="Starting prober" address=host4.example.com interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.737325021Z time=2025-04-03T15:50:34.737Z level=INFO source=main.go:109 msg="Starting prober" address=10.10.20.30 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.787258637Z time=2025-04-03T15:50:34.787Z level=INFO source=main.go:109 msg="Starting prober" address=host1.example.net interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.838254225Z time=2025-04-03T15:50:34.838Z level=INFO source=main.go:109 msg="Starting prober" address=10.10.20.40 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.888978009Z time=2025-04-03T15:50:34.888Z level=INFO source=main.go:109 msg="Starting prober" address=host2.example.net interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.939711914Z time=2025-04-03T15:50:34.939Z level=INFO source=main.go:109 msg="Starting prober" address=10.10.20.50 interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:34.990631470Z time=2025-04-03T15:50:34.990Z level=INFO source=main.go:109 msg="Starting prober" address=host3.example.net interval=1s size_bytes=56 source_address=""
2025-04-03T15:50:35.042488595Z time=2025-04-03T15:50:35.042Z level=INFO source=tls_config.go:347 msg="Listening on" address=[::]:9374
2025-04-03T15:50:35.042514496Z time=2025-04-03T15:50:35.042Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9374
2025-04-03T15:50:35.083051411Z panic: Value kind is Duration, not Time
2025-04-03T15:50:35.083095901Z 
2025-04-03T15:50:35.083107531Z goroutine 55 [running]:
2025-04-03T15:50:35.083119391Z log/slog.Value.Time({{}, 0xc0002be3c0?, {0x9e8c00?, 0xb96fb8?}})
2025-04-03T15:50:35.083128282Z 	/usr/local/go/src/log/slog/value.go:379 +0x168
2025-04-03T15:50:35.083137802Z github.com/prometheus/common/promslog.init.func2({0xc0000f7608?, 0x9e8c00?, 0xb96fb8?}, {{0xaad504, 0x4}, {{}, 0x2a76b, {0x9e8c00, 0xb96fb8}}})
2025-04-03T15:50:35.083145952Z 	/go/pkg/mod/github.com/prometheus/common@v0.62.0/promslog/slog.go:75 +0x90
2025-04-03T15:50:35.083153832Z log/slog.(*handleState).appendAttr(0xc0000f7608, {{0xaad504, 0x4}, {{}, 0x2a76b, {0x9e8c00, 0xb96fb8}}})
2025-04-03T15:50:35.083162512Z 	/usr/local/go/src/log/slog/handler.go:475 +0x21e
2025-04-03T15:50:35.083170562Z log/slog.(*handleState).appendNonBuiltIns.func1({{0xaad504, 0x4}, {{}, 0x2a76b, {0x9e8c00, 0xb96fb8}}})
2025-04-03T15:50:35.083204082Z 	/usr/local/go/src/log/slog/handler.go:342 +0x52
2025-04-03T15:50:35.083237073Z log/slog.Record.Attrs({{0xc1f3c972c4d096a6, 0x3f4c2681, 0xfcbf40}, {0xab26e5, 0xa}, 0xfffffffffffffffc, 0x97eba7, {{{0xaafa68, 0x7}, {{...}, ...}}, ...}, ...}, ...)
2025-04-03T15:50:35.083266373Z 	/usr/local/go/src/log/slog/record.go:84 +0xaa
2025-04-03T15:50:35.083313023Z log/slog.(*handleState).appendNonBuiltIns(_, {{0xc1f3c972c4d096a6, 0x3f4c2681, 0xfcbf40}, {0xab26e5, 0xa}, 0xfffffffffffffffc, 0x97eba7, {{{0xaafa68, 0x7}, ...}, ...}, ...})
2025-04-03T15:50:35.083337844Z 	/usr/local/go/src/log/slog/handler.go:341 +0x4c7
2025-04-03T15:50:35.083372584Z log/slog.(*commonHandler).handle(_, {{0xc1f3c972c4d096a6, 0x3f4c2681, 0xfcbf40}, {0xab26e5, 0xa}, 0xfffffffffffffffc, 0x97eba7, {{{0xaafa68, 0x7}, ...}, ...}, ...})
2025-04-03T15:50:35.083392854Z 	/usr/local/go/src/log/slog/handler.go:308 +0x5a5
2025-04-03T15:50:35.083424304Z log/slog.(*TextHandler).Handle(_, {_, _}, {{0xc1f3c972c4d096a6, 0x3f4c2681, 0xfcbf40}, {0xab26e5, 0xa}, 0xfffffffffffffffc, 0x97eba7, ...})
2025-04-03T15:50:35.083437644Z 	/usr/local/go/src/log/slog/text_handler.go:95 +0x3f
2025-04-03T15:50:35.083469355Z log/slog.(*Logger).log(0xc000028600, {0xba1d90, 0xfee3a0}, 0xfffffffffffffffc, {0xab26e5, 0xa}, {0xc0001d2c20, 0xa, 0xa})
2025-04-03T15:50:35.083486045Z 	/usr/local/go/src/log/slog/logger.go:257 +0x1e4
2025-04-03T15:50:35.083494345Z log/slog.(*Logger).Debug(...)
2025-04-03T15:50:35.083511505Z 	/usr/local/go/src/log/slog/logger.go:200
2025-04-03T15:50:35.083523865Z main.(*SmokepingCollector).updatePingers.func1(0xc000034d80)
2025-04-03T15:50:35.083534755Z 	/app/collector.go:121 +0x3ad
2025-04-03T15:50:35.083545105Z github.com/prometheus-community/pro-bing.(*Pinger).processPacket(0xc0002563c0, 0xc000034d40)
2025-04-03T15:50:35.083555686Z 	/go/pkg/mod/github.com/prometheus-community/pro-bing@v0.6.0/ping.go:870 +0x58f
2025-04-03T15:50:35.083565706Z github.com/prometheus-community/pro-bing.(*Pinger).runLoop(0xc0002563c0, {0xba7a80, 0xc0001866a8}, 0xc0001beee0)
2025-04-03T15:50:35.083576386Z 	/go/pkg/mod/github.com/prometheus-community/pro-bing@v0.6.0/ping.go:630 +0x2f3
2025-04-03T15:50:35.083586596Z github.com/prometheus-community/pro-bing.(*Pinger).run.func3()
2025-04-03T15:50:35.083596636Z 	/go/pkg/mod/github.com/prometheus-community/pro-bing@v0.6.0/ping.go:595 +0x59
2025-04-03T15:50:35.083606846Z golang.org/x/sync/errgroup.(*Group).Go.func1()
2025-04-03T15:50:35.083618406Z 	/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50
2025-04-03T15:50:35.083629116Z created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 52
2025-04-03T15:50:35.083639156Z 	/go/pkg/mod/golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (3 by maintainers)

Commits related to this issue

Most upvoted comments

latest Will be updated when I cut a new release. You can pin to a specific master tag like quay.io/superq/smokeping-prober@sha256:8d55886e44e2051f018f756e1642ae312aa8c21f2da801636507b625d283988f as a temporary workaround.

Yea, time is the problem because it’s a conflicting key with the time in the log line. This was fixed in the upstream Prometheus logging wrapper. With the latest prometheus/commmon, the logs look like this:

time=2025-04-03T20:10:27.360Z level=DEBUG source=collector.go:123 msg="Echo reply" ip_addr=127.0.0.1 bytes_received=64 icmp_seq=2 logged_time=515.787µs ttl=64
time=2025-04-03T20:10:28.360Z level=DEBUG source=collector.go:123 msg="Echo reply" ip_addr=127.0.0.1 bytes_received=64 icmp_seq=3 logged_time=430.446µs ttl=64

@bknowles deferring to @SuperQ here, but it seems like some combination of issues from the adoption of slog is the issue, or perhaps, the inheritance of slog from the prometheus common packages?

fwiw, as far as I know, slog expects “time” fields in a log to be of type Time.time

https://pkg.go.dev/golang.org/x/exp/slog#pkg-constants

	// TimeKey is the key used by the built-in handlers for the time
	// when the log method is called. The associated Value is a [time.Time].
	TimeKey = "time"

However, here it’s being called with pkt.Rtt

https://github.com/SuperQ/smokeping_prober/blob/4dd0082212607983f84885d7e64803a75650f8b1/collector.go#L123-L125

I believe pkt.Rtt in this case is time.Duration https://github.com/prometheus-community/pro-bing/blob/c9b2c133ccf6c6212f29ca33c6258f7400ea76f6/ping.go#L257-L259

Hope that helps. appreciate all the work y’all do here @SuperQ to keep things running smoothly and w/prom in general!