prometheus: Cannot scrape targets specified by mDNS name

Can I poke on this issue? It was opened on March/2017 and the last comment was December/2021 and while there’s been some interesting discussion, it has not led to any resolving of the situation-at-hand.

I would like to point out that in this day-n-age, one would expect an MDNS based hostname (RFC6763 etc) to be accepted by ALL software running on a modern-day operating system. If the host(1) command shows resolution of a hostname, then similar resolution should be provided. Sadly, most of the discussion here is all about GoLang’s purity vs actually solving @moio’s initial (five and half year old) issue.

While I know the right thing for me to do it to buck up and send in a PR; sadly, I’m a C & Python kinda person and GoLang isn’t my forte. But I am someone that’s used Prometheus for a long time and was shocked when I spun Prometheus up at home to do some testing and found it would not cooperate across my two simply configured hosts within my .local DNS range. I was basically shocked (until I read this, then I was further shocked).

Summary: If all other system s/w can resolve a host in the .local domain, so should Prometheus.

Arguably it’s by far the most common SD mechanism. Not in your typical server infrastructure but using Prometheus in a home network/IoT is IMO a reasonable use case.

@msiebuhr I fully agree with this. I don’t know why proprietary, vendor specific SD mechanisms for things like triton or azure are in scope while the most widely deployed open mechanism isn’t.

I know that @SuperQ would also like to see mDNS support. So if we can find lazy consensus here, I’d probably call in a official vote on this.

@msiebuhr What were the main reasons to close the PR? I’d actually love to see this in Prometheus, with the influx of people using Prometheus for IoT use cases etc.

This bug—which is specifically about resolving .local domain names (that come from somewhere else, such as static_configs), not using mDNS for service discovery—seems to have been fixed by golang/go#16739. It should probably be closed

According to my understanding this is not the case - the Go issue above is not really about implementing mDNS resolution in the pure Go resolver.

Reproducer (main.go):

package main

import "fmt"
import "net"

func main() {
	name, err := net.LookupHost("prometheus.local")
	if err == nil {
		fmt.Println(name[0])
	} else {
		fmt.Println(err.Error())
	}
}

On the commandline:

> go version
go version go1.12.17 linux/amd64

> GODEBUG=netdns=cgo go run ./resolve.go
192.168.187.15

> GODEBUG=netdns=go go run ./resolve.go
lookup prometheus.local on 192.168.187.1:53: no such host

So to me the original description of this issue is still valid.

It drags in quite a few large dependencies (low-level networking to get mDNS working), and I was told in private conversation (perhaps @brian-brazil ; cannot remember) that new service discovery mechanism preferably should be built outside core prometheus.

Edit: Just re-read @brian-brazil mention a moratorium on new SD’s above. That may be where I got it from…

But yes, I generally find it weird (from a hobbyist/IoT point-of-view) that Prometheus comes with complex discovery mechanisms for some really big systems and everyone else must implement some custom cron/daemon thing poking at file_sd.

We continue to have a moratorium on new SD mechanisms, and once the example file_sd code is in the plan is to close all existing feature requests and point to that instead.

SDs take a lot of work to maintain, and history shows that there’s little interest in doing so.

We support DNS, with would require specific mDNS support to be added.