prometheus-operator: Unable to add a new service to be monitored with kube-prometheus

What did you do? Created a new servicemonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: sanguine-butterfly-traefik-dashboard
  labels:
    app: sanguine-butterfly-traefik
spec:
  selector:
    matchLabels:
      app: sanguine-butterfly-traefik
  endpoints:
  - port: 8080
    interval: 10s

What did you expect to see? I expected to see it appear in the prometheus dashboard under targets What did you see instead? Under which circumstances? Nothing changed Environment

Kubernetes version information: 1.8.1
Kubernetes cluster kind: kubespray
Manifests:

insert manifests relevant to the issue

Prometheus Operator Logs:

E1106 21:21:42.632764       1 reflector.go:201] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:278: Failed to list *v1.ServiceMonitor: json: cannot unmarshal number into Go struct field Endpoint.port of type string

after seeing the operator logs I changed port to -port: "8080" . Then the error went away but I still did not see my target show up.

New logs

Go struct field Endpoint.port of type string
E1106 21:21:42.632764       1 reflector.go:201] github.com/coreos/prometheus-operator/pkg/prometheus/operator.go:278: Failed to list *v1.ServiceMonitor: json: cannot unmarshal number into Go struct field Endpoint.port of type string
ts=2017-11-06T21:21:43Z caller=operator.go:670 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2017-11-06T21:21:43Z caller=operator.go:979 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2017-11-06T21:21:43Z caller=operator.go:670 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2017-11-06T21:21:44Z caller=operator.go:979 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2017-11-06T21:21:53Z caller=operator.go:321 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2017-11-06T21:21:53Z caller=operator.go:377 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main
ts=2017-11-06T21:23:20Z caller=operator.go:326 component=prometheusoperator msg="Prometheus updated" key=monitoring/k8s
ts=2017-11-06T21:23:20Z caller=operator.go:670 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2017-11-06T21:23:21Z caller=operator.go:979 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2017-11-06T21:23:23Z caller=operator.go:670 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2017-11-06T21:23:23Z caller=operator.go:979 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2017-11-06T21:23:23Z caller=operator.go:670 component=prometheusoperator msg="sync prometheus" key=monitoring/k8s
ts=2017-11-06T21:23:23Z caller=operator.go:979 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2017-11-06T21:23:36Z caller=operator.go:321 component=alertmanageroperator msg="Alertmanager updated" key=monitoring/main
ts=2017-11-06T21:23:36Z caller=operator.go:377 component=alertmanageroperator msg="sync alertmanager" key=monitoring/main

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 16 (5 by maintainers)

Most upvoted comments

If you want to match ALL service monitors without concern for their labels you can put this:

  serviceMonitorSelector: {}

If you don’t include the key at all you have to manually specify service monitors.

dobesv on Mar 8, 2018

Yeah once I got it in the config I was on the right track. I’m going to leave this open and write some docs from a fresh point of view.

shadycuz on Nov 7, 2017

The LabelSelector object doesn’t work that way unfortunately. ServiceMonitors are definitely encouraged to be used in that way. We are working on a way to make it a bit more flexible, but for the time being you either need to unify the labels on the ServiceMonitors or segregate as you described.

brancz on Dec 18, 2017

ServiceMonitor objects must be in the same namespace as the Prometheus object. The fact that your ServiceMonitor now shows up in the config is a good thing. That means it was selected and parsed by the Prometheus Operator. The namespaceSelector is in regard to the Service objects, that a ServiceMonitor selects. Basically the point you are at is: your Prometheus instance is running, your ServiceMonitor is selected, but doesn’t select the Service yet that you expect.

As a side note (but you probably already figured this out):

serviceMonitorSelector:
    matchExpressions:
    - {key: k8s-app, operator: Exists}
    - {key: app, operator: In, values: [sanguine-butterfly-traefik]}

The entries in the matchExpressions list are connected with a logical AND, not an OR. Therefore, what the above expects is, that the Service objects have a k8s-app label, as well as a app=sanguine-butterfly-traefik label.

brancz on Nov 7, 2017

I was facing the same issue of it showing up in the prometheus configs but not showing up in targets (after going through some digging about that serviceMonitorSelectors too heh).

My service is in the default namespace and it wouldn’t show up if I used any: true in the namespaceSelector as such:

  namespaceSelector:
    any: true

And it correctly shows up in this case:

  namespaceSelector:
    matchNames:
    - default

So if anyone else is having similar issues, you could try that out too.

aranair on Feb 23, 2018

@brancz Is there a way to have a logical OR in the serviceMonitorSelector. For instance I’d like to have the it match:

matchExpressions:
    - {key: k8s-app, operator: Exists}

But I’d also like it to match the labale monitor: true (or whatever other label). Is there a way to combine matchExpression and matchLabels?

EDIT: or is this not the idea behind operator. Should I just create a new prometheus instance for different “namespaces/teams” ? One for monitoring the cluster, one for “teamA” apps, one for “teamB” apps etc?

denibertovic on Dec 18, 2017