prometheus-operator: ServiceMonitor not being scraped

What did you do? I have deployed a service with a service monitor so that it can be picked up and scraped by prometheus. Here is the deployed servicemonitor

Name:         skipper-metrics
Namespace:    monitoring
Labels:       app=skipper
              chart=skipper-0.1.0
              heritage=Tiller
              prometheus=kube-prometheus
              release=skipper
Annotations:  <none>
API Version:  monitoring.coreos.com/v1
Kind:         ServiceMonitor
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-06-20T10:09:27Z
  Generation:          1
  Resource Version:    1371114
  Self Link:           /apis/monitoring.coreos.com/v1/namespaces/monitoring/servicemonitors/skipper-metrics
  UID:                 03f5c9b1-7472-11e8-bdcf-42010a8a00b3
Spec:
  Endpoints:
    Interval:  30s
    Path:      /metrics
    Port:      skipper-metrics
  Job Label:   skipper-ingress
  Namespace Selector:
    Match Names:
      snab
  Selector:
    App:        skipper
    Chart:      skipper-0.1.0
    Component:  ingress
    Heritage:   Tiller
    Release:    skipper
Events:         <none>

What did you expect to see? I expected to see this service monitor configured in http://localhost:9090/config but that was not the case. I also expected to see references to the service monitor in the logs of the prometheus-operator pod.

What did you see instead? Under which circumstances? Basically, no metrics are being collected.

Environment GKE v1.10.4-gke.0

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.4-gke.0", GitCommit:"c08fe9548257472151a8fa298bd6c3e9ea1b8c09", GitTreeState:"clean", BuildDate:"2018-06-06T11:34:07Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind: It’ a google managed service

• Manifests:

Name:         skipper-metrics
Namespace:    monitoring
Labels:       app=skipper
              chart=skipper-0.1.0
              heritage=Tiller
              prometheus=kube-prometheus
              release=skipper
Annotations:  <none>
API Version:  monitoring.coreos.com/v1
Kind:         ServiceMonitor
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-06-20T10:09:27Z
  Generation:          1
  Resource Version:    1371114
  Self Link:           /apis/monitoring.coreos.com/v1/namespaces/monitoring/servicemonitors/skipper-metrics
  UID:                 03f5c9b1-7472-11e8-bdcf-42010a8a00b3
Spec:
  Endpoints:
    Interval:  30s
    Path:      /metrics
    Port:      skipper-metrics
  Job Label:   skipper-ingress
  Namespace Selector:
    Match Names:
      snab
  Selector:
    App:        skipper
    Chart:      skipper-0.1.0
    Component:  ingress
    Heritage:   Tiller
    Release:    skipper
Events:         <none>

• Prometheus Operator Logs:

Here is the log, which shows that the service monitor is not being serviced

level=info ts=2018-06-26T23:09:48.395757161Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:10:14.231551778Z caller=operator.go:396 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:10:20.301123217Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:10:20.403000277Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:10:20.595862229Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:10:20.686702629Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:10:21.307419569Z caller=operator.go:396 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:10:26.643415702Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:11:11.560520105Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:11:11.654841649Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:11:22.042365367Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:11:22.12900519Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:11:41.015135657Z caller=operator.go:396 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:12:14.312737947Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:12:41.698095722Z caller=operator.go:396 component=alertmanageroperator msg="sync alertmanager" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:12:59.560052667Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:12:59.652620352Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:13:07.249862817Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus
level=info ts=2018-06-26T23:13:07.490789908Z caller=operator.go:731 component=prometheusoperator msg="sync prometheus" key=monitoring/kube-prometheus