prometheus-operator: Operator doesn't find corresponding target services to servicemonitor

What did you do? want to monitor postgres service and installed kubedb and the needed resources, jet the operator doesn’t get the tragets What did you expect to see? targets appended to the target list What did you see instead? Under which circumstances? operator didn’t get the targets. I’m guessing it’s because of missing labels. but i can’t find the mistake Environment minikube v0.28.2

Prometheus Operator version:

Image:         quay.io/coreos/prometheus-operator:v0.23.0
Args:
      --kubelet-service=kube-system/kubelet
      --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1
      --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.23.0

Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:14:26Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind:

minikube start --kubernetes-version=v1.10.1 --bootstrapper=kubeadm --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.address=0.0.0.0 --extra-config=controller-manager.address=0.0.0.0 --extra-config=apiserver.authorization-mode=RBAC --memory 10240 --cpus 4

Manifests:

$ kubectl describe pod prometheus-k8s-0 -n monitoring
Name:           prometheus-k8s-0
Namespace:      monitoring
Node:           minikube/10.0.2.15
Start Time:     Mon, 13 Aug 2018 16:21:14 +0200
Labels:         app=prometheus
                controller-revision-hash=prometheus-k8s-869f95c64c
                prometheus=k8s
                statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:    <none>
Status:         Running
IP:             172.17.0.16
Controlled By:  StatefulSet/prometheus-k8s
Containers:
  prometheus:
    Container ID:  docker://46c7551a50a1ad734a9fef704f92495d6275c460db24350c808fc62bcec8a0f8
    Image:         quay.io/prometheus/prometheus:v2.3.2
    Image ID:      docker-pullable://quay.io/prometheus/prometheus@sha256:7384a79f4b4991bf8269e7452390249b7c70bcdd10509c8c1c6c6e30e32fb324
    Port:          9090/TCP
    Host Port:     0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --storage.tsdb.retention=24h
      --web.enable-lifecycle
      --storage.tsdb.no-lockfile
      --web.route-prefix=/
    State:          Running
      Started:      Tue, 14 Aug 2018 08:09:27 +0200
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 13 Aug 2018 21:31:47 +0200
      Finished:     Mon, 13 Aug 2018 22:13:46 +0200
    Ready:          True
    Restart Count:  4
    Requests:
      memory:     400Mi
    Liveness:     http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120
    Environment:  <none>
    Mounts:
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /prometheus from prometheus-k8s-db (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from prometheus-k8s-token-7gt98 (ro)
  prometheus-config-reloader:
    Container ID:  docker://1ee097a2d3f7fdb6bd3bccacecddfbd79ea90df41f7c2e553fc5937ab256d24f
    Image:         quay.io/coreos/prometheus-config-reloader:v0.23.0
    Image ID:      docker-pullable://quay.io/coreos/prometheus-config-reloader@sha256:c7229ef9fb172ad15eb096d652f37badc49acea7080328a02a052a1ee343f998
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/prometheus-config-reloader
    Args:
      --log-format=logfmt
      --reload-url=http://localhost:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
    State:          Running
      Started:      Tue, 14 Aug 2018 08:09:28 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Mon, 13 Aug 2018 21:31:48 +0200
      Finished:     Mon, 13 Aug 2018 22:13:46 +0200
    Ready:          True
    Restart Count:  3
    Limits:
      cpu:     10m
      memory:  50Mi
    Requests:
      cpu:     10m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from prometheus-k8s-token-7gt98 (ro)
  rules-configmap-reloader:
    Container ID:  docker://f1a9a5ff50647e112a83fec5bfd7465df0675c70d2dbc577a76413dcbdcc684d
    Image:         quay.io/coreos/configmap-reload:v0.0.1
    Image ID:      docker-pullable://quay.io/coreos/configmap-reload@sha256:e2fd60ff0ae4500a75b80ebaa30e0e7deba9ad107833e8ca53f0047c42c5a057
    Port:          <none>
    Host Port:     <none>
    Args:
      --webhook-url=http://localhost:9090/-/reload
      --volume-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    State:          Running
      Started:      Tue, 14 Aug 2018 08:09:29 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Mon, 13 Aug 2018 21:31:49 +0200
      Finished:     Mon, 13 Aug 2018 22:13:46 +0200
    Ready:          True
    Restart Count:  3
    Limits:
      cpu:     5m
      memory:  10Mi
    Requests:
      cpu:        5m
      memory:     10Mi
    Environment:  <none>
    Mounts:
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from prometheus-k8s-token-7gt98 (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s
    Optional:    false
  config-out:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional:  false
  prometheus-k8s-db:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  prometheus-k8s-token-7gt98:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-token-7gt98
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

$ kubectl describe pod prometheus-operator-bf54b8564-hgrv7 -n monitoring
Name:           prometheus-operator-bf54b8564-hgrv7
Namespace:      monitoring
Node:           minikube/10.0.2.15
Start Time:     Mon, 13 Aug 2018 16:21:32 +0200
Labels:         k8s-app=prometheus-operator
                operator=prometheus
                pod-template-hash=691064120
Annotations:    <none>
Status:         Running
IP:             172.17.0.13
Controlled By:  ReplicaSet/prometheus-operator-bf54b8564
Containers:
  prometheus-operator:
    Container ID:  docker://f8bbe2bc01b921c66f5808aef3b7ac9ab179268eb31366e1dcab957f5bdd7e1d
    Image:         quay.io/coreos/prometheus-operator:v0.23.0
    Image ID:      docker-pullable://quay.io/coreos/prometheus-operator@sha256:56185c9a4553b452fe620aa0edd1622ef7b677c6c9752953a4bfa949133b75f4
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      --kubelet-service=kube-system/kubelet
      --config-reloader-image=quay.io/coreos/configmap-reload:v0.0.1
      --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.23.0
    State:          Running
      Started:      Tue, 14 Aug 2018 08:10:11 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 14 Aug 2018 08:09:25 +0200
      Finished:     Tue, 14 Aug 2018 08:09:56 +0200
    Ready:          True
    Restart Count:  8
    Limits:
      cpu:     200m
      memory:  100Mi
    Requests:
      cpu:        100m
      memory:     50Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from prometheus-operator-token-bvztb (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  prometheus-operator-token-bvztb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-operator-token-bvztb
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

$ kubectl describe servicemonitors.monitoring.coreos.com kubedb-postgres-hot-postgres -n monitoring
Name:         kubedb-postgres-hot-postgres
Namespace:    monitoring
Labels:       k8s-app=kubedb
              monitoring.appscode.com/service=hot-postgres.postgres
Annotations:  <none>
API Version:  monitoring.coreos.com/v1
Kind:         ServiceMonitor
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-08-14T09:29:16Z
  Generation:          1
  Resource Version:    154394
  Self Link:           /apis/monitoring.coreos.com/v1/namespaces/monitoring/servicemonitors/kubedb-postgres-hot-postgres
  UID:                 83c6cc09-9fa4-11e8-b639-08002736c3be
Spec:
  Endpoints:
    Interval:     30s
    Path:         /kubedb.com/v1alpha1/namespaces/postgres/postgreses/hot-postgres/metrics
    Port:         prom-http
    Target Port:  0
  Namespace Selector:
    Match Names:
      postgres
  Selector:
    Match Labels:
      Kubedb . Com / Kind:  Postgres
      Kubedb . Com / Name:  hot-postgres
Events:

$ kubectl describe svc hot-postgres -n postgres
Name:              hot-postgres
Namespace:         postgres
Labels:            kubedb.com/kind=Postgres
                   kubedb.com/name=hot-postgres
Annotations:       monitoring.appscode.com/agent=prometheus.io/coreos-operator
Selector:          kubedb.com/kind=Postgres,kubedb.com/name=hot-postgres,kubedb.com/role=primary
Type:              ClusterIP
IP:                10.107.136.143
Port:              api  5432/TCP
TargetPort:        api/TCP
Endpoints:         172.17.0.12:5432
Port:              prom-http  56790/TCP
TargetPort:        prom-http/TCP
Endpoints:         172.17.0.12:56790
Session Affinity:  None
Events:            <none>

$ kubectl describe svc hot-postgres-replicas -n postgres
Name:              hot-postgres-replicas
Namespace:         postgres
Labels:            kubedb.com/kind=Postgres
                   kubedb.com/name=hot-postgres
Annotations:       <none>
Selector:          kubedb.com/kind=Postgres,kubedb.com/name=hot-postgres
Type:              ClusterIP
IP:                10.105.182.127
Port:              api  5432/TCP
TargetPort:        api/TCP
Endpoints:         172.17.0.12:5432,172.17.0.7:5432,172.17.0.9:5432
Port:              prom-http  56790/TCP
TargetPort:        prom-http/TCP
Endpoints:         172.17.0.12:56790,172.17.0.7:56790,172.17.0.9:56790
Session Affinity:  None
Events:            <none>

$ cat postgres.yaml 
apiVersion: kubedb.com/v1alpha1
kind: Postgres
metadata:
  name: hot-postgres
  namespace: postgres
spec:
  version: "9.6"
  replicas: 3
  standbyMode: hot
  storage:
    storageClassName: "standard"
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 50Mi
  monitor:
    agent: prometheus.io/coreos-operator
    prometheus:
      namespace: monitoring
      labels:
        k8s-app: kubedb
      port: 56790
      interval: 30s

the replicas themselves are running just fine and the failover on service level as well. Do the services need the label k8s-app: kubedb too?

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 18 (8 by maintainers)

Most upvoted comments

Boom! Got it! That was the ticket. I thought the label had to be in the Deployment template metadata.

Thank you so much @brancz ! I really appreciate the help.

Also, via experimentation, and so that there is an entry in the permanent record, I got the following error when I ran my ServiceMonitor in the monitoring namespace,

level=error ts=2018-10-31T16:56:56.554710816Z caller=main.go:234 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:288: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list pods in the namespace \"new_namespace\""
level=error ts=2018-10-31T16:56:56.554728622Z caller=main.go:234 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:287: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list services in the namespace \"new_namespace\""
level=error ts=2018-10-31T16:56:56.554736294Z caller=main.go:234 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:286: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list endpoints in the namespace \"new_namespace\""

But when I run in the new_namespace namespace everything behaves as expected.

vincentjorgensen on Oct 31, 2018

Your Service object has no labels, but in your ServiceMonitor objects you specify you want to select Services with the component: new_service label.

brancz on Oct 31, 2018

“helm delete” can delete a helm release. Yes once deleted you can just kubectl apply the generated manifests directory.

brancz on Oct 27, 2018

It’s because we give the Prometheus server only the minimally necessary roles and role bindings to monitor things in its own namespace, kube-system and default (the Kubernetes API is discovered through this). How do you install the Prometheus Operator / kube-prometheus stack (if you’re using jsonnet you can just do this)?

brancz on Aug 14, 2018