prometheus-operator: etcd3 metrics missing

I installed all the prometheus-operator examples, quite a lot of metrics…

But missing etcd metrics… Have running

NAME                    ENDPOINTS                                                              AGE
alertmanager-main       10.244.18.7:9093,10.244.46.10:9093,10.244.83.18:9093                   14d
alertmanager-operated   10.244.18.7:9093,10.244.46.10:9093,10.244.83.18:9093 + 3 more...       14d
etcd-k8s                10.142.0.2:2379                                                        1h
grafana                 10.244.46.7:3000                                                       14d
kube-state-metrics      10.244.46.6:8080                                                       14d
node-exporter           172.31.1.143:9100,172.31.10.210:9100,172.31.18.136:9100 + 12 more...   14d
postgres-exporter       172.31.27.181:9187                                                     8d
prometheus-k8s          10.244.18.9:9090,10.244.8.10:9090                                      14d
prometheus-operated     10.244.18.9:9090,10.244.46.12:9090,10.244.8.10:9090                    14d
prometheus-operator     10.244.46.4:8080                                                       14d
redis-exporter          172.31.27.181:9121

and getting on http://localhost:2379/metrics (kubectl proxy --port 2379) not the expected results of etcd_server values…

Kubernetes version information:

tom.stark@MBproTStark prometheus-monitoring (aws)  $ kubectl --context=aws version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"922a86cfcd65915a9b2f69f3f193b8907d741d9c", GitTreeState:"clean", BuildDate:"2017-07-21T19:06:19Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6+coreos.1", GitCommit:"42a5c8b99c994a51d9ceaed5d0254f177e97d419", GitTreeState:"clean", BuildDate:"2017-06-21T01:10:07Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 19 (6 by maintainers)

Most upvoted comments

The etcd_helper_* metrics are actually coming from the apiserver, not etcd. You are not successfully discovering etcd yet.

The ServiceMonitor you referenced needs to be adapted, as it specifically selects only those service/endpoints objects that have the k8s-app: kube-controller-manager label. What you need to do is label your service and endpoints object properly and then adapt the ServiceMonitor manifest. You should end up with something like this (assuming you are deploying this in the monitoring namespace).

apiVersion: v1
kind: Service
metadata:
  name: etcd-k8s
  labels:
    k8s-app: etcd
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: api
    port: 2379
    protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
  name: etcd-k8s
  labels:
    k8s-app: etcd
subsets:
- addresses:
  - ip: 10.11.12.52
    nodeName: 10.11.12.52
  ports:
  - name: api
    port: 2379
    protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: etcd-k8s
  labels:
    k8s-app: etcd-k8s
spec:
  jobLabel: k8s-app
  endpoints:
  - port: api
    interval: 30s
  selector:
    matchLabels:
      k8s-app: etcd
  namespaceSelector:
    matchNames:
    - monitoring

I haven’t tried this out, but it should give you guidance. Let me know how it goes 🙂 .

brancz on Aug 29, 2017

I’m able to curl my etcd endpoint by passing the certificates:

curl -L -v --cacert ./ca.pem --cert ./etcd-client.pem --key ./etcd-client-key.pem https://etcd0.mycompany-internal-domain.local:2379/metrics

Where etcd0.mycompany-internal-domain.local is the full DNS for my first etcd node. I have 3 of them, each with its own DNS (Route53)

etcd0.mycompany-internal-domain.local
etcd0.mycompany-internal-domain.local
etcd1.mycompany-internal-domain.local

I want to alter the ServiceMonitor spec to specify tlsConfig:

spec:
  jobLabel: k8s-app
  endpoints:
  - port: api
    interval: 30s
    scheme: https
    tlsConfig:
      caFile: credentials/ca.pem
      certFile: credentials/etcd-client.pem
      keyFile: credentials/etcd-client-key.pem

where credentials is the path to the credential folder on my laptop.

The problem I see is that Endpoint takes an IP address:

- addresses:
  - ip: 10.11.12.52
    nodeName: 10.11.12.52

The certificates are not valid for the IP address. Not a valid name. I tried with curl, it won’t take it. How can I use DNS based service?

Kubernetes has the concept of ExternalName for a service:

spec:
  type: ExternalName
  externalName: etcd0.mycompany-internal-domain.local

How do I link the ServiceMonitor to this ExternalName endpoint?

Vince-Cercury on Aug 30, 2017

I installed exactly that https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/README.md setup, but without helm… In my initial description I display the service and endpoints.

But I’m not sure to get the right IP for putting into the setup … What ip address I have to use? Normally I should use a IP of the master nodes?!

https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/manifests/etcd/etcd-bootkube-vagrant-multi.yaml

fragtom on Aug 22, 2017