prometheus: Error polling from Kubernetes API server

Getting this error polling the api server:

Get https://kubernetes.default.svc:443/metrics: x509: certificate signed by unknown authority

Here’s the snippet from my config:

  - job_name: kubernetes
    scrape_interval: 30s
    scrape_timeout: 10s
    kubernetes_sd_configs:
      - api_servers:
        - https://kubernetes.default.svc
        in_cluster: true
        tls_config:
          insecure_skip_verify: true

version: 0.17.0

Certificate served by api-server (removed some clutter):

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            59:8f:55:59:f9:b3:bc:3d:96:90:e1:9e:cd:d1:b3:59
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Databricks, CN=Sub CA
        Validity
            Not Before: Jan 20 01:01:15 2016 GMT
            Not After : Jan 19 01:01:15 2017 GMT
        Subject: C=US, O=Databricks, CN=master.central-a.kube.dev.databricks.com
        X509v3 extensions:

            X509v3 Authority Key Identifier: 
                keyid:A2:F2:F1:0C:48:76:3C:88:98:0F:85:A1:38:42:21:83:B5:49:6E:15

            X509v3 Basic Constraints: critical
                CA:FALSE

            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Subject Key Identifier: 
                1F:A9:8A:75:57:78:C9:D0:57:DD:07:F3:5D:99:49:0E:5D:B0:7F:C6
            X509v3 Subject Alternative Name: 
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc,

ca.crt in serviceaccount (again, removed some clutter):

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            88:56:59:48:78:d9:2e:0f:ee:4a:b9:ac:8a:86:a7:b6
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Databricks, CN=Root CA
        Validity
            Not Before: Jan 13 23:21:52 2016 GMT
            Not After : Jan 12 23:21:52 2017 GMT
        Subject: C=US, O=Databricks, CN=Sub CA

        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:39:FA:60:37:05:A0:F8:34:0B:4A:2B:4E:03:8F:B8:C0:B9:26:55:54

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier: 
                A2:F2:F1:0C:48:76:3C:88:98:0F:85:A1:38:42:21:83:B5:49:6E:15

Note that X509v3 Authority Key Identifier in the server cert matches the X509v3 Subject Key Identifier in the CA cert.

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 16 (8 by maintainers)

Most upvoted comments

Thanks @jimmidyson I finally understand the difference between scraping and service discovery. The following config works fine:

scrape_configs:
  - job_name: 'kubernetes-cluster'

    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token: <hidden>

    kubernetes_sd_configs:
    - api_servers:
      - 'https://kubernetes.default.svc'
      in_cluster: false

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token: <hidden>

The kubernetes_sd_configs section is for discovery of targets, it this case it queries the kubernetes api and returns a list of targets like https://kubernetes.default.svc:443/metrics then prometheus scrapes that target.

In this case the target and discovery endpoints are the same, but if you had the kubernetes_sd_configs discover other application target nodes, those might have different certs and bearer_tokens.

mindfulmonk on Jul 1, 2016