prometheus-operator: Thanos sidecar cannot reach prometheus

What did you do? Install prometheus-operator with Thanos

What did you expect to see? Thanos sidecar can communicate with Prometheus

What did you see instead? Under which circumstances?

level=warn ts=2019-03-14T14:17:40.855608496Z caller=sidecar.go:132 msg="failed to fetch initial external labels. Is Prometheus running? Retrying" err="request flags against http://localhost:9090/api/v1/status/config: Get http://localhost:9090/api/v1/status/config: dial tcp 127.0.0.1:9090: connect: connection refused"

Environment

• Prometheus Operator version:

0.29

• Kubernetes version information: Client Version: version.Info{Major:“1”, Minor:“13”, GitVersion:“v1.13.1”, GitCommit:“eec55b9ba98609a46fee712359c7b5b365bdd920”, GitTreeState:“clean”, BuildDate:“2018-12-13T10:39:04Z”, GoVersion:“go1.11.2”, Compiler:“gc”, Platform:“linux/amd64”}

• Kubernetes cluster kind: Kubespray with kubeadm method

• Manifests:

prometheus-prometheus.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: quay.io/prometheus/prometheus
  nodeSelector:
    beta.kubernetes.io/os: linux
  podMetadata:
    labels:
      thanos-peers: "true"
  replicas: 2
  enableAdminAPI: true
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  thanos:
    baseImage: improbable/thanos
    objectStorageConfig:
      key: thanos.yaml
      name: thanos-objstore-config
    peers: thanos-peers.monitoring.svc:10900
    version: v0.3.2
  version: v2.7.2

• Prometheus statefulset decription:

Name:               prometheus-k8s
Namespace:          monitoring
CreationTimestamp:  Tue, 12 Mar 2019 18:31:52 +0100
Selector:           app=prometheus,prometheus=k8s,thanos-peers=true
Labels:             prometheus=k8s
Annotations:        kubectl.kubernetes.io/last-applied-configuration:
                      {"apiVersion":"monitoring.coreos.com/v1","kind":"Prometheus","metadata":{"annotations":{},"labels":{"prometheus":"k8s"},"name":"k8s","name...
                    prometheus-operator-input-hash: 3465753947044873037
Replicas:           824641509168 desired | 2 total
Update Strategy:    RollingUpdate
Pods Status:        2 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=prometheus
                    prometheus=k8s
                    thanos-peers=true
  Service Account:  prometheus-k8s
  Containers:
   prometheus:
    Image:      quay.io/prometheus/prometheus:v2.7.2
    Port:       9090/TCP
    Host Port:  0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --storage.tsdb.retention=1d
      --web.enable-lifecycle
      --storage.tsdb.no-lockfile
      --web.route-prefix=/
      --storage.tsdb.min-block-duration=2h
      --storage.tsdb.max-block-duration=2h
    Requests:
      memory:     400Mi
    Liveness:     http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=120
    Environment:  <none>
    Mounts:
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /prometheus from prometheus-k8s-db (rw)
   prometheus-config-reloader:
    Image:      quay.io/coreos/prometheus-config-reloader:v0.29.0
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/prometheus-config-reloader
    Args:
      --log-format=logfmt
      --reload-url=http://localhost:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
    Limits:
      cpu:     50m
      memory:  50Mi
    Environment:
      POD_NAME:   (v1:metadata.name)
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
   rules-configmap-reloader:
    Image:      quay.io/coreos/configmap-reload:v0.0.1
    Port:       <none>
    Host Port:  <none>
    Args:
      --webhook-url=http://localhost:9090/-/reload
      --volume-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:        100m
      memory:     25Mi
    Environment:  <none>
    Mounts:
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
   thanos-sidecar:
    Image:       improbable/thanos:v0.3.2
    Ports:       10902/TCP, 10901/TCP, 10900/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Args:
      sidecar
      --prometheus.url=http://localhost:9090/
      --tsdb.path=/prometheus
      --cluster.address=[$(POD_IP)]:10900
      --grpc-address=[$(POD_IP)]:10901
      --cluster.peers=thanos-peers.monitoring.svc:10900
      --objstore.config=$(OBJSTORE_CONFIG)
    Environment:
      POD_IP:            (v1:status.podIP)
      HOST_IP:           (v1:status.hostIP)
      OBJSTORE_CONFIG:  <set to the key 'thanos.yaml' in secret 'thanos-objstore-config'>  Optional: false
    Mounts:
      /prometheus from prometheus-k8s-db (rw)
  Volumes:
   config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s
    Optional:    false
   config-out:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
   prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional:  false
   prometheus-k8s-db:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
Volume Claims:  <none>

• Workaround: If i edit the statefulset and remplace --prometheus.url=http://localhost:9090/ by --prometheus.url=http://prometheus-k8s.monitoring.svc:9090/ it works