prometheus: Unexpected EOF with Istio mTLS and HTTP2

What did you do? I want to monitor some apps (with servicemonitor) hosted in namespaces with Istio mTLS (sidecar injection). I have added Istio TLS certificates into the Prometheus pods with a secret t(there are hosted in a namespace without sidecar injection).

What did you expect to see? All the targets with Istio mTLS will be OK

What did you see instead? Under which circumstances?

All the targets with Istio mTLS are KO with unexpected EOF error
If I add the PROMETHEUS_COMMON_DISABLE_HTTP2=1 environment variable, all targets with mTLS will recover to OK
If I downgrade Prometheus to v2.30.x version, all targets with mTLS will recover to OK

Environment

GKE 1.19.14
Prometheus-operator v0.53.1
Istio 1.15.1

Prometheus version:

Version | 2.32.1
Revision | 41f1a8125e664985dd30674e5bdf6b683eff5d32
Branch | HEAD
BuildUser | root@54b6dbd48b97
BuildDate | 20211217-22:08:06
GoVersion | go1.17.5

Prometheus configuration file:

- job_name: serviceMonitor/b3s/gateway/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: https
  tls_config:
    ca_file: /etc/prometheus/secrets/istio.default/root-cert.pem
    cert_file: /etc/prometheus/secrets/istio.default/cert-chain.pem
    key_file: /etc/prometheus/secrets/istio.default/key.pem
    insecure_skip_verify: true
  follow_redirects: true
  relabel_configs:
  ...
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    namespaces:
      names:
      - b3s

Logs:

{"caller":"scrape.go:1292","component":"scrape manager","err":"Get \"https://x.x.x.x:yyyy/metrics\": unexpected EOF","level":"debug","msg":"Scrape failed","scrape_pool":"serviceMonitor/b3s/gateway/0","target":"https://x.x.x.x:yyyy/metrics","ts":"2022-01-26T13:53:52.593Z"}

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 16 (7 by maintainers)

Most upvoted comments

Hi there, I was facing what I judge to be the exact same issue discussed here, after proper configuring and injecting the Istio sidecar in the Prometheus pod and also configuring the tls_config at the scrapping job level, I started to get unexpected EOF on Prometheus side.

In the end, it turns to be an issue on the configuration of the Kubernetes Service for the deployment/pods Prometheus is supposed to be scrapping, the issue is related to Istio Protocol Selection, my Service was not properly defining the port as HTTP and then the ALPN between Prometheus and the Service endpoint was failing.

So this is pretty much what I had and what I did to fix it:

What I had

apiVersion: v1
kind: Service
metadata:
  name: some-service
spec:
  type: ClusterIP
  ports:
  - name: web
    port: 9898
    protocol: TCP
    targetPort: 9898

What fixed the issue:

Adding the appProtocol: http to the ports item as following:

apiVersion: v1
kind: Service
metadata:
  name: some-service
spec:
  type: ClusterIP
  ports:
  - name: web
    port: 9898
    protocol: TCP
    targetPort: 9898
    appProtocol: http

+11

rodrigorfk on Mar 11, 2022

Note that now you can just use enable_http2: false

roidelapluie on Jun 30, 2022

I have the same issue. I have configured pod annotations of the StatefulSet for prometheus-operator to inject the certificates from the sidecar in a memory volume.

While the cert mount works, this is what I get from the sidecar when I try to cURL the metrics endpoint:

* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

yuliyantsvetkov on Mar 9, 2022

I have the file. thanks.

roidelapluie on Feb 4, 2022