istio: Prometheus not able to scrape from Pod

Bug description I’ve deployed Prometheus with mTLS enabled. This prometheus pod is started with secret istio.default mounted at /etc/prometheus/secrets/istio.default/. I’ve another deployment with mTLS enabled where a pod has necessary annotations like below.

prometheus.io.port: 8080
prometheus.io.scrape: true

But prometheus is not able to scrape from this pod. I’m getting error. Get https://100.117.155.31:8080/metrics: http: server gave HTTP response to HTTPS client I’ve created below: -

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubernetes-pods-monitor-istio-secure
  namespace: foo-infra
  labels:
    monitoring: kube-pods
spec:
  selector:
    matchExpressions:
      - {key: istio-prometheus-ignore, operator: DoesNotExist}
  namespaceSelector:
    matchNames:
      - default
      - ops-infra
      - istio-system
  jobLabel: kubernetes-pods-monitor-istio-secure
  endpoints:
  - interval: 30s
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/istio.default/root-cert.pem
      certFile: /etc/prometheus/secrets/istio.default/cert-chain.pem
      keyFile: /etc/prometheus/secrets/istio.default/key.pem
      insecureSkipVerify: true  # prometheus does not support secure naming.
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: 'true'
    # sidecar status annotation is added by sidecar injector and
    # istio_workload_mtls_ability can be specifically placed on a pod to indicate its ability to receive mtls traffic.
    - sourceLabels: [__meta_kubernetes_pod_annotation_sidecar_istio_io_status, __meta_kubernetes_pod_annotation_istio_mtls]
      action: keep
      regex: (([^;]+);([^;]*))|(([^;]*);(true))
    - sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
      action: drop
      regex: (http)
    - sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      targetLabel: __metrics_path__
      regex: (.+)
    - sourceLabels: [__address__]  # Only keep address that is host:port
      action: keep    # otherwise an extra target with ':443' is added for https scheme
      regex: ([^:]+):(\d+)
    - sourceLabels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      targetLabel: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - sourceLabels: [__meta_kubernetes_pod_name]
      action: replace
      targetLabel: pod_name

Expected behavior Prometheus should scrape from a pod.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

❯ istioctl version --remote
client version: 1.4.5
control plane version: 1.4.5
data plane version: 1.4.5 (16 proxies)
❯ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-23T14:21:54Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.10", GitCommit:"1bea6c00a7055edef03f1d4bb58b773fa8917f11", GitTreeState:"clean", BuildDate:"2020-02-11T20:05:26Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed? istioctl manifest apply --set profile=demo --set values.grafana.enabled=false --set values.prometheus.enabled=false --set values.global.mtls.enabled=true --set values.global.controlPlaneSecurityEnabled=true