istio: Istio standard service metrics not showing up in prometheus

Bug description We have prometheus with prometheus operator installed in our cluster and we don’t use Istio prometheus. We were using ServiceMonitor to integrate Istio metrics with prometheus. For 1.4.5 version we used below ServiceMonitor config to query mixer for Istio standard metrics in https://istio.io/latest/docs/reference/config/metrics/#metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istio-mixer
  labels:
    istio: mixer
    k8s-app: istio-mixer
spec:
  jobLabel: k8s-app
  selector:
    matchLabels:
      istio: mixer
  namespaceSelector:
    matchNames:
    - istio-system
  endpoints:
  - port: prometheus
    interval: 15s
    scrapeTimeout: 5s

Once we migrated istio from 1.4.5 to 1.8 , we stopped seeing istio-metrics in prometheus and updated ServiceMonitor by referring to example in https://github.com/istio/istio/blob/1.8.0/samples/addons/extras/prometheus-operator.yaml. Both PodMonitor and ServiceMonitor is shown as target in prometheus dashboard but i don’t see Istio standard metrics like istio_requests_total, istio_request_duration_milliseconds, istio_request_bytes etc in https://istio.io/latest/docs/reference/config/metrics/#metrics. Initially we disabled meshConfig.enablePrometheusMerge=false but now I have enabled meshConfig.enablePrometheusMerge=true but no help. Below is my config, Can you help me what is missing to query Istio standard service level metrics.

ServiceMonitor below shows only istio_build metric. PodMonitor below shows only istio_agent_XXX metrics But not seeing Istio standard metrics like istio_requests_total, istio_request_duration_milliseconds, istio_request_bytes etc

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 name: istio-component-monitor
 labels:
   k8s-app: istio-components
spec:
 jobLabel: k8s-app
 selector:
   matchExpressions:
   - {key: istio, operator: In, values: [pilot,istiod]}
 namespaceSelector:
   matchNames:
   - istio-system
 endpoints:
 - port: http-monitoring
   interval: 15s
   scrapeTimeout: 5s
---    
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
 name: envoy-stats-monitor
 labels:
   k8s-app: istio-proxies
spec:
 jobLabel: k8s-app
 selector:
   matchExpressions:
   - {key: istio-prometheus-ignore, operator: DoesNotExist}
 namespaceSelector:
   any: true
 podMetricsEndpoints:
 - path: /stats/prometheus
   interval: 15s
   scrapeTimeout: 5s
   relabelings:
   - action: keep
     sourceLabels: [__meta_kubernetes_pod_container_name]
     regex: "istio-proxy"
   - action: keep
     sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape]
   - sourceLabels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
     action: replace
     regex: ([^:]+)(?::\d+)?;(\d+)
     replacement: $1:$2
     targetLabel: __address__
   - action: labeldrop
     regex: "__meta_kubernetes_pod_label_(.+)"
   - sourceLabels: [__meta_kubernetes_namespace]
     action: replace
     targetLabel: namespace
   - sourceLabels: [__meta_kubernetes_pod_name]
     action: replace
     targetLabel: pod_name

[X ] Docs [X ] Installation [ ] Networking [ ] Performance and Scalability [X ] Extensions and Telemetry [ ] Security [X ] Test and Release [ X] User Experience [ ] Developer Infrastructure [ X] Upgrade

Expected behavior Not seeing Istio standard metrics like istio_requests_total, istio_request_duration_milliseconds, istio_request_bytes etc

Steps to reproduce the bug Install prometheus with prometheus operator Add ServiceMonitor as in example https://github.com/istio/istio/blob/1.8.0/samples/addons/extras/prometheus-operator.yaml Not seeing Istio standard metrics like istio_requests_total, istio_request_duration_milliseconds, istio_request_bytes etc in prometheus dashboard

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm) 1.8

How was Istio installed? istiocrl manifest -f custom-config.yaml > 1.8.0

  mesh: |-
    accessLogFile: /dev/stdout
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      proxyMetadata:
        DNS_AGENT: ""
      tracing:
        zipkin:
          address: zipkin.istio-system:9411
    enableAutoMtls: true
    enablePrometheusMerge: true
    enableTracing: false
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY
    rootNamespace: istio-system
    trustDomain: cluster.local

Environment where the bug was observed (cloud vendor, OS, etc) AWS EKS 1.18 with istio 1.8

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue. Refer cluster state archive for more details.