istio: Prometheus can't scrape mTLS pods in v1.5

Bug description There seems to be an issue where the Prometheus shipped with Istio v1.5 can no longer scrape pods with the “prometheus.io/scrape: true” annotation. This seems to be because the job “kubernetes-pods-istio-secure” is gated behind the flag “.Values.security.enabled” which seems to not be present in the default profile when installing Istio with istioctl. Adding the flag manually by calling istioctl with the parameter “–set values.security.enabled=true” seems to resolve the issue. After installing Istio as described the workaround from #21843 seems to also be needed for the “kubernetes-pods-istio-secure” job to get the appropriate certificates.

Expected behavior Prometheus should be able to scrape pods protected with mTLS.

Steps to reproduce the bug

• Install istio with “istioctl manifest apply”
• Deploy a pod with the annotation “prometheus.io/scrape: true”
• Check Prometheus targets. The pod should not have been picked up as the “kubernetes-pods-istio-secure” job is missing and the “kubernetes-pods” drops mTLS pods because of the annotation “sidecar.istio.io/status”.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm) $ istioctl version --remote client version: 1.5.0 control plane version: 1.5.0 data plane version: 1.5.0 (17 proxies) $ kubectl version Client Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.3”, GitCommit:“06ad960bfd03b39c8310aaf92d1e7c12ce618213”, GitTreeState:“clean”, BuildDate:“2020-02-12T13:43:46Z”, GoVersion:“go1.13.7”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.3”, GitCommit:“06ad960bfd03b39c8310aaf92d1e7c12ce618213”, GitTreeState:“clean”, BuildDate:“2020-02-11T18:07:13Z”, GoVersion:“go1.13.6”, Compiler:“gc”, Platform:“linux/amd64”}

How was Istio installed? With istioctl CLI

Environment where bug was observed (cloud vendor, OS, etc) Ubuntu 18.04 LTS running Microk8s (not the bundled Istio)