istio: isito-validation initContainer is not working in istio-cni-node

Bug description

• I have installed istio with Helm and everything works except of automatic sidecar injection
• I have tried Istio versions between 1.4.8, 1.5.0, 1.5.1 and 1.5.2, but same result
• with 1.4.8 version, pod get stuck in Init:0/1 state
• with 1.5.X versions, pod get killed immediately
• going back to Istio 1.4.4 makes sidecar injection work and updating to 1.4.5 breaks it. so, there is some code that was introduced in 1.4.5 that broke it. Looking at issues I see people describing same behavior in https://github.com/istio/istio/issues/21266

Setup details

Kuberetes: v1.15.11 Istio CNI configuration on the server::

ls -alt /etc/cni/net.d/
total 8
drwxr-xr-x. 2 root root   68 May  1 13:08 .
-rw-r--r--. 1 root root  426 May  1 13:08 YYY-istio-cni.conflist
-rw-------. 1 root root 2824 May  1 13:08 ZZZ-istio-cni-kubeconfig
drwxr-xr-x. 3 root root   19 Apr 30 16:39 ..

Contents of the files:

cat  /etc/cni/net.d/*
{
  "plugins": [
    {
      "cniVersion": "0.3.1",
      "name": "istio-cni",
      "type": "istio-cni",
      "log_level": "debug",
      "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/ZZZ-istio-cni-kubeconfig",
        "cni_bin_dir": "/opt/cni/bin",
        "exclude_namespaces": [
          "istio-system",
          "kube-system"
        ]
      }
    }
  ],
  "name": "k8s-pod-network",
  "cniVersion": "0.3.0"
}
# Kubeconfig file for Istio CNI plugin.
apiVersion: v1
kind: Config
clusters:
- name: local
  cluster:
    server: https://[192.168.128.1]:443
    certificate-authority-data: DDDDDDD
users:
- name: istio-cni
  user:
    token: "ZZZZZ"
contexts:
- name: istio-cni-context
  context:
    cluster: local
    user: istio-cni
current-context: istio-cni-context

When running with 1.4.8 I was able to capture Docker log for the pod "istio-iptables-go -…" 2 minutes ago Up 2 minutes k8s_istio-validation_.... Log:

in new validator: 172.30.22.223
Listening on 127.0.0.1:15001
Listening on 127.0.0.1:15006
Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused

• CNI node pod log:

kubectl logs istio-cni-node-zzk9m repair-cni
{"level":"info","time":"2020-05-01T13:08:36.637040Z","msg":"Controller Option: Running as a Daemon."}
{"level":"info","time":"2020-05-01T13:08:36.637071Z","msg":"Controller Option: Labeling broken pods with label cni.istio.io/uninitialized=true"}
{"level":"info","time":"2020-05-01T13:08:36.637075Z","msg":"Filter option: Only managing pods with an annotation with key sidecar.istio.io/status"}
{"level":"info","time":"2020-05-01T13:08:36.637080Z","msg":"Filter option: Only managing pods with field selector spec.nodeName=SSSSSS,"}
{"level":"info","time":"2020-05-01T13:08:36.637084Z","msg":"Filter option: Only managing pods where init container is named istio-validation"}
{"level":"info","time":"2020-05-01T13:08:36.637088Z","msg":"Filter option: Only managing pods where init container exit status is 126"}
{"level":"info","time":"2020-05-01T13:13:50.917311Z","msg":"Pod detected as broken, adding label: sampleapp/sample-app-sox-sample-ap-deployment-5b56f4dff-zktlw"}
{"level":"info","time":"2020-05-01T13:13:50.917627Z","msg":"Labeling pod sampleapp/sample-app-sox-sample-ap-deployment-5b56f4dff-zktlw with label cni.istio.io/uninitialized=true"}
{"level":"info","time":"2020-05-01T13:13:51.048565Z","msg":"Pod detected as broken, adding label: sampleapp/sample-app-sox-sample-ap-deployment-5b56f4dff-zktlw"}
{"level":"info","time":"2020-05-01T13:13:51.048804Z","msg":"Pod sampleapp/sample-app-sox-sample-ap-deployment-5b56f4dff-zktlw already has label with key cni.istio.io/uninitialized, skipping"}

Expected behavior

• sidecar should be injected

Steps to reproduce the bug

• install istio
• configure auto injection on namespace
• deploy application

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm) istioctl version --remote client version: 1.7-alpha.6d103c03fa30c66945843025aa029c9fd8517b67 control plane version: 1.4.8

kubectl version Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"d94a81c724ea8e1ccc9002d89b7fe81d58f89ede", GitTreeState:"clean", BuildDate:"2020-03-12T21:00:06Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

helm2 version Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}

How was Istio installed?

• Istio installed with helm

Environment where bug was observed (cloud vendor, OS, etc)

• BYO K8s cluster baremetal and GCE