istio: failed calling webhook "sidecar-injector.istio.io

Basic information I had setup kubernetes 1.15.9 in 4 servers, one of them is master node. The kube-apiserver/kube-controller-manager/kube-scheduler running on bare metal, not in docker containers.

Isito version: 1.4.3 calico cni plugin: 3.9.5

Install istio

curl -L https://istio.io/downloadIstio | sh -
kubectl create namespace istio-system
helm template install/kubernetes/helm/istio-init  --name istio-init --namespace istio-system | kubectl apply -f -

# wait several seconds
helm template install/kubernetes/helm/istio -**-set gateways.istio-ingressgateway.type=NodePort --set global.useMCP=false** --name istio --namespace istio-system | kubectl apply -f -

# see pods
[root@nknode-1 istio-1.4.3]# kubectl -n istio-system get pods
NAME                                      READY   STATUS      RESTARTS   AGE
istio-citadel-bc567d8b5-ttjff             1/1     Running     0          11m
istio-galley-74c86d8ffb-vdqlv             1/1     Running     0          11m
istio-ingressgateway-7765b64c69-s9fwm     0/1     Running     0          11m
istio-init-crd-10-1.4.3-49lfq             0/1     Completed   0          11m
istio-init-crd-11-1.4.3-9dtpg             0/1     Completed   0          11m
istio-init-crd-14-1.4.3-czjzc             0/1     Completed   0          11m
istio-pilot-868bf7c864-skhmh              2/2     Running     2          11m
istio-policy-8655576d5b-fnskh             2/2     Running     0          11m
istio-security-post-install-1.4.3-x65tt   0/1     Completed   0          11m
istio-sidecar-injector-7587d46668-vhfxr   1/1     Running     0          11m
istio-telemetry-5bb585bcd9-xkpjn          2/2     Running     0          11m
prometheus-8685f659f-h7nlx                1/1     Running     0          11m

# label namespace default auto-inject
kubectl label namespace default istio-injection=enabled

# create test pod
kubectl apply -f ./test.yaml

# show pods
[root@nknode-1 istio-1.4.3]# kubectl get deployments
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
test-deployment   0/2     0            0           11m

# describe deployment
  Warning  FailedCreate  19s (x9 over 11m)      replicaset-controller  Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

logs of kube-apiserver

[root@nknode-1 kube-controller]# tailf logs/kube-apiserver.WARNING
W0206 12:35:56.672627    2486 dispatcher.go:140] Failed calling webhook, failing closed sidecar-injector.istio.io: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
W0206 12:36:26.676852    2486 dispatcher.go:140] Failed calling webhook, failing closed sidecar-injector.istio.io: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: context deadline exceeded
W0206 12:36:56.681078    2486 dispatcher.go:140] Failed calling webhook, failing closed sidecar-injector.istio.io: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

logs of sidecar-injector

  {{- end }}
     traffic.sidecar.istio.io/kubevirtInterfaces: "{{ index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` }}"
2020-02-06T04:35:03.039752Z	info	http: TLS handshake error from 10.244.178.192:33284: remote error: tls: unknown certificate authority

The network is ok,

[root@nknode-1 ~]# curl -k https://istio-sidecar-injector.istio-system.svc:443
404 page not found
[root@nknode-1 ~]# ifconfig |grep 10.244
        inet 10.244.178.192  netmask 255.255.255.255

Then, I followed the guide from https://istio.io/docs/ops/common-problems/injection/ trying to find the root cause:

# Verify the caBundle 
[root@nknode-1 ~]# kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | md5sum
1ef3a85c567202c84e38a53f9997774f  -
[root@nknode-1 ~]# kubectl -n istio-system get secret istio.istio-sidecar-injector-service-account -o jsonpath='{.data.root-cert\.pem}' | md5sum
1ef3a85c567202c84e38a53f9997774f  -
[root@nknode-1 ~]#  kubectl -n istio-system get pod -listio=sidecar-injector
NAME                                      READY   STATUS    RESTARTS   AGE
istio-sidecar-injector-7587d46668-vhfxr   1/1     Running   0          82m
[root@nknode-1 ~]# kubectl -n istio-system get endpoints istio-sidecar-injector
NAME                     ENDPOINTS                             AGE
istio-sidecar-injector   10.244.76.66:15014,10.244.76.66:443   82m

It seems the sidecar-injector and citadel worked fine, so I wonder if the problem is the tls key mismatch between kube-api-server and sidecar-injector.

Then, I try to use the root ca pem of kube-api-server in sidecar-injector.

# clean the environment

# for kube-apiserver the tls as following:

/data/home/kube-controller//bin/kube-apiserver --log-dir=/data/home/kube-controller//logs/ --logtostderr=false --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --allow-privileged=true --client-ca-file=/data/home/kube-controller//etc//pki/ca.pem --disable-admission-plugins=PersistentVolumeLabel --enable-admission-plugins=NodeRestriction,ValidatingAdmissionWebhook,MutatingAdmissionWebhook,ResourceQuota --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --etcd-cafile=/data/home/kube-controller//etc//pki/ca.pem --etcd-certfile=/data/home/kube-controller//etc//pki/apiserver.pem --etcd-keyfile=/data/home/kube-controller//etc//pki/apiserver-key.pem --etcd-servers=https://192.168.0.76:2379 --insecure-port=0 --kubelet-certificate-authority=/data/home/kube-controller//etc//pki/ca.pem --kubelet-client-certificate=/data/home/kube-controller//etc//pki/apiserver-kubelet.pem --kubelet-client-key=/data/home/kube-controller//etc//pki/apiserver-kubelet-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --service-account-key-file=/data/home/kube-controller//etc//pki/service-account.pem --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/data/home/kube-controller//etc//pki/apiserver.pem --tls-private-key-file=/data/home/kube-controller//etc//pki/apiserver-key.pem --v=5

# for sidecar-injector, use the same ca files
root-cert.pem:  same as /data/home/kube-controller//etc//pki/ca.pem
ca-cert.pem: same as /data/home/kube-controller//etc//pki/ca.pem
ca-key.pem: same as /data/home/kube-controller//etc//pki/ca-key.pem
cert-chain.pem: empty

# install istio

# create ns
kubectl create namespace istio-system

# create secret using custom tls keys
kubectl create secret generic cacerts -n istio-system --from-file=./certs/ca-cert.pem --from-file=./certs/ca-key.pem --from-file=./certs/cert-chain.pem --from-file=./certs/root-cert.pem

# init 
helm template install/kubernetes/helm/istio-init  --name istio-init --namespace istio-system | kubectl apply -f -

helm template install/kubernetes/helm/istio --set security.selfSigned=false --set gateways.istio-ingressgateway.type=NodePort --set global.useMCP=false --name istio --namespace istio-system | kubectl apply -f -

# watch pods status
[root@nknode-1 istio-1.4.3]# kubectl -n istio-system get pods
NAME                                      READY   STATUS      RESTARTS   AGE
istio-citadel-764564964d-vsvht            1/1     Running     0          4m22s
istio-galley-74c86d8ffb-v6zdr             1/1     Running     0          4m22s
istio-ingressgateway-7765b64c69-7vmt5     0/1     Running     0          4m22s
istio-init-crd-10-1.4.3-h6gmn             0/1     Completed   0          4m50s
istio-init-crd-11-1.4.3-44975             0/1     Completed   0          4m50s
istio-init-crd-14-1.4.3-ttdc6             0/1     Completed   0          4m50s
istio-pilot-868bf7c864-2qq9j              2/2     Running     2          4m22s
istio-policy-8655576d5b-j9gdh             2/2     Running     0          4m22s
istio-security-post-install-1.4.3-q8rd8   0/1     Completed   0          4m22s
istio-sidecar-injector-7587d46668-sv4k7   1/1     Running     0          4m21s
istio-telemetry-5bb585bcd9-zqmws          2/2     Running     0          4m22s
prometheus-8685f659f-fkknx                1/1     Running     0          4m22s

# create test pod

logs of kube-apiserver

E0206 14:03:20.499389    2486 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted
W0206 14:09:31.134699    2486 dispatcher.go:140] Failed calling webhook, failing closed sidecar-injector.istio.io: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: net/http: TLS handshake timeout
W0206 14:09:41.141094    2486 dispatcher.go:140] Failed calling webhook, failing closed sidecar-injector.istio.io: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: net/http: TLS handshake timeout
W0206 14:09:51.144517    2486 dispatcher.go:140] Failed calling webhook, failing closed sidecar-injector.istio.io: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: net/http: TLS handshake timeout

logs of sidecar-injector

  {{- end }}
     traffic.sidecar.istio.io/kubevirtInterfaces: "{{ index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` }}"
2020-02-06T06:09:31.134890Z	info	http: TLS handshake error from 10.244.178.192:38846: EOF
2020-02-06T06:09:41.141157Z	info	http: TLS handshake error from 10.244.178.192:38880: EOF
2020-02-06T06:09:51.144581Z	info	http: TLS handshake error from 10.244.178.192:38916: EOF
2020-02-06T06:10:01.149519Z	info	http: TLS handshake error from 10.244.178.192:38952: EOF
2020-02-06T06:10:11.152854Z	info	http: TLS handshake error from 10.244.178.192:38986: EOF
2020-02-06T06:10:21.158827Z	info	http: TLS handshake error from 10.244.178.192:39020: EOF
2020-02-06T06:10:31.163541Z	info	http: TLS handshake error from 10.244.178.192:39056: EOF
2020-02-06T06:10:41.167073Z	info	http: TLS handshake error from 10.244.178.192:39090: EOF

test network connection

[root@nknode-1 istio-1.4.3]# kubectl -n istio-system get svc|grep sidecar
istio-sidecar-injector   ClusterIP   10.107.145.97    <none>        443/TCP,15014/TCP                                                                                                                            8m16s
[root@nknode-1 istio-1.4.3]# curl -k https://10.107.145.97:443

^C
[root@nknode-1 istio-1.4.3]# nc -v 10.107.145.97 443
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.107.145.97:443.

check ca bundle

[root@nknode-1 istio-1.4.3]# kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | md5sum
e152e27deadf0609903c810de6f041c0  -
[root@nknode-1 istio-1.4.3]# kubectl -n istio-system get secret istio.istio-sidecar-injector-service-account -o jsonpath='{.data.root-cert\.pem}' | md5sum
e152e27deadf0609903c810de6f041c0  -

Please help me to make it work. Many thanks.