istio: Readiness probe failed: HTTP probe failed with statuscode: 503

How to fix this - if possible?

Download istio 1.5.2 installed by

    istioctl manifest apply --set profile=default --set values.gateways.istio-ingressgateway.type=NodePort

root@master ~]# kubectl describe pod istio-ingressgateway-6cf56d9598-ptqp7
   .....
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  <unknown>             default-scheduler  Successfully assigned istio-system/istio-ingressgateway-6cf56d9598-ptqp7 to node1.yr
  Normal   Pulled     21m                   kubelet, node1.yr  Container image "docker.io/istio/proxyv2:1.5.2" already present on machine
  Normal   Created    21m                   kubelet, node1.yr  Created container istio-proxy
  Normal   Started    21m                   kubelet, node1.yr  Started container istio-proxy
  Warning  Unhealthy  111s (x600 over 21m)  kubelet, node1.yr  Readiness probe failed: HTTP probe failed with statuscode: 503

[root@master ~]# kubectl get po
NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-6cf56d9598-ptqp7   0/1     Running   0          25m
istiod-c7757dcf7-2dpgn                  1/1     Running   0          25m
prometheus-dfd976959-5jxxf              1/2     Running   0          25m
[root@master ~]# 

[root@master ~]# kubectl get endpoints -n istio-system
NAME                   ENDPOINTS                                                             AGE
istio-ingressgateway                                                                         30m
istio-pilot            100.96.79.25:8080,100.96.79.25:15011,100.96.79.25:15010 + 3 more...   30m
istiod                 100.96.79.25:15012,100.96.79.25:15017                                 30m
prometheus      

Environment: Virutal Machine on private cloud

[root@master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:30:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
[root@master ~]# 

[root@master ~]# kubectl get po -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-6b4c6ff56f-w4n4w   1/1     Running   1          31h
calico-node-gks95                          1/1     Running   1          31h
calico-node-q6fng                          1/1     Running   1          31h
coredns-689857ddd7-942qw                   1/1     Running   1          31h
coredns-689857ddd7-zw6bb                   1/1     Running   1          31h
etcd-master.yr                             1/1     Running   1          31h
kube-apiserver-master.yr                   1/1     Running   1          31h
kube-controller-manager-master.yr          1/1     Running   1          31h
kube-proxy-6s6bd                           1/1     Running   1          31h
kube-proxy-t9v48                           1/1     Running   1          31h
kube-scheduler-master.yr                   1/1     Running   1          31h
[root@master ~]# 

Logs

[root@master ~]# kubectl logs istio-ingressgateway-6cf56d9598-ptqp7

2020-04-26T08:40:15.309590Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-04-26T08:40:17.309575Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-04-26T08:40:19.309387Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-04-26T08:40:21.309381Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-04-26T08:40:23.309601Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-04-26T08:40:25.309706Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2020-04-26 08:40:26.977][20][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2020-04-26 08:40:26.977][20][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-04-26T08:40:27.309739Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2020-04-26T08:40:29.309719Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2020-04-26 08:40:30.144][20][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2020-04-26 08:40:30.144][20][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2020-04-26T08:40:31.309331Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 7
  • Comments: 19 (1 by maintainers)

Most upvoted comments

I am facing same error. When i applyed this offical document configretion demo,this error had been resloved. https://istio.io/latest/zh/docs/tasks/security/authentication/authn-policy/ I guess this problem because of istio default require mutual TLS. when you applyed last config,please regenerate pod.

[root@hd1-msb-testk8scluster-m1]# kubectl apply -f - <<EOF apiVersion: “networking.istio.io/v1alpha3” kind: “DestinationRule” metadata: name: “default” namespace: “istio-system” spec: host: “*.local” trafficPolicy: tls: mode: ISTIO_MUTUAL EOF