istio: istio-ingressgateway readiness probe statuscode: 503

Describe the bug istio-ingressgateway readiness check produces 503’s for 1-2 minutes.

Expected behavior istio-ingressgateway ready to run without a 2 minute delay after the rest of the control plane is operational.

Steps to reproduce the bug Deploy istio using preliminary documentation.

Version rc.4, Kubernetes 1.10.3, minikube + minikube tunnel rc.4, Kubernetes 1.13.3, metallb 073

Installation basic install - helm template or helm install

Environment bare metal linux, macos, seems to not matter.

Cluster state

minikube describe on istio-ingressgateway:

Events:
  Type     Reason                 Age                 From               Message
  ----     ------                 ----                ----               -------
  Normal   Scheduled              48m                 default-scheduler  Successfully assigned istio-ingressgateway-7c8b5f67dc-pphn8 to minikube
  Normal   SuccessfulMountVolume  48m                 kubelet, minikube  MountVolume.SetUp succeeded for volume "ingressgateway-ca-certs"
  Normal   SuccessfulMountVolume  48m                 kubelet, minikube  MountVolume.SetUp succeeded for volume "ingressgateway-certs"
  Normal   SuccessfulMountVolume  48m                 kubelet, minikube  MountVolume.SetUp succeeded for volume "istio-certs"
  Normal   SuccessfulMountVolume  48m                 kubelet, minikube  MountVolume.SetUp succeeded for volume "istio-ingressgateway-service-account-token-5rd7n"
  Normal   Pulling                48m                 kubelet, minikube  pulling image "docker.io/istio/proxyv2:1.1.0-rc.4"
  Normal   Pulled                 48m                 kubelet, minikube  Successfully pulled image "docker.io/istio/proxyv2:1.1.0-rc.4"
  Normal   Created                48m                 kubelet, minikube  Created container
  Normal   Started                48m                 kubelet, minikube  Started container
  Warning  Unhealthy              47m (x17 over 48m)  kubelet, minikube  Readiness probe failed: HTTP probe failed with statuscode: 503

minikube logs on istio-ingressgateway:

Stevens-MacBook-Pro:kube sdake$ kubectl logs istio-ingressgateway-7c8b5f67dc-pphn8 -n istio-system
2019-03-13T05:47:52.918731Z	info	FLAG: --applicationPorts="[]"
2019-03-13T05:47:52.918752Z	info	FLAG: --binaryPath="/usr/local/bin/envoy"
2019-03-13T05:47:52.918756Z	info	FLAG: --concurrency="0"
2019-03-13T05:47:52.918758Z	info	FLAG: --configPath="/etc/istio/proxy"
2019-03-13T05:47:52.918760Z	info	FLAG: --connectTimeout="10s"
2019-03-13T05:47:52.918762Z	info	FLAG: --controlPlaneAuthPolicy="NONE"
2019-03-13T05:47:52.918765Z	info	FLAG: --controlPlaneBootstrap="true"
2019-03-13T05:47:52.918767Z	info	FLAG: --customConfigFile=""
2019-03-13T05:47:52.918768Z	info	FLAG: --disableInternalTelemetry="false"
2019-03-13T05:47:52.918770Z	info	FLAG: --discoveryAddress="istio-pilot:15010"
2019-03-13T05:47:52.918772Z	info	FLAG: --domain="istio-system.svc.cluster.local"
2019-03-13T05:47:52.918774Z	info	FLAG: --drainDuration="45s"
2019-03-13T05:47:52.918776Z	info	FLAG: --envoyMetricsServiceAddress=""
2019-03-13T05:47:52.918779Z	info	FLAG: --help="false"
2019-03-13T05:47:52.918781Z	info	FLAG: --id=""
2019-03-13T05:47:52.918783Z	info	FLAG: --ip=""
2019-03-13T05:47:52.918784Z	info	FLAG: --lightstepAccessToken=""
2019-03-13T05:47:52.918786Z	info	FLAG: --lightstepAddress=""
2019-03-13T05:47:52.918788Z	info	FLAG: --lightstepCacertPath=""
2019-03-13T05:47:52.918789Z	info	FLAG: --lightstepSecure="false"
2019-03-13T05:47:52.918791Z	info	FLAG: --log_as_json="false"
2019-03-13T05:47:52.918797Z	info	FLAG: --log_caller=""
2019-03-13T05:47:52.918799Z	info	FLAG: --log_output_level="info"
2019-03-13T05:47:52.918801Z	info	FLAG: --log_rotate=""
2019-03-13T05:47:52.918803Z	info	FLAG: --log_rotate_max_age="30"
2019-03-13T05:47:52.918805Z	info	FLAG: --log_rotate_max_backups="1000"
2019-03-13T05:47:52.918807Z	info	FLAG: --log_rotate_max_size="104857600"
2019-03-13T05:47:52.918809Z	info	FLAG: --log_stacktrace_level="default:none"
2019-03-13T05:47:52.918813Z	info	FLAG: --log_target="[stdout]"
2019-03-13T05:47:52.918817Z	info	FLAG: --parentShutdownDuration="1m0s"
2019-03-13T05:47:52.918819Z	info	FLAG: --proxyAdminPort="15000"
2019-03-13T05:47:52.918825Z	info	FLAG: --proxyLogLevel="warning"
2019-03-13T05:47:52.918828Z	info	FLAG: --serviceCluster="istio-ingressgateway"
2019-03-13T05:47:52.918829Z	info	FLAG: --serviceregistry="Kubernetes"
2019-03-13T05:47:52.918831Z	info	FLAG: --statsdUdpAddress=""
2019-03-13T05:47:52.918833Z	info	FLAG: --statusPort="15020"
2019-03-13T05:47:52.918834Z	info	FLAG: --templateFile=""
2019-03-13T05:47:52.918836Z	info	FLAG: --trust-domain=""
2019-03-13T05:47:52.918838Z	info	FLAG: --zipkinAddress="zipkin:9411"
2019-03-13T05:47:52.918848Z	info	Version root@5ceb25bc-4506-11e9-b4f5-0a580a2c0404-docker.io/istio-1.1.0-rc.4-d23daa3d242e51b8fff513f3ac86a708e9bcceba-Clean
2019-03-13T05:47:52.918950Z	info	Obtained private IP [172.17.0.5]
2019-03-13T05:47:52.918994Z	info	Proxy role: &model.Proxy{ClusterID:"", Type:"router", IPAddresses:[]string{"172.17.0.5", "172.17.0.5"}, ID:"istio-ingressgateway-7c8b5f67dc-pphn8.istio-system", Locality:(*core.Locality)(nil), DNSDomain:"istio-system.svc.cluster.local", ConfigNamespace:"", TrustDomain:"cluster.local", Metadata:map[string]string(nil), SidecarScope:(*model.SidecarScope)(nil), ServiceInstances:[]*model.ServiceInstance(nil)}
2019-03-13T05:47:52.919001Z	info	PilotSAN []string(nil)
2019-03-13T05:47:52.919241Z	info	Effective config: binaryPath: /usr/local/bin/envoy
configPath: /etc/istio/proxy
connectTimeout: 10s
discoveryAddress: istio-pilot:15010
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-ingressgateway
statNameLength: 189
tracing:
  zipkin:
    address: zipkin:9411

2019-03-13T05:47:52.919266Z	info	Monitored certs: []envoy.CertSource{envoy.CertSource{Directory:"/etc/certs/", Files:[]string{"cert-chain.pem", "key.pem", "root-cert.pem"}}}
2019-03-13T05:47:52.919289Z	info	PilotSAN []string(nil)
2019-03-13T05:47:52.919327Z	info	Starting proxy agent
2019-03-13T05:47:52.919346Z	info	Opening status port 15020

2019-03-13T05:47:52.919408Z	info	Received new config, resetting budget
2019-03-13T05:47:52.919415Z	info	Reconciling retry (budget 10)
2019-03-13T05:47:52.919423Z	info	Epoch 0 starting
2019-03-13T05:47:52.920750Z	info	Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~172.17.0.5~istio-ingressgateway-7c8b5f67dc-pphn8.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --allow-unknown-fields -l warning]
[2019-03-13 05:47:52.935][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.listener.Filter.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-03-13 05:47:52.935][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.Cluster.hosts'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-03-13 05:47:52.935][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.Cluster.hosts'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-03-13 05:47:52.935][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.api.v2.Cluster.hosts'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-03-13 05:47:52.935][19][warning][misc] [external/envoy/source/common/protobuf/utility.cc:129] Using deprecated option 'envoy.config.trace.v2.Tracing.Http.config'. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-03-13 05:47:52.942][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, no healthy upstream
[2019-03-13 05:47:52.942][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:56] Unable to establish new stream
2019-03-13T05:47:54.309583Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:47:56.308911Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:47:58.307156Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:00.306480Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:02.306478Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2019-03-13 05:48:03.170][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
2019-03-13T05:48:04.306107Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:06.306419Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:08.307967Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:10.308049Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:12.308517Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2019-03-13 05:48:13.275][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
2019-03-13T05:48:14.307630Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:16.307839Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:18.313255Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:20.312621Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:22.307925Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:24.309232Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:26.308613Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:28.306106Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2019-03-13 05:48:28.879][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
2019-03-13T05:48:30.307999Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:32.306720Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:34.306124Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:36.308847Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:38.307669Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:40.308421Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:42.308044Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:44.306672Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:46.307659Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:48.307971Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2019-03-13 05:48:48.601][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers
2019-03-13T05:48:50.308386Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:52.306262Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:54.308398Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:56.307549Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:48:58.309141Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:49:00.307829Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:49:02.306073Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:49:04.307255Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:49:06.307360Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:49:07.891704Z	info	watchFileEvents: "/etc/certs//..2019_03_13_05_49_07.873275507": CREATE
2019-03-13T05:49:07.891760Z	info	watchFileEvents: "/etc/certs//..2019_03_13_05_49_07.873275507": MODIFY|ATTRIB
2019-03-13T05:49:07.892059Z	info	watchFileEvents: "/etc/certs//cert-chain.pem": CREATE
2019-03-13T05:49:07.892088Z	info	watchFileEvents: "/etc/certs//key.pem": CREATE
2019-03-13T05:49:07.892094Z	info	watchFileEvents: "/etc/certs//root-cert.pem": CREATE
2019-03-13T05:49:07.892110Z	info	watchFileEvents: "/etc/certs//..data_tmp": RENAME
2019-03-13T05:49:07.892114Z	info	watchFileEvents: "/etc/certs//..data": CREATE
2019-03-13T05:49:07.892118Z	info	watchFileEvents: "/etc/certs//..2019_03_13_05_47_42.170198611": DELETE
2019-03-13T05:49:08.306797Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:49:10.307371Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:49:12.307077Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-03-13T05:49:14.307371Z	info	Envoy proxy is ready
2019-03-13T05:49:17.893048Z	info	watchFileEvents: notifying

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 17 (8 by maintainers)

Most upvoted comments

Any news?

Same in istio-1.5.1

istio-ingressgateway-57896bd575-w45x5   0/1     Running   0          8m55s
istio-pilot-74c44c647f-n5fmq            1/1     Running   0          8m55s

JCzz on Apr 3, 2020

I had the same issue. Reason was that istio-pilot did not start (Ready 0/1) as it’s increased resource requirements (cpu:500m, memory: 2014Mi) could not get fulfilled. In your case another service might not have started, but pilot consumes in current RC the most resources. You can run from your istio folder “istio\install\kubernetes\helm” the command

helm template istio --name istio --namespace istio-system --values istio/values-istio-demo-auth.yaml --set pilot.resources.requests.cpu=200m,pilot.resources.requests.memory=256Mi | kubectl apply -f -

to reduce the resource requirements.

wailua on Mar 13, 2019

I see this issue too in istio 1.1 rc3. Is there any solution/workaround?

I’ve reduced the CPU and memory utilization of pilot by passing --set pilot.resources.requests.cpu=30m --set pilot.resources.requests.memory=256Mi when installing istio using helm, and still see istio’s ingressgateway and egressgateway pods not running.

See 0/1 below:

$ kubectl get pods -n=istio-system 
NAME                                      READY   STATUS    RESTARTS   AGE
grafana-8688f5d5d5-48s96                  1/1     Running   0          28m
istio-citadel-9f9cb7ddd-nvb9h             1/1     Running   0          28m
istio-egressgateway-7b57ffd59-nv5zp       0/1     Running   0          33m <======
istio-ingressgateway-7fbcc95554-9skd9     0/1     Running   0          33m <======
istio-pilot-7f474897d6-pgqf7              2/2     Running   0          33m
istio-policy-6bd4dc558-82lwp              2/2     Running   0          33m
istio-sidecar-injector-64f7b875c9-xjg9m   1/1     Running   1          28m
istio-telemetry-5cd4965f7b-ggl9n          2/2     Running   0          33m
prometheus-b557699f6-bhv2s                1/1     Running   0          28m

Pilot’s reduced CPU and memory utilization:

$ kubectl describe pod istio-pilot-7f474897d6-pgqf7 -n=istio-system
...
    Requests:
      cpu:      30m
      memory:   256Mi
...

$ kubectl get pod istio-pilot-7f474897d6-pgqf7 -n=istio-system  -o yaml
...
    resources:
      requests:
        cpu: 30m
        memory: 256Mi
...

I see a lot of the following messages in the pod logs of istio-egressgateway-7b57ffd59-nv5zp and istio-ingressgateway-7fbcc95554-9skd9:

[2019-04-01 21:58:13.279][27][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:102] gRPC config stream closed: 14, no healthy upstream
[2019-04-01 21:58:13.279][27][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:56] Unable to establish new stream
2019-04-01T21:58:14.233230Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-04-01T21:58:16.233123Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

Also, I see calico and kube-proxy pods crashing when istio 1.1 rc3 is installed:

$ kubectl get pods --all-namespaces
NAMESPACE      NAME               READY   STATUS              RESTARTS   AGE
...
kube-system    calico-node-bdfb2  0/1     CrashLoopBackOff    5          16m        
kube-system    kube-proxy-nx7jz   0/1     ContainerCreating   0          1s         
...

But, after I delete istio and its CRDs (see below), both calico and kube-proxy will be up. Looks like istio 1.1 rc3 is crashing calico and kube-proxy.

helm delete --purge istio
helm delete --purge istio-init
kubectl get crds --all-namespaces | \
    grep -i 'istio\.io' | awk '{print $1}' | xargs kubectl delete crd
kubectl delete -n=istio-system job --all
# wait 2-3 minutes and calico and kube-proxy will be up at this point

The k8s nodes are ubuntu VMs with 2 vCPUs and 12 GB memory.

vhosakot on Apr 1, 2019