istio: Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?)

(This is used to report product bugs, please visit https://discuss.istio.io for questions on using Istio)

Bug description Envoy proxy takes a long time to get ready. 2019-05-17T11:51:20.184647Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 2 successful, 0 rejected; lds updates: 0 successful, 0 rejected Above message comes repeatedly for a while. Even after envoy is ready, we get 503 service unavailable and the below log in the istio proxy of the pod. [2019-05-17T12:04:26.236Z] "GET /upload/static/up.html?user=blahblah HTTP/1.1" 503 UAEX "UNAVAILABLE:Cluster not available" 0 33 0 - "10.148.0.56" "Wget/1.20.1 (linux-gnu)" "3d8f7289-0004-9fb5-ba10-c408ad7f03c3" "35.247.147.85" "-" - - 10.8.59.5:80 10.148.0.56:0 outbound_.80_._.cs-upload.cs1.svc.cluster.local Application layer works perfectly since inside the pod, queries works fine.

Expected behavior no 503 envoy should get ready faster

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version) version.BuildInfo{Version:“1.1.5”, GitRevision:“9b6d31b74d1c0cc9358cc82d395b53f71393326b”, User:“root”, Host:“3e29fde4-6c3f-11e9-b00d-0a580a2c0205”, GolangVersion:“go1.10.4”, DockerHub:“docker.io/istio”, BuildStatus:“Clean”, GitTag:“1.1.4-10-g9b6d31b”} Error: error execing into istio-pilot-54758fb779-cj5pn discovery container: unable to upgrade connection: container not found (“discovery”)

Client Version: version.Info{Major:“1”, Minor:“12”, GitVersion:“v1.12.7”, GitCommit:“6f482974b76db3f1e0f5d24605a9d1d38fad9a2b”, GitTreeState:“clean”, BuildDate:“2019-03-25T02:52:13Z”, GoVersion:“go1.10.8”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“12+”, GitVersion:“v1.12.7-gke.10”, GitCommit:“8d9b8641e72cf7c96efa61421e87f96387242ba1”, GitTreeState:“clean”, BuildDate:“2019-04-12T22:59:24Z”, GoVersion:“go1.10.8b4”, Compiler:“gc”, Platform:“linux/amd64”}

How was Istio installed? using demo-auth approach

Environment where bug was observed (cloud vendor, OS, etc) GKE in GCP

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure [ ] Docs [ ] Installation [ ] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [ ] User Experience

Additionally, please consider attaching a cluster state archive by attaching the dump file to this issue.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 18 (6 by maintainers)

Most upvoted comments

I have the same problem

2019-09-28T15:41:49.548538Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
...
...
2019-09-28T15:47:19.549043Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-28T15:47:21.548349Z	info	Envoy proxy is ready
+36
lou-lan on Sep 28, 2019

I have the same problem

2019-09-28T15:41:49.548538Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
...
...
2019-09-28T15:47:19.549043Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-28T15:47:21.548349Z	info	Envoy proxy is ready

same error here. I found out it was my coredns didn’t start up, no nodes match its node selector. I solved this problem by fix the coredns pod.

+10
ryan4yin on Dec 5, 2021

I had the same error “Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?)”. my scenario is I created two separate ingress gateway in two different namespaces. Like below

Namespace cicdapi-testing
-  cicdapi-testing-ingress-gateway
Namespace cicdapi-production
-  cicdapi-production-ingress-gateway
Namespace istio-system
-  istio-ingress-gateway

And I keep getting this error. It’s pods of ingress gateway unable to connect to pilot in istio-system. my istio version 1.1.8

[2019-07-01 09:30:22.970][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 14, no healthy upstream
[2019-07-01 09:30:22.970][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:49] Unable to establish new stream
2019-07-01T09:30:24.492920Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-07-01T09:30:26.492774Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-07-01T09:30:28.492827Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-07-01T09:30:30.493626Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-07-01T09:30:32.492754Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

And i checked the logs from discovery and below i replace IP with <IP>.

2019-07-01T08:39:35.436665Z	info	ads	Cluster init time 293.85µs 2019-07-01T08:37:12Z/26
2019-07-01T08:39:35.436816Z	info	ads	PushAll done 2019-07-01T08:37:12Z/26 119.478µs
2019-07-01T08:39:35.495954Z	info	Handle EDS endpoint cicdapi-ingressgateway in namespace cicdapi-testing -> [] []
2019-07-01T08:39:35.616065Z	info	Handling event update for pod cicdapi-ingressgateway-77bbc77f9b-p9cz8 in namespace cicdapi-production -> <IP>
2019-07-01T08:39:35.638679Z	info	Handle EDS endpoint cicdapi-ingressgateway in namespace cicdapi-production -> [{[] [{<IP>  0xc4201c8e30 &ObjectReference{Kind:Pod,Namespace:cicdapi-production,Name:cicdapi-ingressgateway-77bbc77f9b-p9cz8,UID:c0934420-9bdb-11e9-8ac7-42010a8c0113,APIVersion:,ResourceVersion:251245,FieldPath:,}}] [{https-grafana 15031 TCP} {https-kiali 15029 TCP} {http2 80 TCP} {tcp 31400 TCP} {https-tracing 15032 TCP} {https-prometheus 15030 TCP} {https 443 TCP}]}] []
2019-07-01T08:39:36.287016Z	info	Handling event update for pod cicdapi-ingressgateway-db64b6495-dnns4 in namespace cicdapi-testing -> 10.48.4.21
2019-07-01T08:39:36.290230Z	info	Handle EDS endpoint cicdapi-ingressgateway in namespace cicdapi-testing -> [{[] [{10.48.4.21  0xc4201c9a50 &ObjectReference{Kind:Pod,Namespace:cicdapi-testing,Name:cicdapi-ingressgateway-db64b6495-dnns4,UID:c1376182-9bdb-11e9-8ac7-42010a8c0113,APIVersion:,ResourceVersion:251250,FieldPath:,}}] [{https-grafana 15031 TCP} {https-kiali 15029 TCP} {http2 80 TCP} {tcp 31400 TCP} {https-tracing 15032 TCP} {https-prometheus 15030 TCP} {https 443 TCP}]}] []
2019-07-01T08:39:37.967674Z	info	Handling event update for pod cicdapi-ingressgateway-77bbc77f9b-sxh2f in namespace cicdapi-production -> <IP>
2019-07-01T08:39:37.977815Z	info	Handle EDS endpoint cicdapi-ingressgateway in namespace cicdapi-production -> [{[] [{<IP>  0xc4209f9b70 &ObjectReference{Kind:Pod,Namespace:cicdapi-production,Name:cicdapi-ingressgateway-77bbc77f9b-sxh2f,UID:c1fa61b0-9bdb-11e9-8ac7-42010a8c0113,APIVersion:,ResourceVersion:251279,FieldPath:,}} {<IP>  0xc4209f9bb0 &ObjectReference{Kind:Pod,Namespace:cicdapi-production,Name:cicdapi-ingressgateway-77bbc77f9b-p9cz8,UID:c0934420-9bdb-11e9-8ac7-42010a8c0113,APIVersion:,ResourceVersion:251245,FieldPath:,}}] [{https-grafana 15031 TCP} {https-kiali 15029 TCP} {http2 80 TCP} {tcp 31400 TCP} {https-tracing 15032 TCP} {https-prometheus 15030 TCP} {https 443 TCP}]}] []
2019-07-01T08:39:37.988493Z	info	Handling event update for pod cicdapi-ingressgateway-db64b6495-h96p4 in namespace cicdapi-testing -> <IP>
2019-07-01T08:39:37.994699Z	info	Handle EDS endpoint cicdapi-ingressgateway in namespace cicdapi-testing -> [{[] [{<IP>  0xc420fb8290 &ObjectReference{Kind:Pod,Namespace:cicdapi-testing,Name:cicdapi-ingressgateway-db64b6495-h96p4,UID:c21930a7-9bdb-11e9-8ac7-42010a8c0113,APIVersion:,ResourceVersion:251281,FieldPath:,}} {<IP>  0xc420fb82a0 &ObjectReference{Kind:Pod,Namespace:cicdapi-testing,Name:cicdapi-ingressgateway-db64b6495-dnns4,UID:c1376182-9bdb-11e9-8ac7-42010a8c0113,APIVersion:,ResourceVersion:251250,FieldPath:,}}] [{https-grafana 15031 TCP} {https-kiali 15029 TCP} {http2 80 TCP} {tcp 31400 TCP} {https-tracing 15032 TCP} {https-prometheus 15030 TCP} {https 443 TCP}]}] []
2019-07-01T08:39:39.653581Z	info	Handling event update for pod cicdapi-ingressgateway-66875f6b58-dpjjt in namespace cicdapi-production -> <IP>
gc 99 @6028.999s 0%: 0.025+4.6+0.092 ms clock, 0.10+0.99/4.4/7.7+0.37 ms cpu, 11->11->6 MB, 12 MB goal, 4 P
2019-07-01T08:39:39.676450Z	info	Handling event update for pod cicdapi-ingressgateway-66875f6b58-dpjjt in namespace cicdapi-production -> <IP>
2019-07-01T08:39:40.322704Z	info	Handling event update for pod cicdapi-ingressgateway-66875f6b58-r27bc in namespace cicdapi-production -> <IP>
2019-07-01T08:39:41.337247Z	info	Handling event update for pod cicdapi-ingressgateway-66875f6b58-r27bc in namespace cicdapi-production -> <IP>
2019-07-01T08:39:41.345404Z	info	Handling event delete for pod cicdapi-ingressgateway-66875f6b58-r27bc in namespace cicdapi-production -> <IP>
2019-07-01T08:39:41.367773Z	info	Handling event update for pod cicdapi-ingressgateway-77497485d9-sv8g2 in namespace cicdapi-testing -> <IP>
2019-07-01T08:39:41.475488Z	info	Handling event update for pod cicdapi-ingressgateway-77497485d9-4b257 in namespace cicdapi-testing -> <IP>
2019-07-01T08:39:44.855931Z	info	Handling event update for pod cicdapi-ingressgateway-66875f6b58-dpjjt in namespace cicdapi-production -> <IP>
2019-07-01T08:39:44.870603Z	info	Handling event delete for pod cicdapi-ingressgateway-66875f6b58-dpjjt in namespace cicdapi-production -> <IP>
2019-07-01T08:39:49.030800Z	info	Handling event update for pod cicdapi-ingressgateway-77497485d9-4b257 in namespace cicdapi-testing -> <IP>
2019-07-01T08:39:49.039517Z	info	Handling event delete for pod cicdapi-ingressgateway-77497485d9-4b257 in namespace cicdapi-testing -> <IP>
2019-07-01T08:39:50.770459Z	info	Handling event update for pod cicdapi-ingressgateway-77497485d9-sv8g2 in namespace cicdapi-testing -> <IP>
2019-07-01T08:39:50.779087Z	info	Handling event delete for pod cicdapi-ingressgateway-77497485d9-sv8g2 in namespace cicdapi-testing -> <IP>
gc 100 @6099.913s 0%: 0.12+4.0+0.72 ms clock, 0.51+0.67/3.8/6.5+2.8 ms cpu, 11->11->6 MB, 12 MB goal, 4 P
scvg40: 1 MB released
scvg40: inuse: 13, idle: 2, sys: 16, released: 2, consumed: 14 (MB)

I don’t know why it can’t find pilot and how can i fix it ?

+1
ishallbethat on Jul 1, 2019

Is this related to this issue I raised? #14095

We are seeing similar in our upgraded cluster, Sidecars will disconnect from Pilot after a while and never reconnect until pilot is restarted

+1
johne9087 on May 17, 2019
Read more comments on GitHub
← v8-to-istanbul: Cannot read properties of undefined (reading 'endCol')
istio: Exposing jager via ingress gateway gives strange errors with http2 - Envoy breaks http1.1 <-> http2 ALPN →