istio: Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?)

(NOTE: This is used to report product bugs: To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities/ To ask questions about how to use Istio, please visit https://discuss.istio.io )

Bug description I freshly installed Istio 1.3.1 with SDS example on the GKE cluster by following the instructions over here: https://istio.io/docs/setup/install/helm/

I see the following logs in the istio-proxy sidecar:

concurrency: 2 configPath: /etc/istio/proxy connectTimeout: 10s controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istio-pilot.istio-system:15011 drainDuration: 45s envoyAccessLogService: {} envoyMetricsService: {} parentShutdownDuration: 60s proxyAdminPort: 15000 serviceCluster: sleep.foo statNameLength: 189 tracing: zipkin: address: zipkin.istio-system:9411

2019-10-02T01:06:58.831477Z info waiting 1m0s for /var/run/sds/uds_path 2019-10-02T01:06:58.831505Z info PilotSAN []string{“spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account”} 2019-10-02T01:06:58.831664Z info Opening status port 15020

2019-10-02T01:06:58.831865Z info Starting proxy agent 2019-10-02T01:06:58.832175Z warn watching /etc/certs encountered an error no such file or directory 2019-10-02T01:06:58.832193Z info Received new config, resetting budget 2019-10-02T01:06:58.832198Z info Reconciling retry (budget 10) 2019-10-02T01:06:58.832206Z info Epoch 0 starting 2019-10-02T01:06:58.844126Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster sleep.foo --service-node sidecar~10.36.4.5~sleep-68dd4965b8-dqh6p.foo~foo.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error --concurrency 2] [2019-10-02 01:06:58.863][11][warning][config] [external/envoy/source/server/options_impl.cc:193] --allow-unknown-fields is deprecated, use --allow-unknown-static-fields instead. [2019-10-02 01:06:58.885][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream [2019-10-02 01:06:58.885][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream [2019-10-02 01:06:59.040][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure [2019-10-02 01:06:59.221][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream [2019-10-02 01:06:59.221][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream [2019-10-02 01:06:59.623][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure [2019-10-02 01:06:59.914][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to get root cert 2019-10-02T01:07:00.971189Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [2019-10-02 01:07:01.293][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to get root cert [2019-10-02 01:07:01.346][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure [2019-10-02 01:07:02.462][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to get root cert 2019-10-02T01:07:02.970964Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [2019-10-02 01:07:03.159][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure 2019-10-02T01:07:04.986559Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [2019-10-02 01:07:05.015][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure [2019-10-02 01:07:06.726][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to get root cert 2019-10-02T01:07:06.970770Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2019-10-02T01:07:08.970709Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2019-10-02T01:07:10.970818Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [2019-10-02 01:07:12.056][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 2, failed to get root cert 2019-10-02T01:07:12.970912Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [2019-10-02 01:07:14.620][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 16, request authenticate failure 2019-10-02T01:07:14.970789Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [2019-10-02 01:07:15.555][11][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure

Affected product area (please put an X in all that apply)

[ x] Configuration Infrastructure [ ] Docs [x ] Installation [ ] Networking [ ] Performance and Scalability [ ] Policies and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Expected behavior The installation shopuld be successful. Steps to reproduce the bug Steps are mentioned here: https://istio.io/docs/setup/install/helm/#option-1-install-with-helm-via-helm-template Version (include the output of istioctl version --remote and kubectl version)

:istio-1.3.1$ istioctl version --remote client version: 1.3.1 citadel version: 1.3.1 galley version: 1.3.1 ingressgateway version: 1.3.1 nodeagent version: nodeagent version: nodeagent version: nodeagent version: nodeagent version: pilot version: 1.3.1 policy version: 1.3.1 sidecar-injector version: 1.3.1

kubectl version Client Version: version.Info{Major:“1”, Minor:“11”, GitVersion:“v1.11.0”, GitCommit:“91e7b4fd31fcd3d5f436da26c980becec37ceefe”, GitTreeState:“clean”, BuildDate:“2018-06-27T20:17:28Z”, GoVersion:“go1.10.2”, Compiler:“gc”, Platform:“darwin/amd64”} Server Version: version.Info{Major:“1”, Minor:“12+”, GitVersion:“v1.12.8-gke.10”, GitCommit:“f53039cc1e5295eed20969a4f10fb6ad99461e37”, GitTreeState:“clean”, BuildDate:“2019-06-19T20:48:40Z”, GoVersion:“go1.10.8b4”, Compiler:“gc”, Platform:“linux/amd64”} :istio-1.3.1$

How was Istio installed? Istio was installed through the steps given here: https://istio.io/docs/setup/install/helm/#option-1-install-with-helm-via-helm-template Environment where bug was observed (cloud vendor, OS, etc) GKE Additionally, please consider attaching a cluster state archive by attaching the dump file to this issue.