minikube: Istio addon not working on multinode
When trying to run istio on a multi node minikube the istio-proxy containers do not start.
Steps to reproduce the issue:
- minikube start --nodes=4 --cpus=4 --memory=8G
- minikube addons enable istio-provisioner
- minikube addons enable istio
Full output of minikube start command used, if not already included:
minikube-start-nodes4-cpus4-memory8g.log
minikube-addons-enable-istio-provisioner.log
minikube-addons-enable-istio.log
Full output of minikube logs command:
minikube-logs.log
Full container logs from istio-operator and istio-system namespaces:
istio-operator-istio-operator-6dbfd4446f-xxqnn-1611512011468363254.log istio-system-istiod-6ccd677dc7-gmzsw-1611512018108421785.log istio-system-istio-ingressgateway-8577c95547-v25xk-1611512014880656996.log istio-system-prometheus-7767dfd55-kd9c8-1611512021807342019.log istio-system-prometheus-7767dfd55-kd9c8-1611512024337276220.log
Relevant Logs from istio-proxy container:
2021-01-24T18:11:40.046139Z info FLAG: --binaryPath="/usr/local/bin/envoy"
2021-01-24T18:11:40.046165Z info FLAG: --concurrency="0"
2021-01-24T18:11:40.046167Z info FLAG: --configPath="/etc/istio/proxy"
2021-01-24T18:11:40.046170Z info FLAG: --connectTimeout="10s"
2021-01-24T18:11:40.046172Z info FLAG: --controlPlaneAuthPolicy="NONE"
2021-01-24T18:11:40.046175Z info FLAG: --controlPlaneBootstrap="true"
2021-01-24T18:11:40.046177Z info FLAG: --customConfigFile=""
2021-01-24T18:11:40.046178Z info FLAG: --datadogAgentAddress=""
2021-01-24T18:11:40.046180Z info FLAG: --disableInternalTelemetry="false"
2021-01-24T18:11:40.046182Z info FLAG: --discoveryAddress="istio-pilot.istio-system.svc:15012"
2021-01-24T18:11:40.046184Z info FLAG: --dnsRefreshRate="300s"
2021-01-24T18:11:40.046191Z info FLAG: --domain="istio-system.svc.cluster.local"
2021-01-24T18:11:40.046193Z info FLAG: --drainDuration="45s"
2021-01-24T18:11:40.046195Z info FLAG: --envoyAccessLogService=""
2021-01-24T18:11:40.046196Z info FLAG: --envoyMetricsService=""
2021-01-24T18:11:40.046198Z info FLAG: --help="false"
2021-01-24T18:11:40.046199Z info FLAG: --id=""
2021-01-24T18:11:40.046201Z info FLAG: --ip=""
2021-01-24T18:11:40.046203Z info FLAG: --lightstepAccessToken=""
2021-01-24T18:11:40.046204Z info FLAG: --lightstepAddress=""
2021-01-24T18:11:40.046206Z info FLAG: --lightstepCacertPath=""
2021-01-24T18:11:40.046208Z info FLAG: --lightstepSecure="false"
2021-01-24T18:11:40.046210Z info FLAG: --log_as_json="false"
2021-01-24T18:11:40.046212Z info FLAG: --log_caller=""
2021-01-24T18:11:40.046214Z info FLAG: --log_output_level="default:info"
2021-01-24T18:11:40.046215Z info FLAG: --log_rotate=""
2021-01-24T18:11:40.046218Z info FLAG: --log_rotate_max_age="30"
2021-01-24T18:11:40.046220Z info FLAG: --log_rotate_max_backups="1000"
2021-01-24T18:11:40.046223Z info FLAG: --log_rotate_max_size="104857600"
2021-01-24T18:11:40.046225Z info FLAG: --log_stacktrace_level="default:none"
2021-01-24T18:11:40.046229Z info FLAG: --log_target="[stdout]"
2021-01-24T18:11:40.046231Z info FLAG: --mixerIdentity=""
2021-01-24T18:11:40.046233Z info FLAG: --outlierLogPath=""
2021-01-24T18:11:40.046235Z info FLAG: --parentShutdownDuration="1m0s"
2021-01-24T18:11:40.046237Z info FLAG: --pilotIdentity=""
2021-01-24T18:11:40.046239Z info FLAG: --proxyAdminPort="15000"
2021-01-24T18:11:40.046241Z info FLAG: --proxyComponentLogLevel="misc:error"
2021-01-24T18:11:40.046243Z info FLAG: --proxyLogLevel="warning"
2021-01-24T18:11:40.046245Z info FLAG: --serviceCluster="istio-ingressgateway"
2021-01-24T18:11:40.046247Z info FLAG: --serviceregistry="Kubernetes"
2021-01-24T18:11:40.046249Z info FLAG: --statsdUdpAddress=""
2021-01-24T18:11:40.046251Z info FLAG: --statusPort="15020"
2021-01-24T18:11:40.046253Z info FLAG: --stsPort="0"
2021-01-24T18:11:40.046255Z info FLAG: --templateFile=""
2021-01-24T18:11:40.046257Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2021-01-24T18:11:40.046259Z info FLAG: --trust-domain="cluster.local"
2021-01-24T18:11:40.046261Z info FLAG: --zipkinAddress="zipkin.istio-system:9411"
2021-01-24T18:11:40.046283Z info Version 1.5.0-c3c353285578eb68b334fc8766746b754b6b3789-Clean
2021-01-24T18:11:40.046388Z info Obtained private IP [10.244.3.3]
2021-01-24T18:11:40.046412Z info Proxy role: &model.Proxy{ClusterID:"", Type:"router", IPAddresses:[]string{"10.244.3.3", "10.244.3.3"}, ID:"istio-ingressgateway-8577c95547-v25xk.istio-system", Locality:(*envoy_api_v2_core.Locality)(nil), DNSDomain:"istio-system.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), WorkloadLabels:labels.Collection(nil), IstioVersion:(*model.IstioVersion)(nil)}
2021-01-24T18:11:40.046416Z info PilotSAN []string(nil)
2021-01-24T18:11:40.046418Z info MixerSAN []string(nil)
2021-01-24T18:11:40.046742Z info Effective config: binaryPath: /usr/local/bin/envoy
configPath: /etc/istio/proxy
connectTimeout: 10s
discoveryAddress: istio-pilot.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-ingressgateway
statNameLength: 189
tracing:
zipkin:
address: zipkin.istio-system:9411
2021-01-24T18:11:40.046751Z info JWT policy is third-party-jwt
2021-01-24T18:11:40.046806Z info Using user-configured CA istio-pilot.istio-system.svc:15012
2021-01-24T18:11:40.046808Z info istiod uses self-issued certificate
2021-01-24T18:11:40.046836Z info the CA cert of istiod is: -----BEGIN CERTIFICATE-----
MIIC3jCCAcagAwIBAgIRAP3mz2eowkXT6AkWrglnGJQwDQYJKoZIhvcNAQELBQAw
GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMTAxMjQxODExMDZaFw0zMTAx
MjIxODExMDZaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQCV6xd981Y2QyhbxLohoqh3Vidxkr3V+uJd70Wg
zKJFEepQrkLx6sL27BHRqmSfB/po6CTVYmNUFHES9z+XQPQ8DFJtg446+JlwT5u9
BVLGpuskdb2dp/YdT0ZeXSAETYMOlsT6zjK3zAWATgDGYxif8yu80jwy/DvTzIDO
VRi2yIyjKTNT80G4i9M3iYjdHwGaFdFlek3ZbskhGpe75R84uXE9zR1OS+HkDTN5
FVZiENmEI62Qz6BXpmHTMX4Oy9u1nvaDFauuyxlFSdSPnm1296Pz5kesNAyB7F+r
fsTEpjMtf9NNcK6NGeHVzVAWNHqWk4tY1L1RnSflaE6TAo2HAgMBAAGjIzAhMA4G
A1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IB
AQAxMxbEqwK3pr5wzp6lTPQlXn4gZjMlUFhPlecFmo+6J9SYH07tDvY5LdAhEGE/
fVES2AHTKV7KZa5AxXCRqFqShD51l68aCyg2N1w0Mrev5H3KjT0UuxVDaJ/M09Na
m+VIT1VsPiEkPBdFj+inSYQP0BWsMBXhe1Mg9oJyXZCKzNDHYCMoxi5Vde3KGZ6L
Si+2l7kd3zwssIzgF8LaUqwcN0WC0fl/zgH/JYo+1hz/kcMGj2K6zcOjsvXAyZ/0
hi2T4ZUL/DV+1SBb1Q9hLLs1vzUsmennxuEC940fXc3krMw3bKtRtG0R8bgcWbPb
wCPIQvsx2EmSwdtSYOqof9Wv
-----END CERTIFICATE-----
2021-01-24T18:11:40.046922Z info parsed scheme: ""
2021-01-24T18:11:40.046927Z info scheme "" not registered, fallback to default scheme
2021-01-24T18:11:40.046938Z info ccResolverWrapper: sending update to cc: {[{istio-pilot.istio-system.svc:15012 <nil> 0 <nil>}] <nil> <nil>}
2021-01-24T18:11:40.046942Z info ClientConn switching balancer to "pick_first"
2021-01-24T18:11:40.047057Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING <nil>}
2021-01-24T18:11:40.085873Z info Starting gateway SDS
2021-01-24T18:11:40.187466Z info sds SDS gRPC server for workload UDS starts, listening on "/etc/istio/proxy/SDS"
2021-01-24T18:11:40.187542Z info sds SDS gRPC server for ingress gateway controller starts, listening on "/var/run/ingress_gateway/sds"
2021-01-24T18:11:40.187531Z info sds Start SDS grpc server
2021-01-24T18:11:40.187567Z info sds Start SDS grpc server for ingress gateway proxy
2021-01-24T18:11:40.187571Z info PilotSAN []string{"istio-pilot.istio-system.svc"}
2021-01-24T18:11:40.187621Z info Starting proxy agent
2021-01-24T18:11:40.187662Z info Opening status port 15020
2021-01-24T18:11:40.187693Z info Received new config, creating new Envoy epoch 0
2021-01-24T18:11:40.187738Z info Epoch 0 starting
2021-01-24T18:11:41.406363Z info Envoy proxy is NOT ready: Get http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager.cds|listener_manager.lds).(update_success|update_rejected)$: dial tcp 127.0.0.1:15000: connect: connection refused
2021-01-24T18:11:42.189868Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning --component-log-level misc:error]
[Envoy (Epoch 0)] [2021-01-24 18:11:42.224][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:11:42.224][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:11:43.406922Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:11:43.589][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:11:43.589][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:11:45.406869Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:11:46.011][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:11:46.011][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:11:47.406933Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:11:47.444][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:11:47.444][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:11:49.406768Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:11:51.407092Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:11:53.406907Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:11:55.406834Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:11:57.407212Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:11:59.406811Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:00.047149Z info grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012 <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
2021-01-24T18:12:00.047245Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"}
2021-01-24T18:12:01.047412Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING <nil>}
2021-01-24T18:12:01.406750Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:03.406980Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:05.407071Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:12:07.274][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:12:07.274][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:12:07.406742Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:09.406889Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:11.406554Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:13.406755Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:15.406844Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:17.406537Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:19.406795Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:21.047495Z info grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012 <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
2021-01-24T18:12:21.047542Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"}
2021-01-24T18:12:21.406880Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:22.813360Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING <nil>}
2021-01-24T18:12:23.406873Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:25.406651Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:27.406715Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:29.406651Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:31.406598Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:12:32.515][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:12:32.515][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:12:33.406663Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:35.407079Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:12:35.718][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:12:35.718][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:12:37.407151Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:39.406682Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:41.406570Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:12:42.660][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:12:42.660][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:12:42.813440Z info grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012 <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
2021-01-24T18:12:42.813483Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"}
2021-01-24T18:12:43.406825Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:45.406872Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:45.857720Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING <nil>}
2021-01-24T18:12:47.407005Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:49.406834Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:51.406617Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:53.407196Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:55.406961Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:57.228538Z info sds node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default new connection
2021-01-24T18:12:57.407219Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:12:59.406660Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:00.903701Z info grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012 <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout". Reconnecting...
2021-01-24T18:13:00.903785Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"}
2021-01-24T18:13:00.903816Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:00.953958Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 50 millisec
2021-01-24T18:13:00.954081Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:01.054248Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 100 millisec
2021-01-24T18:13:01.054333Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:01.254467Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 200 millisec
2021-01-24T18:13:01.254537Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:01.406843Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:01.654701Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 400 millisec
2021-01-24T18:13:01.654778Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:02.454924Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 800 millisec
2021-01-24T18:13:02.455019Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:03.406835Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:04.055157Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 1600 millisec
2021-01-24T18:13:04.055244Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"
2021-01-24T18:13:04.734418Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING <nil>}
2021-01-24T18:13:05.406606Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[Envoy (Epoch 0)] [2021-01-24 18:13:07.019][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream
[Envoy (Epoch 0)] [2021-01-24 18:13:07.019][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream
2021-01-24T18:13:07.255368Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 3200 millisec
2021-01-24T18:13:07.406709Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:09.406937Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:11.406790Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:13.407097Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:15.406754Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:17.407000Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:19.406913Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:21.406575Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:23.406549Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:24.734524Z info grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012 <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
2021-01-24T18:13:24.734584Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"}
2021-01-24T18:13:24.734613Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.734637Z error cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.734668Z error cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.734684Z error sds node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default Close connection. Failed to get secret for proxy "router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.734770Z info sds node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
[Envoy (Epoch 0)] [2021-01-24 18:13:24.734][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.776637Z info sds node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default new connection
2021-01-24T18:13:24.846060Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.896173Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 50 millisec
2021-01-24T18:13:24.896212Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:24.996341Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 100 millisec
2021-01-24T18:13:24.996435Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:25.196564Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 200 millisec
2021-01-24T18:13:25.196607Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:25.406711Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:25.596787Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 400 millisec
2021-01-24T18:13:25.596889Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:26.397058Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 800 millisec
2021-01-24T18:13:26.397135Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:27.406821Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:27.997277Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 1600 millisec
2021-01-24T18:13:27.997362Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-01-24T18:13:29.406690Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:31.021823Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING <nil>}
2021-01-24T18:13:31.197481Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 3200 millisec
2021-01-24T18:13:31.406707Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2021-01-24T18:13:33.406815Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (5 by maintainers)
I’m running into the same issue. Is this possible with istio and minikube?