istio: etcd not working

Bug Description

i deploy etcd in k8s out of istio cluster, other service in istio cluster can not connect etcd, below is logging:

istio-proxy logging: $ kubectl logs -f admin-659864d95c-z9wnt -c istio-proxy -n content-security-mesh [2021-09-15T09:51:24.137Z] “- - HTTP/2” 0 DPE http2.invalid.header.field - “-” 0 0 0 - “-” “-” “-” “-” “-” - - 10.96.181.119:2379 10.100.219.87:56256 - - [2021-09-15T09:51:25.162Z] “- - HTTP/2” 0 DPE http2.invalid.header.field - “-” 0 0 0 - “-” “-” “-” “-” “-” - - 10.96.181.119:2379 10.100.219.87:56290 - - [2021-09-15T09:51:26.191Z] “- - HTTP/2” 0 DPE http2.invalid.header.field - “-” 0 0 0 - “-” “-” “-” “-” “-” - - 10.96.181.119:2379 10.100.219.87:56334 - - [2021-09-15T09:51:27.216Z] “- - HTTP/2” 0 DPE http2.invalid.header.field - “-” 0 0 0 - “-” “-” “-” “-” “-” - - 10.96.181.119:2379 10.100.219.87:56358 - - [2021-09-15T09:51:28.244Z] “- - HTTP/2” 0 DPE http2.invalid.header.field - “-” 0 0 0 - “-” “-” “-” “-” “-” - - 10.96.181.119:2379 10.100.219.87:56384 - - [2021-09-15T09:51:29.271Z] “- - HTTP/2” 0 DPE http2.invalid.header.field - “-” 0 0 0 - “-” “-” “-” “-” “-” - - 10.96.181.119:2379 10.100.219.87:56472 - -

service logging: $ k logs -f admin-659864d95c-z9wnt -n content-security-mesh {“level”:“warn”,“ts”:“2021-09-15T17:51:33.349+0800”,“logger”:“etcd-client”,“caller”:“v3@v3.5.0-rc.1/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc0003c7880/#initially=[etcd-client.infra:2379]”,“attempt”:96,“error”:“rpc error: code = Unavailable desc = error reading from server: EOF”} {“level”:“warn”,“ts”:“2021-09-15T17:51:34.376+0800”,“logger”:“etcd-client”,“caller”:“v3@v3.5.0-rc.1/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc0003c7880/#initially=[etcd-client.infra:2379]”,“attempt”:97,“error”:“rpc error: code = Unavailable desc = error reading from server: EOF”} {“level”:“warn”,“ts”:“2021-09-15T17:51:35.403+0800”,“logger”:“etcd-client”,“caller”:“v3@v3.5.0-rc.1/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc0003c7880/#initially=[etcd-client.infra:2379]”,“attempt”:98,“error”:“rpc error: code = Unavailable desc = error reading from server: EOF”} {“level”:“warn”,“ts”:“2021-09-15T17:51:36.428+0800”,“logger”:“etcd-client”,“caller”:“v3@v3.5.0-rc.1/retry_interceptor.go:62”,“msg”:“retrying of unary invoker failed”,“target”:“etcd-endpoints://0xc0003c7880/#initially=[etcd-client.infra:2379]”,“attempt”:99,“error”:“rpc error: code = Unavailable desc = error reading from server: EOF”}

Version

❯ istioctl version
client version: 1.11.2
control plane version: 1.11.2
data plane version: 1.11.2 (6 proxies)

Additional Information

Target cluster context: k-113

Running with the following config:

istio-namespace: istio-system full-secrets: false timeout (mins): 30 include: { } exclude: { Namespaces: kube-system, kube-public, kube-node-lease, local-path-storage } AND { Namespaces: kube-system, kube-public, kube-node-lease, local-path-storage } end-time: 2021-09-15 17:50:08.300589 +0800 CST

The following Istio control plane revisions/versions were found in the cluster: Revision default: &version.MeshInfo{ { Component: “pilot”, Info: version.BuildInfo{Version:“1.11.2”, GitRevision:“96710172e1e47cee227e7e8dd591a318fdfe0326”, GolangVersion:“”, BuildStatus:“Clean”, GitTag:“1.11.2”}, }, }

The following proxy revisions/versions were found in the cluster: Revision default: Versions {1.11.2}

Fetching proxy logs for the following containers:

bookstore-microservices/bookstore-microservices-domain-account/bookstore-microservices-domain-account-688f5878df-qf8tw/account bookstore-microservices/bookstore-microservices-domain-payment/bookstore-microservices-domain-payment-b55c4d9c8-66bnh/payment bookstore-microservices/bookstore-microservices-domain-security/bookstore-microservices-domain-security-745b47dcdb-dlg6l/security bookstore-microservices/bookstore-microservices-domain-warehouse/bookstore-microservices-domain-warehouse-cc4c86fb5-s4zf4/warehouse bookstore-microservices/bookstore-microservices-platform-gateway/bookstore-microservices-platform-gateway-56c5bf7b5-9h5x6/gateway content-security-mesh/admin/admin-659864d95c-z9wnt/admin content-security-mesh/admin/admin-659864d95c-z9wnt/istio-proxy content-security-mesh/master/master-67f65c944f-x2nmj/istio-proxy content-security-mesh/master/master-67f65c944f-x2nmj/master content-security-mesh/security/security-d8957d449-mwwrq/istio-proxy content-security-mesh/security/security-d8957d449-mwwrq/security content-security-mesh/worker/worker-868665f777-877cl/istio-proxy content-security-mesh/worker/worker-868665f777-877cl/worker default//etcd-0/etcd default//rebbitmq-rabbitmq-ha-0/rabbitmq-ha default//rebbitmq-rabbitmq-ha-1/rabbitmq-ha default//rebbitmq-rabbitmq-ha-2/rabbitmq-ha default/mongodb/mongodb-6767d9f67d-kzj49/mongodb infra//etcd0/etcd0 infra//etcd1/etcd1 infra//etcd2/etcd2 infra//rabbitmq-0/rabbitmq infra/mongodb/mongodb-9b67f668f-krcbc/mongodb istio-system/grafana/grafana-556f8998cd-mq55z/grafana istio-system/istio-egressgateway/istio-egressgateway-79546b9866-xt646/istio-proxy istio-system/istio-ingressgateway/istio-ingressgateway-77ffcc64cd-nvskj/istio-proxy istio-system/istiod/istiod-665845d58f-n4vrn/discovery istio-system/jaeger/jaeger-5f65fdbf9b-c5r55/jaeger istio-system/kiali/kiali-787bc487b7-2vshc/kiali istio-system/prometheus/prometheus-9f4947649-5vjw6/prometheus-server istio-system/prometheus/prometheus-9f4947649-5vjw6/prometheus-server-configmap-reload kubernetes-dashboard/dashboard-metrics-scraper/dashboard-metrics-scraper-79c5968bdc-qqw6k/dashboard-metrics-scraper kubernetes-dashboard/kubernetes-dashboard/kubernetes-dashboard-9f9799597-hxt5l/kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard/kubernetes-dashboard-9f9799597-sx2b7/kubernetes-dashboard

Fetching Istio control plane information from cluster.

Running istio analyze on all namespaces and report as below: Analysis Report: Info [IST0102] (Namespace bookstore-microservices) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace bookstore-microservices istio-injection=enabled’ to enable it, or ‘kubectl label namespace bookstore-microservices istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace default istio-injection=enabled’ to enable it, or ‘kubectl label namespace default istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace infra) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace infra istio-injection=enabled’ to enable it, or ‘kubectl label namespace infra istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace istio-system) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace istio-system istio-injection=enabled’ to enable it, or ‘kubectl label namespace istio-system istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0102] (Namespace kubernetes-dashboard) The namespace is not enabled for Istio injection. Run ‘kubectl label namespace kubernetes-dashboard istio-injection=enabled’ to enable it, or ‘kubectl label namespace kubernetes-dashboard istio-injection=disabled’ to explicitly mark it as not needing injection. Info [IST0118] (Service account.bookstore-microservices) Port name (port: 80, targetPort: http-server) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service admin.content-security-mesh) Port name pprof (port: 28800, targetPort: 28800) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service dashboard-metrics-scraper.kubernetes-dashboard) Port name (port: 8000, targetPort: 8000) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service etcd-client.infra) Port name etcd-client-port (port: 2379, targetPort: 2379) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service etcd-headless.default) Port name client (port: 2379, targetPort: client) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service etcd-headless.default) Port name peer (port: 2380, targetPort: peer) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service etcd.default) Port name client (port: 2379, targetPort: client) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service etcd.default) Port name peer (port: 2380, targetPort: peer) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service etcd0.infra) Port name client (port: 2379, targetPort: 2379) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service etcd0.infra) Port name server (port: 2380, targetPort: 2380) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service etcd1.infra) Port name client (port: 2379, targetPort: 2379) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service etcd1.infra) Port name server (port: 2380, targetPort: 2380) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service etcd2.infra) Port name client (port: 2379, targetPort: 2379) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service etcd2.infra) Port name server (port: 2380, targetPort: 2380) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service gateway.bookstore-microservices) Port name (port: 8080, targetPort: http-server) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service grafana.istio-system) Port name service (port: 3000, targetPort: 3000) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service jaeger-collector.istio-system) Port name jaeger-collector-grpc (port: 14250, targetPort: 14250) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service jaeger-collector.istio-system) Port name jaeger-collector-http (port: 14268, targetPort: 14268) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service kubernetes-dashboard.kubernetes-dashboard) Port name (port: 443, targetPort: 8443) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service master.content-security-mesh) Port name pprof (port: 28600, targetPort: 28600) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service master.content-security-mesh) Port name rpc (port: 8600, targetPort: 8600) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service payment.bookstore-microservices) Port name (port: 80, targetPort: http-server) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service rabbitmq-headless.infra) Port name amqp (port: 5672, targetPort: amqp) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service rabbitmq-headless.infra) Port name dist (port: 25672, targetPort: dist) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service rabbitmq-headless.infra) Port name epmd (port: 4369, targetPort: epmd) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service rabbitmq.infra) Port name amqp (port: 5672, targetPort: amqp) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service rabbitmq.infra) Port name dist (port: 25672, targetPort: dist) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service rabbitmq.infra) Port name epmd (port: 4369, targetPort: epmd) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service rebbitmq-rabbitmq-ha-discovery.default) Port name amqp (port: 5672, targetPort: amqp) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service rebbitmq-rabbitmq-ha-discovery.default) Port name epmd (port: 4369, targetPort: epmd) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service rebbitmq-rabbitmq-ha.default) Port name amqp (port: 5672, targetPort: amqp) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service rebbitmq-rabbitmq-ha.default) Port name epmd (port: 4369, targetPort: epmd) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service security.bookstore-microservices) Port name (port: 80, targetPort: http-server) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service security.content-security-mesh) Port name pprof (port: 28700, targetPort: 28700) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service security.content-security-mesh) Port name rpc (port: 8700, targetPort: 8700) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service warehouse.bookstore-microservices) Port name (port: 80, targetPort: http-server) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service worker.content-security-mesh) Port name pprof (port: 28500, targetPort: 28500) doesn’t follow the naming convention of Istio port. Info [IST0118] (Service worker.content-security-mesh) Port name rpc (port: 8500, targetPort: 8500) doesn’t follow the naming convention of Istio port. Creating an archive at /Users/lixian/Work/content-security-server/bug-report.tar.gz. Cleaning up temporary files in /var/folders/pr/pcy1xr895k7_y7pjzy2_m89w0000gn/T/bug-report.