istio: Sidecar envoys gets wrong initial config

Describe the bug

Sometimes it takes a long time for injected sidecar envoy proxies to be healthy, because the initial config from Pilot is not contains the necessary listeners with the pod IP

kind: Deployment
metadata:
  name: echo
  labels:
    k8s-app: echo
  namespace: default
spec:
  replicas: 20
  selector:
    matchLabels:
      k8s-app: echo
  template:
    metadata:
      labels:
        k8s-app: echo
    spec:
      terminationGracePeriodSeconds: 1
      containers:
      - name: echo-service
        image: k8s.gcr.io/echoserver:1.10
        ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: echo
  labels:
    k8s-app: echo
  namespace: default
spec:
  ports:
  - name: http
    port: 8000
  selector:
    k8s-app: echo

NAME                   READY   STATUS    RESTARTS   AGE
echo-5d495b548f-2ws66   1/2     Running   0          1m
echo-5d495b548f-4cw8s   2/2     Running   0          1m
echo-5d495b548f-59hfg   1/2     Running   0          1m
echo-5d495b548f-5cfkt   2/2     Running   0          1m
echo-5d495b548f-6l699   1/2     Running   0          1m
echo-5d495b548f-7hfd2   1/2     Running   0          1m
echo-5d495b548f-7wkcs   2/2     Running   0          1m
echo-5d495b548f-92wlg   1/2     Running   0          1m
echo-5d495b548f-96cnj   2/2     Running   0          1m
echo-5d495b548f-hf529   2/2     Running   0          1m
echo-5d495b548f-j6qhd   1/2     Running   0          1m
echo-5d495b548f-kx59c   2/2     Running   0          1m
echo-5d495b548f-l7n8f   1/2     Running   0          1m
echo-5d495b548f-lkkq6   1/2     Running   0          1m
echo-5d495b548f-qhv9b   2/2     Running   0          1m
echo-5d495b548f-rjvgz   1/2     Running   0          1m
echo-5d495b548f-rkmqm   1/2     Running   0          1m
echo-5d495b548f-scsqg   1/2     Running   0          1m
echo-5d495b548f-vmpwc   1/2     Running   0          1m
echo-5d495b548f-x2qjr   1/2     Running   0          1m

$ k logs -f echo-5d495b548f-j6qhd -c istio-proxy

* failed checking application ports. listeners="0.0.0.0:15090","10.0.9.253:443","10.0.14.181:443","10.0.14.62:14268","10.0.13.140:443","10.0.0.10:53","10.0.14.181:15443","10.0.15.199:16686","10.0.11.84:8443","10.0.15.190:15030","10.0.15.190:15443","10.0.7.215:42422","10.0.15.190:443","10.0.5.139:15011","10.0.3.222:80","10.0.15.190:31400","10.0.15.190:15020","10.0.0.1:443","10.0.15.190:15029","10.0.4.88:443","10.0.5.201:443","10.0.8.138:443","10.0.15.190:15031","10.0.15.190:15032","10.0.14.62:14267","0.0.0.0:15014","0.0.0.0:15010","0.0.0.0:80","0.0.0.0:8060","0.0.0.0:8000","0.0.0.0:3000","0.0.0.0:9901","0.0.0.0:9090","0.0.0.0:8080","0.0.0.0:9411","0.0.0.0:9091","0.0.0.0:20001","0.0.0.0:15004","0.0.0.0:15001"
* envoy missing listener for inbound application port: 8000

Expected behavior

The initial envoy config should contain podIP:8000 and goes healthy right after it gets the inital config from Pilot

Steps to reproduce the bug

Fresh GKE cluster with upstream Istio 1.1.x

Version

Istio version

client version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-17-g2b13318"}
egressgateway version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-17-g2b13318"}
ingressgateway version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-17-g2b13318"}
pilot version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}
K8s version

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-03-01T23:34:27Z", GoVersion:"go1.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.8-gke.5", GitCommit:"a9aff9e146595b2843839905a8fb6d1cf206f97b", GitTreeState:"clean", BuildDate:"2019-03-13T11:18:24Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

Installation

kubectl apply -f install/kubernetes/istio-demo.yaml

Environment

GKE, upstream Istio 1.1.1

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 28 (25 by maintainers)

Most upvoted comments

@hzxuzhonghu I am experiencing the same issues. Some of my pods are taking too long to become ready

Warning  Unhealthy  116s (x150 over 6m54s)  kubelet, ip-<node> Readiness probe failed: HTTP probe failed with statuscode: 503

and istio-proxy logs are full of:

2019-04-09T15:12:29.680267Z    info    Envoy proxy is NOT ready: 3 errors occurred:

* failed checking application ports. listeners="0.0.0.0:15090","110.65.156.1:80","110.64.112.114:10314","110.69.59.244:80","110.70.52.225:10097","110.67.110.117:80","110.68.196.229:80","110.70.182.174:80","110.66.183.135:80","110.69.67.97:80","110.70.234.229:80","110.65.225.174:8083","110.68.107.58:9090","110.65.205.240:80","110.69.107.176:80","110.64.249.187:10235","110.67.67.63:80","110.69.185.44:6379","110.65.225.174:1883","0.0.0.0:9093","110.69.169.28:1883","110.66.89.77:18083","110.71.112.236:80","110.65.254.2:80","110.64.51.31:9100","110.71.141.124:80","110.64.0.10:53","0.0.0.0:9090","110.70.242.18:80","110.64.248.227:6379","110.69.156.184:6379","110.69.106.232:8083","110.69.106.232:18083","110.64.0.1:443","110.71.209.112:80","110.69.47.189:80","110.66.240.167:80","110.71.149.153:6379","110.70.182.200:44134","110.65.190.81:80","110.69.86.113:6379","110.69.246.63:6379","110.67.82.124:80","110.69.250.123:80","110.71.91.49:80","110.68.32.4:80","110.68.180.9:6379","0.0.0.0:10250","110.69.106.232:1883","0.0.0.0:6783","110.68.64.232:80","110.64.203.228:443","110.65.139.162:15011","110.66.93.199:80","110.71.126.193:80","110.65.225.174:18083","110.67.223.125:80","110.69.233.126:6379","110.68.246.218:9093","110.67.19.59:443","110.64.181.192:15672","110.66.89.77:1883","110.66.122.41:80","110.66.74.142:80","110.71.47.112:80","110.64.181.192:5672","110.66.222.166:80","110.69.191.213:80","110.64.7.134:80","110.70.72.27:80","110.68.4.160:80","110.70.191.41:42422","110.68.239.160:80","110.66.89.77:8083","110.65.143.139:80","110.66.54.126:80","0.0.0.0:4001","0.0.0.0:8080","0.0.0.0:15014","0.0.0.0:80","0.0.0.0:10252","0.0.0.0:10054","0.0.0.0:15004","0.0.0.0:20001","0.0.0.0:10055","0.0.0.0:3000","0.0.0.0:15010","0.0.0.0:8060","0.0.0.0:9083","0.0.0.0:10251","0.0.0.0:9091","0.0.0.0:15001"
* envoy missing listener for inbound application port: 9083
* envoy missing listener for inbound application port: 10030

Is this normal?