istio: Sidecar envoys gets wrong initial config
Describe the bug
Sometimes it takes a long time for injected sidecar envoy proxies to be healthy, because the initial config from Pilot is not contains the necessary listeners with the pod IP
kind: Deployment
metadata:
name: echo
labels:
k8s-app: echo
namespace: default
spec:
replicas: 20
selector:
matchLabels:
k8s-app: echo
template:
metadata:
labels:
k8s-app: echo
spec:
terminationGracePeriodSeconds: 1
containers:
- name: echo-service
image: k8s.gcr.io/echoserver:1.10
ports:
- containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
name: echo
labels:
k8s-app: echo
namespace: default
spec:
ports:
- name: http
port: 8000
selector:
k8s-app: echo
NAME READY STATUS RESTARTS AGE
echo-5d495b548f-2ws66 1/2 Running 0 1m
echo-5d495b548f-4cw8s 2/2 Running 0 1m
echo-5d495b548f-59hfg 1/2 Running 0 1m
echo-5d495b548f-5cfkt 2/2 Running 0 1m
echo-5d495b548f-6l699 1/2 Running 0 1m
echo-5d495b548f-7hfd2 1/2 Running 0 1m
echo-5d495b548f-7wkcs 2/2 Running 0 1m
echo-5d495b548f-92wlg 1/2 Running 0 1m
echo-5d495b548f-96cnj 2/2 Running 0 1m
echo-5d495b548f-hf529 2/2 Running 0 1m
echo-5d495b548f-j6qhd 1/2 Running 0 1m
echo-5d495b548f-kx59c 2/2 Running 0 1m
echo-5d495b548f-l7n8f 1/2 Running 0 1m
echo-5d495b548f-lkkq6 1/2 Running 0 1m
echo-5d495b548f-qhv9b 2/2 Running 0 1m
echo-5d495b548f-rjvgz 1/2 Running 0 1m
echo-5d495b548f-rkmqm 1/2 Running 0 1m
echo-5d495b548f-scsqg 1/2 Running 0 1m
echo-5d495b548f-vmpwc 1/2 Running 0 1m
echo-5d495b548f-x2qjr 1/2 Running 0 1m
$ k logs -f echo-5d495b548f-j6qhd -c istio-proxy
* failed checking application ports. listeners="0.0.0.0:15090","10.0.9.253:443","10.0.14.181:443","10.0.14.62:14268","10.0.13.140:443","10.0.0.10:53","10.0.14.181:15443","10.0.15.199:16686","10.0.11.84:8443","10.0.15.190:15030","10.0.15.190:15443","10.0.7.215:42422","10.0.15.190:443","10.0.5.139:15011","10.0.3.222:80","10.0.15.190:31400","10.0.15.190:15020","10.0.0.1:443","10.0.15.190:15029","10.0.4.88:443","10.0.5.201:443","10.0.8.138:443","10.0.15.190:15031","10.0.15.190:15032","10.0.14.62:14267","0.0.0.0:15014","0.0.0.0:15010","0.0.0.0:80","0.0.0.0:8060","0.0.0.0:8000","0.0.0.0:3000","0.0.0.0:9901","0.0.0.0:9090","0.0.0.0:8080","0.0.0.0:9411","0.0.0.0:9091","0.0.0.0:20001","0.0.0.0:15004","0.0.0.0:15001"
* envoy missing listener for inbound application port: 8000
Expected behavior
The initial envoy config should contain podIP:8000
and goes healthy right after it gets the inital config from Pilot
Steps to reproduce the bug
Fresh GKE cluster with upstream Istio 1.1.x
Version
Istio version
client version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-17-g2b13318"}
egressgateway version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-17-g2b13318"}
ingressgateway version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-17-g2b13318"}
pilot version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}
K8s version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-03-01T23:34:27Z", GoVersion:"go1.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.8-gke.5", GitCommit:"a9aff9e146595b2843839905a8fb6d1cf206f97b", GitTreeState:"clean", BuildDate:"2019-03-13T11:18:24Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}
Installation
kubectl apply -f install/kubernetes/istio-demo.yaml
Environment
GKE, upstream Istio 1.1.1
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 28 (25 by maintainers)
@hzxuzhonghu I am experiencing the same issues. Some of my pods are taking too long to become ready
and
istio-proxy
logs are full of:Is this normal?