istio: Pilot configures proxy with wrong routes/listeners

Bug description Pilot adds incorrect listeners/routes to envoy proxies, resulting in 503’s. This is the output of proxy-config routes on 2 identical pods created from the same deployment:

This is the proxy that received the correct config:

$ istioctl pc routes pod1.our-namespace
NOTE: This output only contains routes loaded via RDS.
NAME                                                                                                    VIRTUAL HOSTS
8080                                                                                                    4
9091                                                                                                    2
istio-telemetry.istio-system.svc.cluster.local:42422                                                    1
15004                                                                                                   2
15014                                                                                                   2
inbound|8080|http|correct-service.our-namespace.svc.cluster.local     1
inbound|8080|http|correct-service.our-namespace.svc.cluster.local     1
inbound|8080|http|correct-service.our-namespace.svc.cluster.local     1
inbound|8080|http|correct-service.our-namespace.svc.cluster.local     1

Here is the output from an incorrectly configured proxy on the other pod from the same deployment as above:

$ istioctl pc routes pod2.our-namespace
NOTE: This output only contains routes loaded via RDS.
NAME                                                           VIRTUAL HOSTS
8080                                                           4
istio-telemetry.istio-system.svc.cluster.local:42422           1
9091                                                           2
15004                                                          2
15014                                                          2
inbound|8800|http|service-from-another-namespace.wrong-namespace.svc.cluster.local     1
inbound|8800|http|service-from-another-namespace.wrong-namespace.svc.cluster.local     1
inbound|8800|http|service-from-another-namespace.wrong-namespace.svc.cluster.local     1
inbound|8800|http|service-from-another-namespace.wrong-namespace.svc.cluster.local     1

Also notice the port number is not right either. Restarting the second pod will result in a proxy with the same config as the first, which is correct.

Expected behavior Pilot configures proxy with correct routes/listeners for the service.

Steps to reproduce the bug It seems to only happen on a deployment when new pods are created and not every time. Restarting the pods seems to yield correct listeners every time.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

$ istioctl version --remote
client version: 1.4.3
citadel version: 1.4.3
galley version: 1.4.3
galley version: 1.4.3
istio-ig-custom version: 
istio-ig-custom version: 
istio-ig-custom2 version: 
istio-ig-custom2 version: 
istio-ig-custom3 version: 
istio-ig-custom3 version: 
pilot version: 1.4.3
pilot version: 1.4.3
sidecar-injector version: 1.4.3
sidecar-injector version: 1.4.3
telemetry version: 1.4.3
telemetry version: 1.4.3
data plane version: 1.4.0 (1 proxies), 1.4.3 (229 proxies)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:34Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:07:57Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed? Helm 2 with Tiller

Environment where bug was observed (cloud vendor, OS, etc) On premise, centos