istio: zookeeper stops working after injecting istio-proxy

works pretty well after i remove the annotation! @banix @snible thank you much for the suggestion of using quorumListenOnAllIPs.

Here is what I did.

1. I had a generated zookeeper yaml based on helm template cmd. Add the following to zookeeper cm:

echo “quorumListenOnAllIPs=true” >> $ZK_CONFIG_FILE

1. remove all istio related annotations for exclude ports in the zookeeper yaml

2. redeploy zookeeper yaml file. make sure all pods are deployed new and check init container to ensure the ports 2888/3888 aren’t there.

3. all zookeeper pods should be up running. exec into any of the zookerpods.

$ k exec -it zookeeper-0 bash -c istio-proxy
istio-proxy@zookeeper-0:/$ nc -v zookeeper.default.svc.cluster.local 2181
zookeeper.default.svc.cluster.local [172.21.225.83] 2181 (?) open
status
Zookeeper version: 3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653, built on 05/03/2019 12:07 GMT
Clients:
 /127.0.0.1:37924[0](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 5
Sent: 4
Connections: 1
Outstanding: 0
Zxid: 0x700000000
Mode: follower
Node count: 5

istio-proxy@zookeeper-0:/$ netstat -pan | grep 3888
tcp        0      0 127.0.0.1:45104         127.0.0.1:3888          ESTABLISHED 19/envoy            
tcp        0      0 127.0.0.1:45102         127.0.0.1:3888          ESTABLISHED 19/envoy            
tcp6       0      0 :::3888                 :::*                    LISTEN      -                   
tcp6       0      0 127.0.0.1:3888          127.0.0.1:45104         ESTABLISHED -                   
tcp6       0      0 127.0.0.1:3888          127.0.0.1:45102         ESTABLISHED -                   
istio-proxy@zookeeper-0:/$ netstat -pan | grep 2888
tcp        0      0 172.30.244.96:34532     172.30.196.190:2888     ESTABLISHED 19/envoy            
tcp6       0      0 172.30.244.96:34530     172.30.196.190:2888     ESTABLISHED -                   
istio-proxy@zookeeper-0:/$ netstat -pan | grep 2181
tcp6       0      0 :::2181                 :::*                    LISTEN      -                   
tcp6       0      0 127.0.0.1:39262         127.0.0.1:2181          TIME_WAIT   -                   
tcp6       0      0 127.0.0.1:39044         127.0.0.1:2181          TIME_WAIT   - 

Found a workaround for this… zookeeper has 3 ports: 2181/TCP,3888/TCP,2888/TCP

2181 is for client connnections, and 3888/2888 are both used internally for leader election and followers. I went ahead and excluded 3888/2888 for inbound ports, e.g.

spec:
  serviceName: zookeeper-headless
  replicas: 3
  selector:
    matchLabels:
      app: zookeeper
      release: zookeeper
      component: server
  updateStrategy:
    type: RollingUpdate
    
  template:
    metadata:
      annotations:
        traffic.sidecar.istio.io/excludeInboundPorts: "2888,3888"

and redeployed the statefulset. After that, all my zookeeper pods are coming up fine and the quorum are established.

yes, you should not set quorumListenOnAllIPs in istio 1.10. quorumListenOnAllIPs is an experimental flag from zookeeper and not recommended for production anyway.

This looks like the issues we have with apps that do not listen on local host. This can be changed with updating one or more configuration parameters for a given app. Looking at zookeeper docs (https://zookeeper.apache.org/doc/r3.3.5/zookeeperAdmin.html#sc_configuration) I see:

clientPortAddress
New in 3.3.0: the address (ipv4, ipv6 or hostname) to listen for client connections; that is, the address that clients attempt to connect to. This is optional, by default we bind in such a way that any connection to the clientPort for any address/interface/nic on the server will be accepted.

looking into it.

The root cause is that zookeeper listens on pod ip only

# kubectl exec -ti  zk-1 -c istio-proxy sh
$ netstat -pan |grep 3888
tcp6       0      0 10.244.0.95:3888        :::*                    LISTEN      - 

Ref: https://istio.io/faq/applications/#cassandra

This is really a bad UX, @rshriram @howardjohn @lambdai any idea how can we solve this?