istio: Istio proxy does not complete TCP connections until data is sent

Describe the bug

When using a TCP server and client, connections don’t complete to the server until data is sent by client with istio-proxy sidecars in place. Istio is deployed as a sidecar in all pods in this scenario.

Expected behavior

Expected TCP server to see connection immediately when client establishes a connection.

Steps to reproduce the bug

kubectl apply the following yaml to a namespace with istio-proxy sidecars enabled. This will create a create natcat server listening on port 4444 and k8s service for it.

apiVersion: v1
kind: Service
metadata:
  name: tcp-server
  labels:
    app: "tcp-server"
spec:
  type: ClusterIP
  ports:
    - port: 4444
      name: tcp-server
  selector:
    app: "tcp-server"
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: tcp-server
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: tcp-server
        version: v1
    spec:
      containers:
      - name: tcp-server
        image: alpine:3.8
        imagePullPolicy: IfNotPresent
        command:
        - "nc"
        args:
        - -v
        - -lk
        - -p
        - "4444"
        ports:
        - containerPort: 4444

Use the following to watch the logs of netcat:

> kubectl logs -f $(kubectl get pods -l "app=tcp-server" -o name) tcp-server
listening on [::]:4444 ...

Next run another alpine linux container in the same namespace that will use netcat as a client to the server running:

> kubectl run tcp-client -it --rm=true --image alpine:3.8 ash
> nc -w5 -v tcp-server 4444
tcp-server (10.109.191.242:4444) open

The client now has an open connection, but if you look at the logs of the server it is unaware of this connection until you send the first message.

Start the client again but this time send the following message and watch the logs of server

> nc -w5 -v tcp-server 4444
hello

The server should now see the connection and log the message that was sent:

connect to [::ffff:127.0.0.1]:4444 from localhost:35376 ([::ffff:127.0.0.1]:35376)
hello

If you attempt the exact same test in a namespace with the istio sidecar disabled the netcat server will see the connection when the client establishes a connection right away.

This means applications (like NATS) that rely on a tcp client to connect to a server, but have the server send the first message, will not work.

Version

istio version: 1.0.5

k8s version: 1.10.3

Installation

git clone https://github.com/istio/istio
git checkout 1.0.5
helm install install/kubernetes/helm/istio --name istio --namespace istio-system

Environment

Attempted with Kubernetes on Docker-for-Mac and Kubernetes Engine on GCP