istio: Unable to connect to nats from an istio (1.7.1) enabled namespace

Bug description

I am trying to connect to nats from a namespace where istio (and mTLS) is enabled. I am consistently getting i/o timeout error.

Connecting directly to the POD IP works fine.

[ ] Docs [ ] Installation [x] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [x] User Experience [ ] Developer Infrastructure

Expected behavior I should be able to connect to nats service from a namespace where istio is enabled.

Steps to reproduce the bug

1. Create nats.yaml with the following values:

• Feel free to change the namespace and make sure istio is not enabled on the namespace.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nats
  name: nats
  namespace: "default"
spec:
  type: ClusterIP
  ports:
    - port: 4222
      protocol: TCP
      name: clients
  selector:
    app: nats
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nats
  name: nats
  namespace: "default"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nats
  template:
    metadata:
      labels:
        app: nats
    spec:
      containers:
      - name:  nats
        resources:
            limits:
              cpu: 100m
              memory: 128Mi
            requests:
              cpu: 50m
              memory: 64Mi
        image: nats-streaming:0.17.0
        imagePullPolicy: Always
        ports:
        - containerPort: 4222
          protocol: TCP
        command: ["/nats-streaming-server"]
        args:
          - --store
          - memory
          - --cluster_id
          - nats-cluster

1. Deploy nats

kubectl apply -f ./nats.yaml

1. Notice connectivity from non-istio namespace is working fine:

$ kubectl run -i --rm --tty ubuntu --image=ubuntu --restart=Never -n default

# in the ubuntu pod, execute the following:
$ apt-get update
$ apt-get install curl zip -y

$ curl -L https://github.com/nats-io/go-nats-examples/releases/download/0.0.50/go-nats-examples-v0.0.50-linux-amd64.zip -o nats.zip
$ unzip nats.zip
$ cd go-nats-examples-v0.0.50-linux-amd64/

$ ./nats-sub -s nats://nats.default.svc.cluster.local:4222 ">"
Listening on [>]

1. Notice connectivity from istio-enabled namespace is not working:

$ kubectl run -i --rm --tty ubuntu --image=ubuntu --restart=Never -n istio-enabled-ns

# in the ubuntu pod, execute the following:
$ apt-get update
$ apt-get install curl zip -y

$ curl -L https://github.com/nats-io/go-nats-examples/releases/download/0.0.50/go-nats-examples-v0.0.50-linux-amd64.zip -o nats.zip
$ unzip nats.zip
$ cd go-nats-examples-v0.0.50-linux-amd64/

$ ./nats-sub -s nats://nats.default.svc.cluster.local:4222 ">"
read tcp 10.10.32.123:57612->10.10.10.87:4222: i/o timeout

Version

$ istioctl version --remote
client version: 1.7.4
control plane version: 1.7.1
data plane version: 1.7.1 (14 proxies)

$ kubectl version --short
Client Version: v1.19.2
Server Version: v1.17.11

How was Istio installed? Using rancher-istio.