istio: Unable to configure TLS origination with postgres

Bug description I’m trying to use Aiven Postgres which provides managed postgres that uses TLS only. I want Istio to do the TLS origination using the side car proxy instead of using egress gateway. Note that Aiven is outside of the mesh.

[X] Docs [ ] Installation [X] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [X] User Experience [ ] Developer Infrastructure [ ] Upgrade

Expected behavior Ability to connect to postgres using plain TCP from the application pod and have the sidecar do the TLS to postgres itself.

Steps to reproduce the bug DestinationRule:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: external-aiven-postgres
spec:
  host: x.aivencloud.com
  trafficPolicy:
    tls:
      mode: SIMPLE
      caCertificates: /etc/certs/aiven-ca.crt

ServiceEntry:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-aiven-postgres
spec:
  hosts:
  - x.aivencloud.com
  addresses:
  - x.x.x.x/32
  location: MESH_EXTERNAL
  resolution: NONE
  ports:
  - number: 16221
    name: tcp-postgres
    protocol: TCP

Pgadmin deployment for testing:

apiVersion: apps/v1                                                                                                                    [1/59]
kind: Deployment
metadata:
  name: pgadmin4-deployment
  namespace: postgres
  labels:
    app: pgadmin4
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pgadmin4
  template:
    metadata:
      labels:
        app: pgadmin4
      annotations:
        sidecar.istio.io/userVolume: '[{"name": "aiven-ca", "configMap": {"name": "aiven-ca"}}]'
        sidecar.istio.io/userVolumeMount: '[{"name": "aiven-ca", "mountPath": "/etc/certs/aiven-ca.crt", "subPath": "aiven-ca.crt", "readonly
":true}]'
        sidecar.istio.io/logLevel: debug
    spec:
      containers:
          # This is the official pgAdmin 4 container
        - image: dpage/pgadmin4
          name: pgadmin4
          ports:
            - containerPort: 80
              name: pgadmin4
          env:
          - name: PGADMIN_DEFAULT_PASSWORD
            value: bladieblabla
          - name: PGADMIN_DEFAULT_EMAIL
            value: a@a.com

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)

istioctl version --remote
client version: 1.6.7
control plane version: 1.7.3-asm.6
data plane version: 1.7.3-asm.6 (17 proxies)

kubectl version --short
Client Version: v1.18.3
Server Version: v1.17.13-gke.2001

Logs on istio-proxy:

2020-12-23T17:25:36.986952Z     debug   envoy filter    original_dst: New connection accepted                                      [95/66945]
2020-12-23T17:25:36.987208Z     debug   envoy filter    [C22070] new tcp proxy session
2020-12-23T17:25:36.987263Z     debug   envoy filter    [C22070] Creating connection to cluster outbound|16221||x.aiven
cloud.com
2020-12-23T17:25:36.987304Z     debug   envoy upstream  transport socket match, socket default selected for host with address x.x.x.x:1
6221
2020-12-23T17:25:36.987316Z     debug   envoy upstream  Created host x.x.x.x:1
6221.
2020-12-23T17:25:36.987355Z     debug   envoy pool      creating a new connection
2020-12-23T17:25:36.987401Z     debug   envoy upstream  addHost() adding x.x.x.x:1
6221
2020-12-23T17:25:36.987413Z     debug   envoy pool      [C22071] connecting
2020-12-23T17:25:36.987496Z     debug   envoy connection        [C22071] connecting to x.x.x.x:1
6221
2020-12-23T17:25:36.987514Z     debug   envoy upstream  membership update for TLS cluster outbound|16221||x.aivencloud.
com added 1 removed 0
2020-12-23T17:25:36.987525Z     debug   envoy upstream  re-creating local LB for TLS cluster outbound|16221||x.aivenclo
ud.com
2020-12-23T17:25:36.987550Z     debug   envoy upstream  membership update for TLS cluster outbound|16221||x.aivencloud.
com added 1 removed 0
2020-12-23T17:25:36.987568Z     debug   envoy upstream  re-creating local LB for TLS cluster outbound|16221||x.aivenclo
ud.com
2020-12-23T17:25:36.987633Z     debug   envoy connection        [C22071] connection in progress
2020-12-23T17:25:36.987666Z     debug   envoy pool      queueing request due to no available connections
2020-12-23T17:25:36.987673Z     debug   envoy conn_handler      [C22070] new connection
2020-12-23T17:25:36.987702Z     debug   envoy upstream  membership update for TLS cluster outbound|16221||x.aivencloud.
com added 1 removed 0
2020-12-23T17:25:36.987849Z     debug   envoy upstream  re-creating local LB for TLS cluster outbound|16221||x.aivenclo
ud.com
2020-12-23T17:25:36.989903Z     debug   envoy connection        [C22071] connected
2020-12-23T17:25:36.990095Z     debug   envoy connection        [C22071] handshake expecting read
2020-12-23T17:25:36.993936Z     debug   envoy connection        [C22071] handshake error: 5
2020-12-23T17:25:36.993977Z     debug   envoy connection        [C22071]
2020-12-23T17:25:36.993983Z     debug   envoy connection        [C22071] closing socket: 0
2020-12-23T17:25:36.994055Z     debug   envoy pool      [C22071] client disconnected
2020-12-23T17:25:36.994083Z     debug   envoy filter    [C22070] Creating connection to cluster outbound|16221||x.aiven
cloud.com
2020-12-23T17:25:36.994095Z     debug   envoy connection        [C22070] closing data_to_write=0 type=1
2020-12-23T17:25:36.994104Z     debug   envoy connection        [C22070] closing socket: 1

How was Istio installed? ASM install using install_asm script

Environment where the bug was observed (cloud vendor, OS, etc) GKE on GCP

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue. Refer cluster state archive for more details.