istio: Istio Ingressgateway Cannot Proxy HTTPS from port 443

(NOTE: This is used to report product bugs: To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities To ask questions about how to use Istio, please visit https://discuss.istio.io)

Bug description I am using istio with sds to handle HTTPS traffic, the cert is provided by cert-manager. My backend service cannot use port 443 to proxy traffic, while it works when I changed the listening port from 443 to 8443; however, it will result in the duplicate listener error from my ingressgateway, so that new ingressgateway pods would get stuck when starting. Can see the error from config dump as

"details": "duplicate listener 0.0.0.0_8443 found"

or similar in the istio-ingressgateway logs

warning	envoy config	gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8443: duplicate listener 0.0.0.0_8443 found

When my gateway was configured as below, it cannot work.

kind: Gateway
....
  servers:
    - hosts:
        - myapp.mydomain
      port:
        name: http
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: true
    - hosts:
        - myapp.mydomain
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: my-ssl-cert
        mode: SIMPLE
  selector:
    istio: ingressgateway

while simply change the listening port of my https to 8443, I can access my app without problem.

kind: Gateway
....
  servers:
    - hosts:
        - myapp.mydomain
      port:
        name: http
        number: 80
        protocol: HTTP
      tls:
        httpsRedirect: true
    - hosts:
        - myapp.mydomain
      port:
        name: https
        number: 8443
        protocol: HTTPS
      tls:
        credentialName: my-ssl-cert
        mode: SIMPLE
  selector:
    istio: ingressgateway

Once the port changed to 8443, I would see the logs from istio-ingressgateway would say it discovered my secret by sds.

2020-12-01T07:47:54.091277Z	info	sds	resource:my-ssl-cert new connection
2020-12-01T07:47:54.091372Z	info	sds	Skipping waiting for gateway secret
2020-12-01T07:47:54.091389Z	info	cache	GenerateSecret my-ssl-cert
2020-12-01T07:47:54.091431Z	info	sds	resource:my-ssl-cert pushed key/cert pair to proxy

however if I create a new istio-ingressgateway pod, it will get stuck with

2020-12-01T08:31:27.072515Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected

Anyone has idea why or what should I do to fix it?

[ ] Docs [x ] Installation [x] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [x ] Security [ ] Test and Release [x ] User Experience [ ] Developer Infrastructure [ ] Upgrade

Expected behavior I would expect it could work just using 443. Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm) 1.7.4 How was Istio installed? istioctl with profile like below

...
    - name: istio-ingressgateway
      label:
        istio: ingressgateway
      enabled: true
      k8s:
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: nlb
          service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
        env:
          - name: ISTIO_META_ROUTER_MODE
            value: "sni-dnat"
        service:
          ports:
            - port: 15021
              targetPort: 15021
              name: status-port
            - port: 80
              targetPort: 8080
              name: http2
            - port: 443
              targetPort: 8443
              name: https
            - port: 15443
              targetPort: 15443
              name: tls
...

Environment where the bug was observed (cloud vendor, OS, etc) aws Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue. Refer cluster state archive for more details.