envoy: HTTP2 connection pool doesn't work with HTTP CONNECT

Description: I’m using Envoy v1.25.0 as dynamic HTTP forward proxy with connect_matcher. For some reason in this configuration it refuses to utilize HTTP2 connection pool feature, resulting in every single request opening a brand new TCP/IP connection.

I specifically mentioned that it doesn’t work with HTTP CONNECT because I noticed that if I do the tests below with HTTP instead of HTTPS (which doesn’t utilize HTTP CONNECT), then connection pool works perfectly fine.

This bug item is basically a copy of the question 24702 that I raised a few months back (apologies for being too noisy about this, but I would really love to get to the bottom of the issue!). I opened the new item because I’m almost positive that the issue that it’s either a bug, or some undocumented limitation, or perhaps even the result of a misconfiguration (one that I can’t seem to spot) of sorts.

P.S. I also tried to configure it as SNI dynamic forward proxy, but that didn’t help.

Repro steps:

1. Run Envoy with the configuration below;
2. Do an HTTP2 HTTPS requests through Envoy, example with cURL:

curl -Lvx localhost:10000 https://www.microsoft.com

1. Run a command to list outgoing connections, example with ss:

ss -pi | grep -i envoy

Result: the TCP/IP connection created by the request will be closed almost immediately once the request has been completed. Expectations: the TCP/IP connection created by the request will remain active to be re-used.

1. Do an HTTP2 HTTP request through Envoy, example with cURL:

curl -Lvx localhost:10000 www.microsoft.com

1. Run a command to list outgoing connections, example with ss:

ss -pi | grep -i envoy

Result: the TCP/IP connection created by the request will remain active to be re-used.

Config:

static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address:
          protocol: TCP
          address: 127.0.0.1
          port_value: 10000
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                http_protocol_options:
                  accept_http_10: true
                access_log:
                - name: envoy.file_access_log
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                    path: "/var/log/envoy/envoy_access.log"
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: ["*"]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: dynamic_forward_proxy_cluster
                            timeout: 0s
                        - match:
                            connect_matcher:
                              {}
                          route:
                            cluster: dynamic_forward_proxy_cluster
                            timeout: 0s
                            upgrade_configs:
                              - upgrade_type: CONNECT
                                connect_config:
                                  {}
                http_filters:
                  - name: envoy.filters.http.dynamic_forward_proxy
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
                      dns_cache_config:
                        name: dynamic_forward_proxy_cache_config
                        dns_lookup_family: V4_ONLY
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
    - name: dynamic_forward_proxy_cluster
      connect_timeout: 60s
      lb_policy: CLUSTER_PROVIDED
      cluster_type:
        name: envoy.clusters.dynamic_forward_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
          dns_cache_config:
            name: dynamic_forward_proxy_cache_config
            dns_lookup_family: V4_ONLY
admin:
  access_log_path: /var/log/envoy/admin_access.log
  address:
    socket_address:
      protocol: TCP
      address: 0.0.0.0
      port_value: 9901