envoy: Envoy HTTPS ingress always return 404 when SSL config specifies a domain name

Title: The https listener always return 404 if the domain setting is the real domain name of the certificate

Bug Description:

I have a very simple envoy yaml config file. It defines a simple routing rule with a set of cert and key for SSL. The certificate is a self signed and issued to hello.com. In the envoy.yaml file, if I let the domain to be *, the request can be routed propery. But, if I set the domain to be “hello.com”, the request will get a 404. The docker instance has an IP. I add the IP on my testing client with a host name hello.com. When I send request, I also send to the domain name. I also tried the IP. Same error.

SOAP UI request URL: https://hello.com:8443/mock-domain or https://192.168.64.135:8443/mock-domain

SOAP UI http log

Thu Dec 13 14:45:50 EST 2018:DEBUG:<< "HTTP/1.1 404 Not Found[\r][\n]"
Thu Dec 13 14:45:50 EST 2018:DEBUG:<< "date: Thu, 13 Dec 2018 19:45:50 GMT[\r][\n]"
Thu Dec 13 14:45:50 EST 2018:DEBUG:<< "server: envoy[\r][\n]"
Thu Dec 13 14:45:50 EST 2018:DEBUG:<< "connection: close[\r][\n]"
Thu Dec 13 14:45:50 EST 2018:DEBUG:<< "content-length: 0[\r][\n]"
Thu Dec 13 14:45:50 EST 2018:DEBUG:<< "[\r][\n]"

Config envoy.yaml:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 8443 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
#              domains: ["*"]
              domains: ["hello.com"]
              routes:
              - match:
                  prefix: "/mock-domain" # a test for mock-domain
                route:
                  cluster: mock-domain
          http_filters:
          - name: envoy.router
      tls_context:
        common_tls_context:
            tls_certificates:
            - certificate_chain: { filename: "/etc/crt" }
              private_key: { filename: "/etc/key" }
  clusters:
  - name: mock-domain
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    hosts:
    - socket_address:
        address: mock-domain
        port_value: 10080
admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

Startup Logs:

[2018-12-13 19:45:26.928][000005][info][main] [source/server/server.cc:207] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312)
[2018-12-13 19:45:26.928][000005][info][main] [source/server/server.cc:209] statically linked extensions:
[2018-12-13 19:45:26.928][000005][info][main] [source/server/server.cc:211]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2018-12-13 19:45:26.928][000005][info][main] [source/server/server.cc:214]   filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2018-12-13 19:45:26.928][000005][info][main] [source/server/server.cc:217]   filters.listener: envoy.listener.original_dst,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2018-12-13 19:45:26.928][000005][info][main] [source/server/server.cc:220]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2018-12-13 19:45:26.928][000005][info][main] [source/server/server.cc:222]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2018-12-13 19:45:26.928][000005][info][main] [source/server/server.cc:224]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.zipkin
[2018-12-13 19:45:26.928][000005][info][main] [source/server/server.cc:227]   transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2018-12-13 19:45:26.928][000005][info][main] [source/server/server.cc:230]   transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.capture,raw_buffer,tls
[2018-12-13 19:45:26.933][000005][info][main] [source/server/server.cc:272] admin address: 0.0.0.0:8001
[2018-12-13 19:45:26.934][000005][debug][main] [source/server/overload_manager_impl.cc:171] No overload action configured for envoy.overload_actions.stop_accepting_connections.
[2018-12-13 19:45:26.934][000005][debug][main] [source/server/overload_manager_impl.cc:171] No overload action configured for envoy.overload_actions.stop_accepting_connections.
[2018-12-13 19:45:26.934][000005][info][config] [source/server/configuration_impl.cc:51] loading 0 static secret(s)
[2018-12-13 19:45:26.934][000005][info][config] [source/server/configuration_impl.cc:57] loading 1 cluster(s)
[2018-12-13 19:45:26.936][000005][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:818] adding TLS initial cluster mock-domain
[2018-12-13 19:45:26.936][000005][debug][upstream] [source/common/upstream/upstream_impl.cc:1183] starting async DNS resolution for mock-domain
[2018-12-13 19:45:26.936][000005][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 3436 milliseconds
[2018-12-13 19:45:26.936][000005][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:63] cm init: adding: cluster=mock-domain primary=1 secondary=0
[2018-12-13 19:45:26.936][000005][info][config] [source/server/configuration_impl.cc:62] loading 1 listener(s)
[2018-12-13 19:45:26.936][000005][debug][config] [source/server/configuration_impl.cc:64] listener #0:
[2018-12-13 19:45:26.936][000005][debug][config] [source/server/listener_manager_impl.cc:640] begin add/update listener: name=listener_0 hash=6635500297793231887
[2018-12-13 19:45:26.937][000005][debug][config] [source/server/listener_manager_impl.cc:40]   filter #0:
[2018-12-13 19:45:26.937][000005][debug][config] [source/server/listener_manager_impl.cc:41]     name: envoy.http_connection_manager
[2018-12-13 19:45:26.937][000005][debug][config] [source/server/listener_manager_impl.cc:44]   config: {"http_filters":[{"name":"envoy.router"}],"stat_prefix":"ingress_http","codec_type":"AUTO","route_config":{"virtual_hosts":[{"domains":["hello.com"],"routes":[{"match":{"prefix":"/mock-domain"},"route":{"cluster":"mock-domain"}}],"name":"local_service"}],"name":"local_route"}}
[2018-12-13 19:45:26.938][000005][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:312]     http filter #0
[2018-12-13 19:45:26.938][000005][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:313]       name: envoy.router
[2018-12-13 19:45:26.938][000005][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:317]     config: {}
[2018-12-13 19:45:26.942][000005][debug][config] [source/server/listener_manager_impl.cc:527] add active listener: name=listener_0, hash=6635500297793231887, address=0.0.0.0:8443
[2018-12-13 19:45:26.942][000005][info][config] [source/server/configuration_impl.cc:95] loading tracing configuration
[2018-12-13 19:45:26.942][000005][info][config] [source/server/configuration_impl.cc:115] loading stats sink configuration
[2018-12-13 19:45:26.942][000005][info][main] [source/server/server.cc:458] starting main dispatch loop
[2018-12-13 19:45:26.942][000009][debug][grpc] [source/common/grpc/google_async_client_impl.cc:41] completionThread running
[2018-12-13 19:45:26.943][000005][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 4686 milliseconds
[2018-12-13 19:45:26.953][000005][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 3436 milliseconds
[2018-12-13 19:45:26.961][000005][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 4374 milliseconds
[2018-12-13 19:45:26.962][000005][debug][upstream] [source/common/upstream/upstream_impl.cc:1190] async DNS resolution complete for mock-domain
[2018-12-13 19:45:26.962][000005][debug][upstream] [source/common/upstream/upstream_impl.cc:1212] DNS hosts have changed for mock-domain
[2018-12-13 19:45:26.962][000005][debug][upstream] [source/common/upstream/upstream_impl.cc:587] initializing secondary cluster mock-domain completed
[2018-12-13 19:45:26.962][000005][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:953] membership update for TLS cluster mock-domain
[2018-12-13 19:45:26.962][000005][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:91] cm init: init complete: cluster=mock-domain primary=0 secondary=0
[2018-12-13 19:45:26.962][000005][info][upstream] [source/common/upstream/cluster_manager_impl.cc:136] cm init: all clusters initialized
[2018-12-13 19:45:26.962][000005][info][main] [source/server/server.cc:430] all clusters initialized. initializing init manager
[2018-12-13 19:45:26.962][000005][info][config] [source/server/listener_manager_impl.cc:910] all dependencies initialized. starting workers
[2018-12-13 19:45:26.962][000011][debug][main] [source/server/worker_impl.cc:98] worker entering dispatch loop
[2018-12-13 19:45:26.962][000011][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:818] adding TLS initial cluster mock-domain
[2018-12-13 19:45:26.962][000011][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:953] membership update for TLS cluster mock-domain
[2018-12-13 19:45:26.962][000012][debug][main] [source/server/worker_impl.cc:98] worker entering dispatch loop
[2018-12-13 19:45:26.962][000012][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:818] adding TLS initial cluster mock-domain
[2018-12-13 19:45:26.962][000014][debug][grpc] [source/common/grpc/google_async_client_impl.cc:41] completionThread running
[2018-12-13 19:45:26.962][000012][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:953] membership update for TLS cluster mock-domain
[2018-12-13 19:45:26.962][000013][debug][grpc] [source/common/grpc/google_async_client_impl.cc:41] completionThread running
[2018-12-13 19:45:31.943][000005][debug][main] [source/server/server.cc:144] flushing stats
[2018-12-13 19:45:31.961][000005][debug][upstream] [source/common/upstream/upstream_impl.cc:1183] starting async DNS resolution for mock-domain
[2018-12-13 19:45:31.962][000005][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
[2018-12-13 19:45:31.965][000005][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 3124 milliseconds
[2018-12-13 19:45:31.979][000005][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 2812 milliseconds
[2018-12-13 19:45:31.992][000005][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 3750 milliseconds
[2018-12-13 19:45:31.993][000005][debug][upstream] [source/common/upstream/upstream_impl.cc:1190] async DNS resolution complete for mock-domain

Trace Logs for the transaction:

[2018-12-13 19:45:35.229][000012][debug][main] [source/server/connection_handler_impl.cc:236] [C0] new connection
[2018-12-13 19:45:35.230][000012][debug][connection] [source/common/ssl/ssl_socket.cc:135] [C0] handshake error: 2
[2018-12-13 19:45:35.248][000012][debug][connection] [source/common/ssl/ssl_socket.cc:135] [C0] handshake error: 2
[2018-12-13 19:45:35.248][000012][debug][connection] [source/common/ssl/ssl_socket.cc:135] [C0] handshake error: 2
[2018-12-13 19:45:35.265][000012][debug][connection] [source/common/ssl/ssl_socket.cc:135] [C0] handshake error: 2
[2018-12-13 19:45:35.265][000012][debug][connection] [source/common/ssl/ssl_socket.cc:135] [C0] handshake error: 2
[2018-12-13 19:45:35.270][000012][debug][connection] [source/common/ssl/ssl_socket.cc:124] [C0] handshake complete
[2018-12-13 19:45:35.274][000012][debug][http] [source/common/http/conn_manager_impl.cc:200] [C0] new stream
[2018-12-13 19:45:35.277][000012][debug][http] [source/common/http/conn_manager_impl.cc:529] [C0][S13148359650461284382] request headers complete (end_stream=false):
':authority', 'hello.com:8443'
':path', '/mock-domain'
':method', 'POST'
'accept-encoding', 'gzip,deflate'
'content-type', 'text/xml;charset=UTF-8'
'content-length', '799'
'connection', 'Keep-Alive'
'user-agent', 'Apache-HttpClient/4.1.1 (java 1.5)'

[2018-12-13 19:45:35.277][000012][debug][router] [source/common/router/router.cc:221] [C0][S13148359650461284382] no cluster match for URL '/mock-domain'
[2018-12-13 19:45:35.277][000012][debug][http] [source/common/http/conn_manager_impl.cc:1180] [C0][S13148359650461284382] encoding headers via codec (end_stream=true):
':status', '404'
'date', 'Thu, 13 Dec 2018 19:45:34 GMT'
'server', 'envoy'
'connection', 'close'

[2018-12-13 19:45:35.277][000012][debug][connection] [source/common/network/connection_impl.cc:101] [C0] closing data_to_write=116 type=2
[2018-12-13 19:45:35.277][000012][debug][connection] [source/common/network/connection_impl.cc:153] [C0] setting delayed close timer with timeout 1000 ms
[2018-12-13 19:45:35.277][000012][debug][connection] [source/common/network/connection_impl.cc:101] [C0] closing data_to_write=116 type=2
[2018-12-13 19:45:35.293][000012][debug][connection] [source/common/network/connection_impl.cc:460] [C0] remote early close
[2018-12-13 19:45:35.293][000012][debug][connection] [source/common/network/connection_impl.cc:183] [C0] closing socket: 0
[2018-12-13 19:45:35.293][000012][debug][connection] [source/common/ssl/ssl_socket.cc:233] [C0] SSL shutdown: rc=0
[2018-12-13 19:45:35.293][000012][debug][main] [source/server/connection_handler_impl.cc:68] [C0] adding to cleanup list