envoy: gRPC 1.20.1 upgrade causes gRPC connections to fail

Title: gRPC 1.20.1 upgrade causes gRPC connections to fail

Description: I use Envoy’s gRPC-Web filter to integrate against a few gRPC applications built in Python. These applications use a secured connection. Since they upgraded to gRPC v1.20.1 (from v1.19.x), Envoy has been unable to get proper responses. The version upgrade is the only change and other applications have no problem communicating with the updated services. The error I am getting is gRPC status 14 with this message:

upstream connect error or disconnect/reset before headers. reset reason: connection termination

I tried running this against the current stable version as well as a (yesterday’s) dev build and got the same results.

I have also been unable to reproduce this issue. I created a small gRPC application using Python and gRPC 1.20.1 and I could use gRPC to communicate with it. My little test service was unsecured but otherwise pretty similar, so I suspect perhaps a TLS issue.

The only other relevant thing I can think of is that all of these services are behind (TCP) load balancers.

Config: My config is more or less the same as in the example. I fixed the indentation of the bottom part and (of course) updated addresses to match my services.

Logs: These are the logs of a failing request (minus the request itself):

[2019-05-09 19:20:34.438][19][debug][client] [source/common/http/codec_client.cc:26] [C3] connecting
[2019-05-09 19:20:34.438][19][debug][connection] [source/common/network/connection_impl.cc:695] [C3] connecting to <THE IP ADDRESS>:443
[2019-05-09 19:20:34.438][19][debug][connection] [source/common/network/connection_impl.cc:704] [C3] connection in progress
[2019-05-09 19:20:34.438][19][debug][http2] [source/common/http/http2/codec_impl.cc:725] [C3] setting stream-level initial window size to 268435456
[2019-05-09 19:20:34.438][19][debug][http2] [source/common/http/http2/codec_impl.cc:747] [C3] updating connection-level initial window size to 268435456
[2019-05-09 19:20:34.438][19][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
[2019-05-09 19:20:34.438][19][debug][http] [source/common/http/conn_manager_impl.cc:1062] [C0][S2159400524055061755] request end stream
[2019-05-09 19:20:34.454][19][debug][connection] [source/common/network/connection_impl.cc:543] [C3] connected
[2019-05-09 19:20:34.454][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C3] handshake error: 2
[2019-05-09 19:20:34.470][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C3] handshake error: 2
[2019-05-09 19:20:34.470][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C3] handshake error: 2
[2019-05-09 19:20:34.470][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C3] handshake error: 2
[2019-05-09 19:20:34.471][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C3] handshake error: 2
[2019-05-09 19:20:34.471][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C3] handshake error: 2
[2019-05-09 19:20:34.471][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C3] handshake error: 2
[2019-05-09 19:20:34.472][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C3] handshake error: 2
[2019-05-09 19:20:34.472][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C3] handshake error: 2
[2019-05-09 19:20:34.506][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:131] [C3] handshake complete
[2019-05-09 19:20:34.506][19][debug][client] [source/common/http/codec_client.cc:64] [C3] connected
[2019-05-09 19:20:34.506][19][debug][pool] [source/common/http/http2/conn_pool.cc:96] [C3] creating stream
[2019-05-09 19:20:34.506][19][debug][router] [source/common/router/router.cc:1211] [C0][S2159400524055061755] pool ready
[2019-05-09 19:20:34.507][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:175] [C3] 
[2019-05-09 19:20:34.507][19][debug][connection] [source/common/network/connection_impl.cc:511] [C3] remote close
[2019-05-09 19:20:34.507][19][debug][connection] [source/common/network/connection_impl.cc:192] [C3] closing socket: 0
[2019-05-09 19:20:34.507][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:244] [C3] SSL shutdown: rc=0
[2019-05-09 19:20:34.507][19][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:175] [C3] 
[2019-05-09 19:20:34.507][19][debug][client] [source/common/http/codec_client.cc:82] [C3] disconnect. resetting 1 pending requests
[2019-05-09 19:20:34.507][19][debug][client] [source/common/http/codec_client.cc:105] [C3] request reset
[2019-05-09 19:20:34.507][19][debug][pool] [source/common/http/http2/conn_pool.cc:237] [C3] destroying stream: 0 remaining
[2019-05-09 19:20:34.507][19][debug][router] [source/common/router/router.cc:672] [C0][S2159400524055061755] upstream reset: reset reason connection termination
[2019-05-09 19:20:34.507][19][debug][http] [source/common/http/conn_manager_impl.cc:1137] [C0][S2159400524055061755] Sending local reply with details upstream_reset_before_response_started{connection termination}
[2019-05-09 19:20:34.507][19][debug][http] [source/common/http/conn_manager_impl.cc:1329] [C0][S2159400524055061755] encoding headers via codec (end_stream=true):
':status', '200'
'content-type', 'application/grpc-web-text+proto'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination'
'access-control-allow-origin', 'http://localhost:8081'
'access-control-expose-headers', 'grpc-status,grpc-message'
'date', 'Thu, 09 May 2019 19:20:34 GMT'
'server', 'envoy'

[2019-05-09 19:20:34.507][19][debug][pool] [source/common/http/http2/conn_pool.cc:150] [C3] client disconnected
[2019-05-09 19:20:34.507][19][debug][pool] [source/common/http/http2/conn_pool.cc:175] [C3] destroying primary client
[2019-05-09 19:20:39.091][1][debug][main] [source/server/server.cc:147] flushing stats

I tried a bunch of things on the Envoy config side, but nothing made any difference. Could anyone point me at what might be causing the issue or what might help me get to the bottom of this?