istio: istio-proxy error 504 on GRPC Server stream

Istio : 0.2.10 Kubernetes : 1.7.8 on GKE Client/Server made with Go 1.9.2 and gRPC 1.6

Overview : When using the “Server Stream” feature of GRPC/HTTP2, Istio-proxy close the connexion with a 504 error after some timeout.

Details : I have a GRCP client that connect to my Istioified Micro Service (Useredge). This client do regular POST on /lab.useredgesvc.UserEdgeSVC/UserEventReport endpoint, which is described in the Proto file as : rpc UserEventReport (lab.commonmodel.UserEvent) returns (google.protobuf.Empty);

This is working fine.

The client also do a POST request on /lab.useredgesvc.UserEdgeSVC/LiveFeed which is described in the proto file as : rpc LiveFeed(DeviceMsg) returns (stream lab.commonmodel.AlertEvent);

As you can see, this is a gRPC Stream that the server is supposed to use to send some data to the client when needed.

While the first endpoint is sending data every 10 secs without issue, Istio-proxy always report a 504 error on the Stream endpoint :

“POST /lab.useredgesvc.UserEdgeSVC/LiveFeed HTTP/2” 504 UT 30 24 15002 - “-” “grpc-go/1.6.0” “6db37f84-91d3-91d8-ad4d-14ce5a7ab366” “useredged.dev:12900” “127.0.0.1:12900”

For testing purpose, I installed the client in one of my non-istio pods, and I request the service of the server pod which is running with the sidecar.

I tried to add debug logs to some istio-proxy modules, without more luck to identify anything useful.

As the Stream call is the first made when the client start, it sounds like some sort of timeout in the istio-proxy as the error only shows up almost 30s after the client first connect.

I’m still not sure if it’s a bug or a lack of support for such gRPC call ?