grpc-go: Unexpected transport closing: too many pings from client

What version of gRPC are you using?

583a6303969ea5075e9bd1dc4b75805dfe66989a

What version of Go are you using (`go version`)?

1.10

What operating system (Linux, Windows, …) and version?

Linux AMD64, Kernel 4.10

What did you do?

When I have the server configured with GZIP compression as so:

gzcp := grpc.NewGZIPCompressor()
grpcServer := grpc.NewServer(grpc.RPCCompressor(gzcp))

Then when serving thousands of concurrent requests a second, clients will occasionally be disconnected with

rpc error: code = Unavailable desc = transport is closing

I see no errors from the server, and the both the client and server are far from overloaded (<10% CPU usage etc). Not all clients are affected at once, it will just be one connection which gets this error.

While trying to debug this, I disabled GZIP compression so I could use more easily look at packet captures. I am unable to reproduce this error once the GZIP compressor is no longer in use.

This issue is mostly to ask what the best way to proceed with diagnosing the problem is, or if there are any reasons why having a compressor would change the behavior of the system (aside from CPU usage which I don’t think is a problem).

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 33 (13 by maintainers)

Most upvoted comments

If it’s not keepalive then the only thing inside gRPC that sends pings is bdp estimator which will be triggered on receiving data. However, every time the server sends data frames it resets its pingStrike counter. To quickly check if it’s the bdp estimator or not, try turning it off by setting WithInitialWindowSize and WIthInitialConnectionWindowSize dial options on the client.

MakMukhi on Feb 25, 2018

Turn on transport level logs by setting these two env variables: export GRPC_GO_LOG_VERBOSITY_LEVEL=2 export GRPC_GO_LOG_SEVERITY_LEVEL=info

May be the transport on the server of client side sees some error and closes.

On Sat, Feb 24, 2018 at 4:26 PM, Michael Andersen notifications@github.com wrote:

Sorry, I can reproduce this even without the GZIP compressor. I still don’t know how to debug it though. The only error I get on the client with loglevel set to info is:

pickfirstBalancer: HandleSubConnStateChange: 0xc421bc6160, TRANSIENT_FAILURE

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/grpc/grpc-go/issues/1882#issuecomment-368271626, or mute the thread https://github.com/notifications/unsubscribe-auth/ATtnR6yLJFRNEv-Nn5h7ILZfqwbUZoMFks5tYKisgaJpZM4SSF8Z .