grpc: It's hard to debug connectivity_state errors...

What version of gRPC and what language are you using?

Grpc.Core, 1.19.0, C#.

What operating system (Linux, Windows,…) and version?

Windows 10

What runtime / compiler are you using (e.g. python version or version of gcc)

Visual Studio 2017, .NET Core 2.0. dotnet compiler version: 2.1.505

What did you do?

I set my API server endpoint to a non-existent endpoint. However, I cannot get the grpc client library to fail reliably.

Half the time, it fails with Status(StatusCode=Unavailable, Detail="Name resolution failure") The other half, it fails with Status(StatusCode=Unavailable, Detail="channel is in state TRANSIENT_FAILURE")

The more troubling fact is that it doesn’t give me additional useful debugging information. I have logging like this:

Environment.SetEnvironmentVariable("GRPC_VERBOSITY", "DEBUG");
Environment.SetEnvironmentVariable("GRPC_TRACE", "connectivity_state");

But when things fail, the logs don’t tell me what failed. Is it a 404 on the URL? Flaky connection? Something else? The logs look like this instead.

D0401 14:58:01.610543 Grpc.Core.Internal.UnmanagedLibrary Attempting to load native library "C:\Users\anash\.nuget\packages\grpc.core\1.19.0\lib\netstandard1.5\../..\runtimes/win/native\grpc_csharp_ext.x64.dll"
D0401 14:58:01.647022 Grpc.Core.Internal.NativeExtension gRPC native library loaded successfully.
D0401 14:58:01.649159 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\ext\filters\client_channel\resolver\dns\c_ares\dns_resolver_ares.cc:488: Using ares dns resolver
I0401 14:58:01.810156 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\transport\connectivity_state.cc:81: CONWATCH: 00000258EC5E7A10 request_router: get IDLE
I0401 14:58:01.902005 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\transport\connectivity_state.cc:164: SET: 00000258EC5E7A10 request_router: IDLE --> TRANSIENT_FAILURE [resolver_result] error=00000258ED5CBE50 {"created":"@1554145081.901000000","description":"No load balancing policy","file":"T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\ext\filters\client_channel\request_routing.cc","file_line":806}

Additional details: I have multiple users reporting https://github.com/googleads/google-ads-dotnet/issues/59, but the lack of good logging and the random nature of the error being thrown is preventing me from debugging things further.