grpc: verify_peer_callback ignored in SSL handshake

Sorry for Posting this issue again, but I could not reopen #19845 and believe it has been closed due to a misunderstanding. The issue still persists in the Code and unfortunately, I am not proficient enough with the C-core to have an idea how to resolve this.

What version of gRPC and what language are you using?

1.22-2.25 / C#

What operating system (Linux, Windows,…) and version?

Windows 10 64bit

What runtime / compiler are you using (e.g. python version or version of gcc)

.NET 4.7.2

What did you do?

I would like to implement a client that can connect to servers that use self-signed certificates. Upon a connection, I would like to ask the user whether the server should be trusted. Recently, the verify_peer_callback that existed for ages in the C library has been exposed to C#, but it seems to be applied only after a successful SSL handshake. However, in my situation I would need to override the peer verification and handle it in user code, in order to accept connections to servers with self-signed certificates whose CA is unknown in advance.

I tested the Scenario with the following C# code, modifying the greeter example:

Server

var cacert = File.ReadAllText(@"..\ca.crt");
var servercert = File.ReadAllText(@"..\server.crt");
var serverkey = File.ReadAllText(@"..\server.key");
var keypair = new KeyCertificatePair(servercert, serverkey);
var sslCredentials = new SslServerCredentials(new List<KeyCertificatePair>() {keypair}, cacert, SslClientCertificateRequestType.DontRequest);
Server server = new Server
{
    Services = {Greeter.BindService(new GreeterImpl())},
    Ports = {new ServerPort("localhost", Port, sslCredentials)}
};
server.Start();
Console.WriteLine("Greeter server listening on port " + Port);
Console.WriteLine("Press any key to stop the server...");
Console.ReadKey();
server.ShutdownAsync().Wait();

Client

var cacert = File.ReadAllText(@"..\ca2.crt");
var clientcert = File.ReadAllText(@"..\client.crt");
var clientkey = File.ReadAllText(@"..\client.key");
var ssl = new SslCredentials(cacert, new KeyCertificatePair(clientcert, clientkey), VerifyPeer);
var channel = new Channel("localhost", 51000, ssl);
var client = new Greeter.GreeterClient(channel);
String user = "you";

var reply = client.SayHello(new HelloRequest {Name = user});
Console.WriteLine("Greeting: " + reply.Message);

channel.ShutdownAsync().Wait();
Console.WriteLine("Press any key to exit...");
Console.ReadKey();

The method VerifyPeer looks like this:

private static bool VerifyPeer(VerifyPeerContext context)
{
  return true;
}

A complete minimal example is here: https://github.com/georghinkel/grpc/commit/e0a625f5646f50df1161077671a4367b7d636d51

This also includes scripts to generate the self-signed certificates using openssl.

What did you expect to see?

gRPC would call VerifyPeer in order to check whether a connection to the host should be established. Since the method is implemented to just return true, the connection should be established successfully.

What did you see instead?

The script generates two client certificates, one with the same CA as the server and one that uses a different one. If I start the client using the same CA as the server, the verify peer callback is executed and allows me to fail the connection if I returned false. If the client uses a different CA, the verify_peer_callback is not executed but the connection fails before because the SSL handshake does not consider the callback.

Instead, the following error message occurs:

Grpc.Core.RpcException: ‘Status(StatusCode=Unavailable, Detail=“failed to connect to all addresses”)’ at Grpc.Core.Internal.AsyncCall`2.UnaryCall(TRequest msg) in T:\src\github\grpc\src\csharp\Grpc.Core\Internal\AsyncCall.cs:line 78

at Grpc.Core.DefaultCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request) in T:\src\github\grpc\src\csharp\Grpc.Core\DefaultCallInvoker.cs:line 46

at Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[TRequest,TResponse](TRequest req, ClientInterceptorContext`2 ctx) in T:\src\github\grpc\src\csharp\Grpc.Core.Api\Interceptors\InterceptingCallInvoker.cs:line 51

at Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall[TRequest,TResponse](TRequest request, ClientInterceptorContext2 context, BlockingUnaryCallContinuation2 continuation) in T:\src\github\grpc\src\csharp\Grpc.Core\ClientBase.cs:line 174

at Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request) in T:\src\github\grpc\src\csharp\Grpc.Core.Api\Interceptors\InterceptingCallInvoker.cs:line 48

at Helloworld.Greeter.GreeterClient.SayHello(HelloRequest request, CallOptions options) in C:\Projects\grpc\examples\csharp\Helloworld\Greeter\obj\Debug\netstandard1.5\HelloworldGrpc.cs:line 109

at Helloworld.Greeter.GreeterClient.SayHello(HelloRequest request, Metadata headers, Nullable`1 deadline, CancellationToken cancellationToken) in C:\Projects\grpc\examples\csharp\Helloworld\Greeter\obj\Debug\netstandard1.5\HelloworldGrpc.cs:line 99

at GreeterClient.Program.Main(String[] args) in C:\Projects\grpc\examples\csharp\Helloworld\GreeterClient\Program.cs:line 34

I put a breakpoint into the VerifyPeer method, but it has not been called.

The trace is the following:

I1209 14:09:49.231985 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\channel\handshaker.cc:178: handshake_manager 00000232C3F1BB20: calling handshaker security [00000232DC973300] at index 1
E1209 14:09:49.236540 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\tsi\ssl_transport_security.cc:1246: Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
D1209 14:09:49.236650 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\security\transport\security_handshaker.cc:186: Security handshake failed: {"created":"@1575896989.236000000","description":"Handshake failed","file":"T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\security\transport\security_handshaker.cc","file_line":303,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}
I1209 14:09:49.237319 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\channel\handshaker.cc:131: handshake_manager 00000232C3F1BB20: error={"created":"@1575896989.236000000","description":"Handshake failed","file":"T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\security\transport\security_handshaker.cc","file_line":303,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"} shutdown=0 index=2, args={endpoint=0000000000000000, args=0000000000000000 {size=0: (null)}, read_buffer=0000000000000000 (length=0), exit_early=0}
I1209 14:09:49.238382 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\channel\handshaker.cc:164: handshake_manager 00000232C3F1BB20: handshaking complete -- scheduling on_handshake_done with error={"created":"@1575896989.236000000","description":"Handshake failed","file":"T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\security\transport\security_handshaker.cc","file_line":303,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}

I also tried to install the server certificate or the CA in the Windows user certificate store as trusted root certificate but that didn’t help me to connect to the server. Currently, passing the server CA as client CA seems the only option to connect to a server that uses a self-signed certificate.

Anything else we should know about your project / environment?

Our intended usage of gRPC is for laboratory automation where each lab device would act as a SiLA2 server. SiLA2 is a protocol recently established for communication between laboratory devices and it uses gRPC for the transport layer. The standard requires servers to use TLS, but allows self-signed certificates. Of course, I can try to memorize the CA for each server, but I would like to get rid of this manual effort.