grpc: Cannot check peer: missing selected ALPN property in python client (grpcio)
What version of gRPC and what language are you using?
python grpcio 1.29.0
What operating system (Linux, Windows,…) and version?
bug seen on debian buster and ubuntu focal
What runtime / compiler are you using (e.g. python version or version of gcc)
Python 3.8
What did you do?
connecting to a grpc server with self-generated ssl certs. full repro available at https://github.com/hjwp/eventstoreio/tree/alpn-bug, should be a single command to repro (make test) using docker.
What did you expect to see?
a successful connection
What did you see instead?
Python stack trace:
E grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
E status = StatusCode.UNAVAILABLE
E details = "failed to connect to all addresses"
E debug_error_string = "{"created":"@1591774926.055334184","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3962,"referenced_errors":[{"created":"@1591774926.055331102","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":394,"grpc_status":14}]}"
E >
debug logs:
D0610 07:42:06.051285883 1 dns_resolver_ares.cc:503] Using ares dns resolver
I0610 07:42:06.051583095 1 socket_utils_common_posix.cc:385] Disabling AF_INET6 sockets because ::1 is not available.
I0610 07:42:06.052095702 10 socket_utils_common_posix.cc:341] TCP_USER_TIMEOUT not supported for this platform
I0610 07:42:06.052187919 10 tcp_client_posix.cc:326] CLIENT_CONNECT: ipv4:192.168.128.2:1113: asynchronously connecting fd 0x5608db1e1c40
I0610 07:42:06.052204158 10 tcp_client_posix.cc:143] CLIENT_CONNECT: ipv4:192.168.128.2:1113: on_writable: error="No Error"
I0610 07:42:06.052214803 10 tcp_client_posix.cc:106] CLIENT_CONNECT: ipv4:192.168.128.2:1113: on_alarm: error="Cancelled"
I0610 07:42:06.052235207 10 ssl_transport_security.cc:222] HANDSHAKE START - TLS client start_connect - !!!!!!
I0610 07:42:06.052264721 10 ssl_transport_security.cc:222] LOOP - TLS client enter_early_data - !!!!!!
I0610 07:42:06.052269364 10 ssl_transport_security.cc:222] LOOP - TLS client read_server_hello - !!!!!!
I0610 07:42:06.052278849 10 tcp_posix.cc:1566] WRITE 0x7f6e980063e0 (peer=ipv4:192.168.128.2:1113)
D0610 07:42:06.052285548 10 tcp_posix.cc:1570] DATA: [snip]
I0610 07:42:06.052308562 10 tcp_posix.cc:1616] write: "No Error"
I0610 07:42:06.052314530 10 tcp_posix.cc:541] TCP:0x7f6e980063e0 notify_on_read
I0610 07:42:06.054173605 10 tcp_posix.cc:887] TCP:0x7f6e980063e0 got_read: "No Error"
I0610 07:42:06.054183765 10 tcp_posix.cc:869] TCP:0x7f6e980063e0 alloc_slices
I0610 07:42:06.054191069 10 tcp_posix.cc:850] TCP:0x7f6e980063e0 read_allocation_done: "No Error"
I0610 07:42:06.054198411 10 tcp_posix.cc:680] TCP:0x7f6e980063e0 call_cb 0x7f6e98003958 0x7f6eab7b1020:0x7f6e98003790
I0610 07:42:06.054211086 10 tcp_posix.cc:683] READ 0x7f6e980063e0 (peer=ipv4:192.168.128.2:1113) error="No Error"
D0610 07:42:06.054227933 10 tcp_posix.cc:689] DATA: [snip]
I0610 07:42:06.054254150 10 ssl_transport_security.cc:222] LOOP - TLS client read_server_certifi - !!!!!!
I0610 07:42:06.054288928 10 ssl_transport_security.cc:222] LOOP - TLS client read_certificate_st - !!!!!!
I0610 07:42:06.054293858 10 ssl_transport_security.cc:222] LOOP - TLS client verify_server_certi - !!!!!!
I0610 07:42:06.054323654 10 ssl_transport_security.cc:222] LOOP - TLS client read_server_key_exc - !!!!!!
I0610 07:42:06.054386032 10 ssl_transport_security.cc:222] LOOP - TLS client read_certificate_re - !!!!!!
I0610 07:42:06.054394707 10 ssl_transport_security.cc:222] LOOP - TLS client read_server_hello_d - !!!!!!
I0610 07:42:06.054399817 10 ssl_transport_security.cc:222] LOOP - TLS client send_client_certifi - !!!!!!
I0610 07:42:06.054404562 10 ssl_transport_security.cc:222] LOOP - TLS client send_client_key_exc - !!!!!!
I0610 07:42:06.054752050 10 ssl_transport_security.cc:222] LOOP - TLS client send_client_certifi - !!!!!!
I0610 07:42:06.054760016 10 ssl_transport_security.cc:222] LOOP - TLS client send_client_finishe - !!!!!!
I0610 07:42:06.054775387 10 ssl_transport_security.cc:222] LOOP - TLS client finish_flight - !!!!!!
I0610 07:42:06.054782998 10 ssl_transport_security.cc:222] LOOP - TLS client read_session_ticket - !!!!!!
I0610 07:42:06.054787910 10 tcp_posix.cc:1566] WRITE 0x7f6e980063e0 (peer=ipv4:192.168.128.2:1113)
D0610 07:42:06.054796339 10 tcp_posix.cc:1570] DATA: [snip]'
I0610 07:42:06.054818859 10 tcp_posix.cc:1616] write: "No Error"
I0610 07:42:06.054824701 10 tcp_posix.cc:887] TCP:0x7f6e980063e0 got_read: "No Error"
I0610 07:42:06.054828075 10 tcp_posix.cc:879] TCP:0x7f6e980063e0 do_read
I0610 07:42:06.054832197 10 tcp_posix.cc:541] TCP:0x7f6e980063e0 notify_on_read
I0610 07:42:06.055157301 10 tcp_posix.cc:887] TCP:0x7f6e980063e0 got_read: "No Error"
I0610 07:42:06.055168218 10 tcp_posix.cc:879] TCP:0x7f6e980063e0 do_read
I0610 07:42:06.055175776 10 tcp_posix.cc:680] TCP:0x7f6e980063e0 call_cb 0x7f6e98003958 0x7f6eab7b1020:0x7f6e98003790
I0610 07:42:06.055179917 10 tcp_posix.cc:683] READ 0x7f6e980063e0 (peer=ipv4:192.168.128.2:1113) error="No Error"
D0610 07:42:06.055189106 10 tcp_posix.cc:689] DATA: [snip]
I0610 07:42:06.055205044 10 ssl_transport_security.cc:222] LOOP - TLS client process_change_ciph - !!!!!!
I0610 07:42:06.055211849 10 ssl_transport_security.cc:222] LOOP - TLS client read_server_finishe - !!!!!!
I0610 07:42:06.055221738 10 ssl_transport_security.cc:222] LOOP - TLS client finish_client_hands - !!!!!!
I0610 07:42:06.055227526 10 ssl_transport_security.cc:222] LOOP - TLS client done - !!!!!!
I0610 07:42:06.055230842 10 ssl_transport_security.cc:222] HANDSHAKE DONE - TLS client done - !!!!!!
I0610 07:42:06.055239580 10 ssl_transport_security.cc:279] Could not get common name of subject from certificate.
D0610 07:42:06.055268495 10 security_handshaker.cc:184] Security handshake failed: {"created":"@1591774926.055262608","description":"Cannot check peer: missing selected ALPN property.","file":"src/core/lib/security/security_connector/ssl_utils.cc","file_line":142}
Anything else we should know about your project / environment?
found this SO post from sept 2019 claiming the latest python client is built against a broken version of openssl https://stackoverflow.com/questions/57397723/grpc-client-failing-to-connect-to-server-with-tls-certificates – could that still be true?
[update] - it looks like reverting to grpcio 1.19 may fix/bypass the issue. yes
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 7
- Comments: 33 (11 by maintainers)
Commits related to this issue
- rpcserver: fix Python gRPC client ALPN property error There is an open issue for Python gRPC clients that is currently being debugged https://github.com/grpc/grpc/issues/23172 It can be mitigated se... — committed to PierreRochard/pool by PierreRochard 3 years ago
- rpcserver: fix Python gRPC client ALPN property error There is an open issue for Python gRPC clients that is currently being debugged https://github.com/grpc/grpc/issues/23172 It can be mitigated se... — committed to lightninglabs/pool by PierreRochard 3 years ago
- loopd: fix ALPN issue with Python There is an open issue for Python gRPC clients that is currently being debugged grpc/grpc#23172 It can be mitigated server-side by specifying h2 in the metadata hea... — committed to guggero/loop by guggero 3 years ago
Hi, I’m facing a similar issue when try to use SSL certificates in my C++ client and server applications along with SAN(subject alternative names).There is no issue if donot use the SAN feature for the certificate generations.
GRPC Version - 1.37.0-dev openssl version - 1.1.1
Steps to reproduce :-
Generate CA root, server and client keys/certificates for mutual SSL authentication using the steps as attached in gen_certs.sh Provide an extension file named server certificates.zip -ext.cnf/client-ext.cnf so that alternate subject domain names can be inserted while generating server/client certificates. Both pairs of server and client certificates are used in c++ server & client applications. Expected result :- API’s should be triggered without any errors.
Actual result :- Get the below error message on the client side
Handshake failed with fatal error SSL_ERROR_SSL: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed. 14: failed to connect to all addresses RPC failed
On enabling GRPC_VERBOSITY=DEBUG and GRPC_TRACE=ALL , I see the below messages Security handshake failed: {“created”:“@1627636862.516902426”,“description”:“Cannot check peer: missing selected ALPN property.”,“file”:“/home/chirag/grpc/src/core/lib/security/security_connector/ssl_utils.cc”,“file_line”:160} I0730 14:51:02.516927398 29079 tcp_posix.cc:1219] TCP:0x7f50b000ec00 got_error: {“created”:“@1627636862.516920791”,“description”:“FD Shutdown”,“file”:“/home/chirag/grpc/src/core/lib/iomgr/lockfree_event.cc”,“file_line”:199,“referenced_errors”:[{“created”:“@1627636862.516902426”,“description”:“Cannot check peer: missing selected ALPN property.”,“file”:“/home/chirag/grpc/src/core/lib/security/security_connector/ssl_utils.cc”,“file_line”:160}]} I0730 14:51:02.516934247 29079 handshaker.cc:129] handshake_manager 0x7f50b000f5f0: error={“created”:“@1627636862.516902426”,“description”:“Cannot check peer: missing selected ALPN property.”,“file”:“/home/chirag/grpc/src/core/lib/security/security_connector/ssl_utils.cc”,“file_line”:160} shutdown=0 index=1, args={endpoint=(nil), args=(nil) {size=0: }, read_buffer=(nil) (length=0), exit_early=0} I0730 14:51:02.516937277 29079 handshaker.cc:162] handshake_manager 0x7f50b000f5f0: handshaking complete – scheduling on_handshake_done with error={“created”:“@1627636862.516902426”,“description”:“Cannot check peer: missing selected ALPN property.”,“file”:“/home/chirag/grpc/src/core/lib/security/security_connector/ssl_utils.cc”,“file_line”:160} I0730 14:51:02.516941179 29079 timer_generic.cc:470] TIMER 0x7f50b000f658: CANCEL pending=true D0730 14:51:02.516944126 29079 chttp2_server.cc:253] Handshaking failed: {“created”:“@1627636862.516902426”,“description”:“Cannot check peer: missing selected ALPN property.”,“file”:“/home/chirag/grpc/src/core/lib/security/security_connector/ssl_utils.cc”,“file_line”:160} I0730 14:51:02.516962593 29079 resource_quota.cc:953] RQ anonymous_pool_139984526961248 ipv4:192.168.0.104:57436: free 8192; free_pool -> 8192 I0730 14:51:02.516982234 29079 resource_quota.cc:553] RU shutdown 0x7f50b000f470
I’ve been stuck here since past few days and any prompt help would be highly appreciated. certificates.zip
@gnossen same here, it looks so common usage, like Python client simply just cannot connect to gRPC server. Any update is much appreciated.
@niya3 I was able to reproduce using your example! Thanks for the help! 🙂
I’ll keep you updated as we figure out more.
@ZhenLian It’s hermetic in the sense that we use a git submodule to pull it in. Updating it to the latest version is part of our release process, but we may not be 100% successful in that. Regardless, I don’t think we’ve ever fallen more than a couple of months behind.
@niya3 You’ve got the
movinstruction (b8) mixed into the same integer as the version number. That is actually the same as what I posted above. Take a look at the disassembly for that instruction in my comment.