botan: TLS handshake fails with DECRYPT_ERROR after upgrading from 2.3.0 to 2.7.0

We have two components in our product that communicate (in the same process) via a TLS connection. With Botan 2.3.0, the connection could be established but I tried to update to Botan 2.7.0, which raises a decoding error during the handshake.

I’m not really sure how to analyze this error any more because it originates from deep inside of Botan code. I only noticed that the call stacks were different between the two Botan versions (see below).

The DECRYPT_ERROR exception is raised in tls_client.cpp, line 485:

if(!state.server_kex()->verify(server_key, state, policy()))
   {
   throw TLS_Exception(Alert::DECRYPT_ERROR,
                       "Bad signature on server key exchange");
   }
}

The verify() check fails (after a long call stack) in pss_verify() because of the following condition:

if(const_coded[const_coded.size()-1] != 0xBC)
   return false;

Call stack (2.7.0):

Frontend-cli.exe!Botan::`anonymous namespace'::pss_verify(Botan::HashFunction & hash={...}, const std::vector<unsigned char,Botan::secure_allocator<unsigned char> > & const_coded={...}, const std::vector<unsigned char,Botan::secure_allocator<unsigned char> > & raw={...}, unsigned __int64 key_bits=4095) Line 75	C++
Frontend-cli.exe!Botan::PSSR::verify(const std::vector<unsigned char,Botan::secure_allocator<unsigned char> > & coded={...}, const std::vector<unsigned char,Botan::secure_allocator<unsigned char> > & raw={...}, unsigned __int64 key_bits=4095) Line 165	C++
Frontend-cli.exe!Botan::PK_Ops::Verification_with_EMSA::is_valid_signature(const unsigned char * sig=0x000002254a122ca0, unsigned __int64 sig_len=512) Line 124	C++
Frontend-cli.exe!Botan::PK_Verifier::check_signature(const unsigned char * sig=0x000002254a122ca0, unsigned __int64 length=512) Line 310	C++
Frontend-cli.exe!Botan::PK_Verifier::verify_message(const unsigned char * msg=0x000002254a123ec0, unsigned __int64 msg_length=1922, const unsigned char * sig=0x000002254a122ca0, unsigned __int64 sig_length=512) Line 298	C++
Frontend-cli.exe!Botan::PK_Verifier::verify_message<std::allocator<unsigned char>,std::allocator<unsigned char> >(const std::vector<unsigned char,std::allocator<unsigned char> > & msg={...}, const std::vector<unsigned char,std::allocator<unsigned char> > & sig={...}) Line 328	C++
Frontend-cli.exe!Botan::TLS::Callbacks::tls_verify_message(const Botan::Public_Key & key={...}, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & emsa={...}, Botan::Signature_Format format=IEEE_1363, const std::vector<unsigned char,std::allocator<unsigned char> > & msg={...}, const std::vector<unsigned char,std::allocator<unsigned char> > & sig={...}) Line 98	C++
Frontend-cli.exe!Botan::TLS::Server_Key_Exchange::verify(const Botan::Public_Key & server_key={...}, const Botan::TLS::Handshake_State & state={...}, const Botan::TLS::Policy & policy={...}) Line 315	C++
Frontend-cli.exe!Botan::TLS::Client::process_handshake_msg(const Botan::TLS::Handshake_State * active_state=0x0000000000000000, Botan::TLS::Handshake_State & state_base={...}, Botan::TLS::Handshake_Type type=SERVER_KEX, const std::vector<unsigned char,std::allocator<unsigned char> > & contents={...}) Line 483	C++
Frontend-cli.exe!Botan::TLS::Channel::process_handshake_ccs(const std::vector<unsigned char,Botan::secure_allocator<unsigned char> > & record={...}, unsigned __int64 record_sequence=0, Botan::TLS::Record_Type record_type=HANDSHAKE, Botan::TLS::Protocol_Version record_version={...}) Line 436	C++
Frontend-cli.exe!Botan::TLS::Channel::received_data(const unsigned char * input=0x000002254a0f3e2a, unsigned __int64 input_size=150) Line 339	C++

Call stack (2.3.0):

Frontend-cli.exe!Botan::EMSA_PKCS1v15::verify(const std::vector<unsigned char,Botan::secure_allocator<unsigned char> > & coded={...}, const std::vector<unsigned char,Botan::secure_allocator<unsigned char> > & raw={...}, unsigned __int64 key_bits=4095) Line 23763	C++
Frontend-cli.exe!Botan::PK_Ops::Verification_with_EMSA::is_valid_signature(const unsigned char * sig=0x00000201a8394900, unsigned __int64 sig_len=512) Line 47222	C++
Frontend-cli.exe!Botan::PK_Verifier::check_signature(const unsigned char * sig=0x00000201a8394900, unsigned __int64 length=512) Line 47986	C++
Frontend-cli.exe!Botan::PK_Verifier::check_signature<std::allocator<unsigned char> >(const std::vector<unsigned char,std::allocator<unsigned char> > & sig={...}) Line 12325	C++
Frontend-cli.exe!Botan::TLS::Server_Key_Exchange::verify(const Botan::Public_Key & server_key={...}, const Botan::TLS::Handshake_State & state={...}, const Botan::TLS::Policy & policy={...}) Line 58287	C++
Frontend-cli.exe!Botan::TLS::Client::process_handshake_msg(const Botan::TLS::Handshake_State * active_state=0x0000000000000000, Botan::TLS::Handshake_State & state_base={...}, Botan::TLS::Handshake_Type type=SERVER_KEX, const std::vector<unsigned char,std::allocator<unsigned char> > & contents={...}) Line 59904	C++
Frontend-cli.exe!Botan::TLS::Channel::process_handshake_ccs(const std::vector<unsigned char,Botan::secure_allocator<unsigned char> > & record={...}, unsigned __int64 record_sequence=0, Botan::TLS::Record_Type record_type=HANDSHAKE, Botan::TLS::Protocol_Version record_version={...}) Line 59040	C++
Frontend-cli.exe!Botan::TLS::Channel::received_data(const unsigned char * input=0x00000201a8363fca, unsigned __int64 input_size=144) Line 58943	C++

Here are also two Wireshark traces of the TLS handshakes:

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 15 (15 by maintainers)

Most upvoted comments

One thing you could try with just modifications to your application code, override the TLS callbacks that handle signing (TLS::Callbacks::tls_sign_message and tls_verify_message to log their respective inputs and outputs):

std::vector<uint8_t> MyTLSCallbacks::tls_sign_message(
   const Private_Key& key,
   RandomNumberGenerator& rng,
   const std::string& emsa,
   Signature_Format format,
   const std::vector<uint8_t>& msg)                                                                                                                                                            
   {                                                                                                                                                                                           
   PK_Signer signer(key, rng, emsa, format);
    auto sig = signer.sign_message(msg, rng);
  PK_Verifier verifier(key, emsa, format);
   BOTAN_ASSERT_NOMSG(verifier.verify_message(msg, sig));
   // print msg, sig, emsa
   return sig;
   }                                                                                                                                                                                           
                                                                                                                                                                                               
bool MyTLSCallbacks::tls_verify_message(                                                                                                                                                       
   const Public_Key& key,                                                                                                                                                                      
   const std::string& emsa,                                                                                                                                                                    
   Signature_Format format,                                                                                                                                                                    
   const std::vector<uint8_t>& msg,                                                                                                                                                            
   const std::vector<uint8_t>& sig)                                                                                                                                                            
   {                                                                                                                                                                                           
   // print msg, sig, emsa
   PK_Verifier verifier(key, emsa, format);
   return verifier.verify_message(msg, sig);
   }

If the originally generated signature fails to verify, that is interesting. If the input messages for sign and verify differ, that is interesting. If the same signature verifies in on callback and fails in another, that is interesting.

randombit on Aug 29, 2018