openssl: Current master breaks libp11 RSA signature

MacOS Catalina 10.15.3, Xcode-11.3.1, installed OpenSSL-1.1.1d (reference) and (in a separate dir tree) OpenSSL-3.0 master.

Using libp11 to perform RSA signature using hardware token accessible via PKCS#11. Fails with both a real hardware token, and SoftHSMv2.

With OpenSSL-1.1.1d (succeeds):

$ pkcs11-rsa-pss-sign-demo2
Generating ephemeral file /tmp/derive.89314.text to test RSA-PSS signature...

openssl rand -hex -out /tmp/derive.89314.text 5120

Signing file /tmp/derive.89314.text...
openssl dgst -engine pkcs11 -keyform engine -sign "pkcs11:manufacturer=piv_II;object=SIGN%20key;type=private" -sha384 -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-1 -out /tmp/derive.89314.text.sig /tmp/derive.89314.text
engine "pkcs11" set.
Enter PKCS#11 token PIN for xxxxxxxxx:
Signature is stored in /tmp/derive.89314.text.sig

Verifying signature:
openssl dgst -engine pkcs11 -keyform engine -sha384 -verify "pkcs11:manufacturer=piv_II;object=SIGN%20pubkey;type=public" -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-1 -signature /tmp/derive.89314.text.sig  /tmp/derive.89314.text
engine "pkcs11" set.
Verified OK

$

With OpenSSL-3.0 (current master, fails):

$ pkcs11-rsa-pss-sign-demo3
Generating ephemeral file /tmp/derive.89324.text to test RSA-PSS signature...

openssl3 rand -hex -out /tmp/derive.89324.text 5120

Signing file /tmp/derive.89324.text...
openssl3 dgst -engine pkcs11 -keyform engine -sign "pkcs11:manufacturer=piv_II;object=SIGN%20key;type=private" -sha384 -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-1 -out /tmp/derive.89324.text.sig /tmp/derive.89324.text
engine "pkcs11" set.
Enter PKCS#11 token PIN for xxxxxxxxx:
Error Signing Data
C0:2D:B6:09:01:00:00:00:error:rsa routines:rsa_ossl_private_encrypt:missing private key:crypto/rsa/rsa_ossl.c:333:
C0:2D:B6:09:01:00:00:00:error:Provider routines:rsa_sign:RSA lib:providers/implementations/signature/rsa.c:410:

Also, see this libp11 issue for details of a failure trying to sign using SoftHSMv2: https://github.com/OpenSC/libp11/issues/331

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 34 (33 by maintainers)

Commits related to this issue

Most upvoted comments

Thank you for trying it out! It’s really nice to get this kind of bug report so early on (before we even have an alpha release out!).

It looks like OpenSSL 3.0 ignores the engine and tries to sign the key with rsa_ossl_private_encrypt(),

Thanks, I suspected that

The OpenSSL ENGINE API was never designed to be reentrant. The only solution I can think of is disabling the use of engines in SoftHSM2.

I stumbled over the same problem (the OpenSSL ENGINE being not reentrant).

I worked around it by using the p11-kit-proxy.so as the first pkcs#11 module - with a remote: |p11-kit-remote /path/to/original-pkcs11.so. As original-pkcs11.so runs in a different process, it can internally use another OpenSSL ENGINE instance without causing problems.

@levitte: You’re right. It looks more complex indeed.

There are some double free issues, and corrupted heap results in crash locations other than the actual root cause. I collected a valgrind log: valgrind.txt

An example double free:

==1926254== Invalid free() / delete / delete[] / realloc()
==1926254==    at 0x483A9AB: free (vg_replace_malloc.c:540)
==1926254==    by 0x5252DDB: OSDestroyMutex(void*) (osmutex.cpp:92)
==1926254==    by 0x5253BE5: DestroyMutex (MutexFactory.cpp:176)
==1926254==    by 0x5253BE5: DestroyMutex (MutexFactory.cpp:172)
==1926254==    by 0x5253BE5: Mutex::~Mutex() [clone .part.0] (MutexFactory.cpp:54)
==1926254==    by 0x5253C3D: ~Mutex (MutexFactory.cpp:50)
==1926254==    by 0x5253C3D: Mutex::~Mutex() (MutexFactory.cpp:56)
==1926254==    by 0x52725B0: HandleManager::~HandleManager() (HandleManager.cpp:60)
==1926254==    by 0x52725D8: HandleManager::~HandleManager() (HandleManager.cpp:61)
==1926254==    by 0x5229577: SoftHSM::C_Finalize(void*) (SoftHSM.cpp:547)
==1926254==    by 0x520ECE5: C_Finalize (main.cpp:148)
==1926254==    by 0x485301F: pkcs11_CTX_unload (p11_load.c:149)
==1926254==    by 0x484EE0A: ctx_finish (eng_back.c:359)
==1926254==    by 0x484CCE7: engine_finish (eng_front.c:159)
==1926254==    by 0x4A798CE: ??? (in /usr/lib/libcrypto.so.1.1)
==1926254==    by 0x4A7B7B3: ??? (in /usr/lib/libcrypto.so.1.1)
==1926254==    by 0x4A9E7E3: OPENSSL_LH_doall (in /usr/lib/libcrypto.so.1.1)
==1926254==    by 0x4A7BAEE: ??? (in /usr/lib/libcrypto.so.1.1)
==1926254==    by 0x4A79AF5: ??? (in /usr/lib/libcrypto.so.1.1)
==1926254==    by 0x4B03BAF: OPENSSL_sk_pop_free (in /usr/lib/libcrypto.so.1.1)
==1926254==    by 0x4A79E5C: ??? (in /usr/lib/libcrypto.so.1.1)
==1926254==    by 0x4A9BDE0: OPENSSL_cleanup (in /usr/lib/libcrypto.so.1.1)
==1926254==    by 0x4C5A536: __run_exit_handlers (in /usr/lib/libc-2.31.so)
==1926254==    by 0x4C5A6ED: exit (in /usr/lib/libc-2.31.so)
==1926254==    by 0x13628B: ??? (in /usr/bin/openssl)
==1926254==    by 0x4C43022: (below main) (in /usr/lib/libc-2.31.so)
==1926254==  Address 0x4fe5a00 is 0 bytes inside a block of size 40 free'd
==1926254==    at 0x483A9AB: free (vg_replace_malloc.c:540)
==1926254==    by 0x5252DDB: OSDestroyMutex(void*) (osmutex.cpp:92)
==1926254==    by 0x5253BE5: DestroyMutex (MutexFactory.cpp:176)
==1926254==    by 0x5253BE5: DestroyMutex (MutexFactory.cpp:172)
==1926254==    by 0x5253BE5: Mutex::~Mutex() [clone .part.0] (MutexFactory.cpp:54)
==1926254==    by 0x5253C3D: ~Mutex (MutexFactory.cpp:50)
==1926254==    by 0x5253C3D: Mutex::~Mutex() (MutexFactory.cpp:56)
==1926254==    by 0x52725B0: HandleManager::~HandleManager() (HandleManager.cpp:60)
==1926254==    by 0x52725D8: HandleManager::~HandleManager() (HandleManager.cpp:61)
==1926254==    by 0x5228A9C: SoftHSM::~SoftHSM() (SoftHSM.cpp:367)
==1926254==    by 0x5228AE8: SoftHSM::~SoftHSM() (SoftHSM.cpp:373)
==1926254==    by 0x4C5A536: __run_exit_handlers (in /usr/lib/libc-2.31.so)
==1926254==    by 0x4C5A6ED: exit (in /usr/lib/libc-2.31.so)
==1926254==    by 0x13628B: ??? (in /usr/bin/openssl)
==1926254==    by 0x4C43022: (below main) (in /usr/lib/libc-2.31.so)
==1926254==  Block was alloc'd at
==1926254==    at 0x483977F: malloc (vg_replace_malloc.c:309)
==1926254==    by 0x5252D14: OSCreateMutex(void**) (osmutex.cpp:49)
==1926254==    by 0x5253BC4: CreateMutex (MutexFactory.cpp:169)
==1926254==    by 0x5253BC4: CreateMutex (MutexFactory.cpp:165)
==1926254==    by 0x5253BC4: Mutex::Mutex() (MutexFactory.cpp:46)
==1926254==    by 0x5253DCC: MutexFactory::getMutex() (MutexFactory.cpp:125)
==1926254==    by 0x5272567: HandleManager::HandleManager() (HandleManager.cpp:52)
==1926254==    by 0x522930F: SoftHSM::C_Initialize(void*) (SoftHSM.cpp:531)
==1926254==    by 0x520ECC5: C_Initialize (main.cpp:133)
==1926254==    by 0x4852E66: pkcs11_CTX_load (p11_load.c:88)
==1926254==    by 0x484D136: ctx_init_libp11_unlocked (eng_back.c:281)
==1926254==    by 0x484D136: ctx_init_libp11 (eng_back.c:312)
==1926254==    by 0x484D540: ctx_load_key (eng_back.c:681)
==1926254==    by 0x484EF2A: ctx_load_privkey (eng_back.c:966)
==1926254==    by 0x484CC01: load_privkey (eng_front.c:185)
==1926254==    by 0x4A7B23E: ENGINE_load_private_key (in /usr/lib/libcrypto.so.1.1)
==1926254==    by 0x17D5A7: ??? (in /usr/bin/openssl)
==1926254==    by 0x153BB8: ??? (in /usr/bin/openssl)
==1926254==    by 0x14CA36: ??? (in /usr/bin/openssl)
==1926254==    by 0x1363F5: ??? (in /usr/bin/openssl)
==1926254==    by 0x4C43022: (below main) (in /usr/lib/libc-2.31.so)

My understanding of this log is that both OpenSSL and SoftHSMv2 install their own atexit handlers:

  • The SoftHSMv2 atexit handler is invoked first, which releases SoftHSMv2 memory blocks.
  • The OpenSSL atexit handler is invoked next, which invokes libp11 engine_finish(), which invokes SoftHSMv2 C_Finalize, which tries to release the same memory blocks again.

I don’t think it is possible for OpenSSL (or libp11) to know that the PKCS#11 module has already performed its cleanup. I think either OpenSSL should refrain from calling engine finish handlers from within its atexit handler, or SoftHSMv2 should ignore C_Finalize after its destructors have already been invoked, or both.

Just removing them would probably turn out not to be enough. In the constructor, ENGINE_load_rdrand() is guarded. In the destructor, ENGINE_free(rdrand_engine) should be guarded the same way.

The OpenSSL ENGINE API was never designed to be reentrant. The only solution I can think of is disabling the use of engines in SoftHSM2.

Last time I invested it, SoftHSM2 performed OpenSSL cleanup at C_Finalize. This is quite insane, unless they assume their PKCS#11 module will never be used with applications that also directly invoke OpenSSL functions.

I dug deeper, and changed #11193 accordingly. Please try the new attempt.

but in general OpenSSL master should fix it. IMHO.

Maybe… I’ll do a deeper dive to see if that’s even possible in a sane way. I did just notice that you call RSA_set_method(), which is a really antiquated way to hook in engine methods into an RSA, but yeah, maybe there’s a way to get a hint…

This is becoming a libp11 discussion

Respectfully disagree: libp11 passes all the tests quite nicely with OpenSSL-1.1.1. And 3.0 wasn’t supposed to be a disruptive change, as far as I know (unlike 1.0 -> 1.1).

So, it looks like something is broken in OpenSSL-3.0. I’d be happy with a workaround (i.e., add something to libp11 to side-step this problem), but in general OpenSSL master should fix it. IMHO.

I got myself to the offending EVP_SignFinal() call, and here’s what I’m noticing:

230		if (EVP_SignFinal(md_ctx, signature, &siglen, privkey) <= 0) {
(gdb) p *privkey 
$10 = {type = 6, save_type = 6, ameth = 0x7ffff7f8bf40 <rsa_asn1_meths>, 
  engine = 0x0, pmeth_engine = 0x0, pkey = {ptr = 0x5555555a6a30, 
    rsa = 0x5555555a6a30, dsa = 0x5555555a6a30, dh = 0x5555555a6a30, 
    ec = 0x5555555a6a30, ecx = 0x5555555a6a30}, references = 2, 
  lock = 0x5555555aa590, attributes = 0x0, save_parameters = 1, pkeys = {{
      keymgmt = 0x0, keydata = 0x0}, {keymgmt = 0x0, keydata = 0x0}, {
      keymgmt = 0x0, keydata = 0x0}, {keymgmt = 0x0, keydata = 0x0}, {
      keymgmt = 0x0, keydata = 0x0}, {keymgmt = 0x0, keydata = 0x0}, {
      keymgmt = 0x0, keydata = 0x0}, {keymgmt = 0x0, keydata = 0x0}, {
      keymgmt = 0x0, keydata = 0x0}, {keymgmt = 0x0, keydata = 0x0}}, 
  dirty_cnt_copy = 0, cache = {bits = 0, security_bits = 0, size = 0}}
(gdb) p *privkey->pkey.rsa
$11 = {libctx = 0x0, pad = 0, version = 0, meth = 0x5555555a1d70, 
  engine = 0x0, n = 0x5555555a8ea0, e = 0x5555555a8360, d = 0x0, p = 0x0, 
  q = 0x0, dmp1 = 0x0, dmq1 = 0x0, iqmp = 0x0, prime_infos = 0x0, pss = 0x0, 
  ex_data = {ctx = 0x0, sk = 0x5555555a84c0}, references = 1, flags = 38, 
  _method_mod_n = 0x0, _method_mod_p = 0x0, _method_mod_q = 0x0, 
  bignum_data = 0x0, blinding = 0x0, mt_blinding = 0x0, lock = 0x5555555aa210, 
  dirty_cnt = 1}

If you look closely, you’ll notice that all engine pointers available are NULL, so there’s no way to know one applies. This goes all the way back to pkcs11_get_evp_key_rsa(), which does nothing to set any engine pointer. A call like this would be appropriate:

    EVP_PKEY_set1_engine(pk, RSA_get0_engine(rsa));

Now, when looking in pkcs11_get_rsa(), I could also see why the RSA structure isn’t assigned the engine either. It’s preferable to use RSA_new_method() instead of RSA_new() in this case.

That being said, EVP_PKEY_set1_engine() sets pk->pmeth_engine, so I should update #11193 to look at that pointer too.

Ok, I get failures in rsa-testfork.softhsm, pkcs11-uri-without-token.softhsm and search-all-matching-tokens.softhsm. The log from the first of them matches what you showed, so now I know where to start digging

Why is install prefixed with _? That path doesn’t look like anything on my system.

Because it’s my private install, so that’s what I chose to call that directory.

It looks like OpenSSL 3.0 ignores the engine and tries to sign the key with rsa_ossl_private_encrypt(),