openssl: Current master breaks libp11 RSA signature
MacOS Catalina 10.15.3, Xcode-11.3.1, installed OpenSSL-1.1.1d (reference) and (in a separate dir tree) OpenSSL-3.0 master.
Using libp11 to perform RSA signature using hardware token accessible via PKCS#11. Fails with both a real hardware token, and SoftHSMv2.
With OpenSSL-1.1.1d (succeeds):
$ pkcs11-rsa-pss-sign-demo2
Generating ephemeral file /tmp/derive.89314.text to test RSA-PSS signature...
openssl rand -hex -out /tmp/derive.89314.text 5120
Signing file /tmp/derive.89314.text...
openssl dgst -engine pkcs11 -keyform engine -sign "pkcs11:manufacturer=piv_II;object=SIGN%20key;type=private" -sha384 -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-1 -out /tmp/derive.89314.text.sig /tmp/derive.89314.text
engine "pkcs11" set.
Enter PKCS#11 token PIN for xxxxxxxxx:
Signature is stored in /tmp/derive.89314.text.sig
Verifying signature:
openssl dgst -engine pkcs11 -keyform engine -sha384 -verify "pkcs11:manufacturer=piv_II;object=SIGN%20pubkey;type=public" -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-1 -signature /tmp/derive.89314.text.sig /tmp/derive.89314.text
engine "pkcs11" set.
Verified OK
$
With OpenSSL-3.0 (current master, fails):
$ pkcs11-rsa-pss-sign-demo3
Generating ephemeral file /tmp/derive.89324.text to test RSA-PSS signature...
openssl3 rand -hex -out /tmp/derive.89324.text 5120
Signing file /tmp/derive.89324.text...
openssl3 dgst -engine pkcs11 -keyform engine -sign "pkcs11:manufacturer=piv_II;object=SIGN%20key;type=private" -sha384 -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:-1 -out /tmp/derive.89324.text.sig /tmp/derive.89324.text
engine "pkcs11" set.
Enter PKCS#11 token PIN for xxxxxxxxx:
Error Signing Data
C0:2D:B6:09:01:00:00:00:error:rsa routines:rsa_ossl_private_encrypt:missing private key:crypto/rsa/rsa_ossl.c:333:
C0:2D:B6:09:01:00:00:00:error:Provider routines:rsa_sign:RSA lib:providers/implementations/signature/rsa.c:410:
Also, see this libp11 issue for details of a failure trying to sign using SoftHSMv2: https://github.com/OpenSC/libp11/issues/331
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 34 (33 by maintainers)
Commits related to this issue
- EVP: Check if there's an engine in signature init. All of EVP_DigestSignInit_ex(), EVP_DigestSignInit(), EVP_DigestVerifyInit_ex(), EVP_DigestVerifyInit(), EVP_PKEY_sign_init(), EVP_PKEY_verify_init(... — committed to levitte/openssl by levitte 4 years ago
- EVP: Check that key methods aren't foreign when exporting The EVP_PKEY_ASN1_METHOD function export_to() must check that the key we're trying to export has a known libcrypto method, i.e. is a built in... — committed to levitte/openssl by levitte 4 years ago
- EVP: Check that key methods aren't foreign when exporting The EVP_PKEY_ASN1_METHOD function export_to() must check that the key we're trying to export has a known libcrypto method, i.e. is a built in... — committed to levitte/openssl by levitte 4 years ago
- EVP: Check that key methods aren't foreign when exporting The EVP_PKEY_ASN1_METHOD function export_to() must check that the key we're trying to export has a known libcrypto method, i.e. is a built in... — committed to levitte/openssl by levitte 4 years ago
Thank you for trying it out! It’s really nice to get this kind of bug report so early on (before we even have an alpha release out!).
Thanks, I suspected that
I stumbled over the same problem (the OpenSSL ENGINE being not reentrant).
I worked around it by using the
p11-kit-proxy.soas the first pkcs#11 module - with aremote: |p11-kit-remote /path/to/original-pkcs11.so. Asoriginal-pkcs11.soruns in a different process, it can internally use another OpenSSL ENGINE instance without causing problems.@levitte: You’re right. It looks more complex indeed.
There are some double free issues, and corrupted heap results in crash locations other than the actual root cause. I collected a valgrind log: valgrind.txt
An example double free:
My understanding of this log is that both OpenSSL and SoftHSMv2 install their own atexit handlers:
engine_finish(), which invokes SoftHSMv2C_Finalize, which tries to release the same memory blocks again.I don’t think it is possible for OpenSSL (or libp11) to know that the PKCS#11 module has already performed its cleanup. I think either OpenSSL should refrain from calling engine finish handlers from within its atexit handler, or SoftHSMv2 should ignore
C_Finalizeafter its destructors have already been invoked, or both.Just removing them would probably turn out not to be enough. In the constructor,
ENGINE_load_rdrand()is guarded. In the destructor,ENGINE_free(rdrand_engine)should be guarded the same way.The OpenSSL ENGINE API was never designed to be reentrant. The only solution I can think of is disabling the use of engines in SoftHSM2.
Last time I invested it, SoftHSM2 performed OpenSSL cleanup at C_Finalize. This is quite insane, unless they assume their PKCS#11 module will never be used with applications that also directly invoke OpenSSL functions.
I dug deeper, and changed #11193 accordingly. Please try the new attempt.
Maybe… I’ll do a deeper dive to see if that’s even possible in a sane way. I did just notice that you call
RSA_set_method(), which is a really antiquated way to hook in engine methods into an RSA, but yeah, maybe there’s a way to get a hint…Respectfully disagree:
libp11passes all the tests quite nicely with OpenSSL-1.1.1. And 3.0 wasn’t supposed to be a disruptive change, as far as I know (unlike 1.0 -> 1.1).So, it looks like something is broken in OpenSSL-3.0. I’d be happy with a workaround (i.e., add something to
libp11to side-step this problem), but in general OpenSSL master should fix it. IMHO.I got myself to the offending
EVP_SignFinal()call, and here’s what I’m noticing:If you look closely, you’ll notice that all engine pointers available are NULL, so there’s no way to know one applies. This goes all the way back to
pkcs11_get_evp_key_rsa(), which does nothing to set any engine pointer. A call like this would be appropriate:Now, when looking in
pkcs11_get_rsa(), I could also see why the RSA structure isn’t assigned the engine either. It’s preferable to useRSA_new_method()instead ofRSA_new()in this case.That being said,
EVP_PKEY_set1_engine()setspk->pmeth_engine, so I should update #11193 to look at that pointer too.Ok, I get failures in rsa-testfork.softhsm, pkcs11-uri-without-token.softhsm and search-all-matching-tokens.softhsm. The log from the first of them matches what you showed, so now I know where to start digging
Because it’s my private install, so that’s what I chose to call that directory.
It looks like OpenSSL 3.0 ignores the engine and tries to sign the key with rsa_ossl_private_encrypt(),