openssl: Severe efficiency degradation of credential loading in comparison to 1.1.1
This is a follow-up on #14837, which got closed due to a partial solution regarding the heavy use of heap memory allocation.
I’ve just been using the tool provided by @richsalz in #16540, ~compiled like this:~
cd test/
~gcc -g -I../include timing.c ../libcrypto.a -ldl -lpthread -o timing~
Update: PR for tool is now in #18821, called timing_load_creds.
When run on current 3.x master, I typically get figures like these:
./timing_load_creds -wc certs/timing-cert.pem
user 0 sec 57846 microsec
sys 0 sec 69877 microsec
and
./timing_load_creds -wp certs/timing-key.pem
user 0 sec 95863 microsec
sys 0 sec 115708 microsec
Whereas with the latest 1.1.1 branch, I typically get for instance:
./timing_load_creds -wc certs/timing-cert.pem
user 0 sec 17389 microsec
sys 0 sec 17389 microsec
and
./timing_load_creds -wp certs/timing-key.pem
user 0 sec 14719 microsec
sys 0 sec 10223 microsec
BTW, on my Debian system times reported vary a lot between consecutive runs - I’ve copied some ‘average’ outcomes.
So with 1.1.1, cert and key loading appear reasonably efficient, while since 3.0 loading a simple cert (including a public key) degraded by a factor around 3.5, and loading a private key degraded by a factor of more than 6.
@t8m wrote in https://github.com/openssl/openssl/issues/14837#issuecomment-862506046:
The obvious reason being the change from legacy to provider based parsing of the public keys in certificates.
Apparently the inefficiencies are due to generalizing the loading mechanism using the provider mechanism and maybe the STORE API, which appear pretty involved when having a look at the code and stepping through them while debugging.
Is this considered bearable? Hopefully some further bottleneck(s) can be identified and removed.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 5
- Comments: 17 (9 by maintainers)
I’m using Python and this problem has severe performance consequences for multi-threaded Python Code which heavily uses SSL-Requests. We’re talking about double the time and up to 15 times the CPU power. See details here: https://github.com/python/cpython/issues/95031#issuecomment-1748269062
So while I was also hit by this in Python 3.12, I would like to test the issue within OpenSSL alone. Also to check if there is any improvement in 3.2.x or 3.3.x.
TL;DR: I didn’t see much improvement on the speed of
SSL_CTX_load_verify_locationsin 3.3.1. It is still extremely slow especially on Windows. Performance wise, 3.0.14 ~= 3.3.1 and is 15x slower than 1.1.1.Details:
I use the following test code in C:
I compiled it with 1.1.1w, 3.0.14 LTS, and 3.3.1 x64 respectively, all downloaded from here.
I then test their speed against 288 KB cacert.pem from python/certifi (can download here: https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem) 50 times and calculate the average speed:
The result:
Before 3.0 it was already possible to load keys of all known types, even without specifying which key type(s) to expect, in reasonable time. Even if the extra flexibility of handling further (provider-defined) key types is the main reason, with credential loading now taking 3-6 times longer, something must have gone pretty wrong here.
At least when loading cert/key material having any well-known key type, the generalization to potentially handle further key types should produce just negligible overhead. And even for new key types there should be rather efficient ways of determining which specific (sub-)parser to use.
For python, as mentioned here https://github.com/python/cpython/issues/95031#issuecomment-1748269062 , you could do:
Please also note that on Windows (i.e., comment https://github.com/openssl/openssl/issues/18814#issuecomment-2194020984) the biggest part of the performance degradation comes from the use of unbuffered IO when reading PEM files which this PR resolves: https://github.com/openssl/openssl/pull/25716
This is still a major problem years later, especially now that 1.1.1 is all but dead. The performance degradation is significant. When will resources be applied to find a solution to this issue?
This comment still stands to reason and is well liked:
https://github.com/openssl/openssl/issues/18814#issuecomment-1187567135
See also #16791 and #17627