openssl: Severe efficiency degradation of credential loading in comparison to 1.1.1

This is a follow-up on #14837, which got closed due to a partial solution regarding the heavy use of heap memory allocation.

I’ve just been using the tool provided by @richsalz in #16540, ~compiled like this:~ cd test/ ~gcc -g -I../include timing.c ../libcrypto.a -ldl -lpthread -o timing~

Update: PR for tool is now in #18821, called timing_load_creds.

When run on current 3.x master, I typically get figures like these:

./timing_load_creds -wc certs/timing-cert.pem
user      0 sec 57846 microsec
sys       0 sec 69877 microsec

and

./timing_load_creds -wp certs/timing-key.pem 
user      0 sec 95863 microsec
sys       0 sec 115708 microsec

Whereas with the latest 1.1.1 branch, I typically get for instance:

./timing_load_creds -wc certs/timing-cert.pem
user      0 sec 17389 microsec
sys       0 sec 17389 microsec

and

./timing_load_creds -wp certs/timing-key.pem
user      0 sec 14719 microsec
sys       0 sec 10223 microsec

BTW, on my Debian system times reported vary a lot between consecutive runs - I’ve copied some ‘average’ outcomes.

So with 1.1.1, cert and key loading appear reasonably efficient, while since 3.0 loading a simple cert (including a public key) degraded by a factor around 3.5, and loading a private key degraded by a factor of more than 6.

@t8m wrote in https://github.com/openssl/openssl/issues/14837#issuecomment-862506046:

The obvious reason being the change from legacy to provider based parsing of the public keys in certificates.

Apparently the inefficiencies are due to generalizing the loading mechanism using the provider mechanism and maybe the STORE API, which appear pretty involved when having a look at the code and stepping through them while debugging.

Is this considered bearable? Hopefully some further bottleneck(s) can be identified and removed.

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 5
  • Comments: 17 (9 by maintainers)

Most upvoted comments

I’m using Python and this problem has severe performance consequences for multi-threaded Python Code which heavily uses SSL-Requests. We’re talking about double the time and up to 15 times the CPU power. See details here: https://github.com/python/cpython/issues/95031#issuecomment-1748269062

So while I was also hit by this in Python 3.12, I would like to test the issue within OpenSSL alone. Also to check if there is any improvement in 3.2.x or 3.3.x.

TL;DR: I didn’t see much improvement on the speed of SSL_CTX_load_verify_locations in 3.3.1. It is still extremely slow especially on Windows. Performance wise, 3.0.14 ~= 3.3.1 and is 15x slower than 1.1.1.


Details:

I use the following test code in C:

#include <openssl/ssl.h>
#include <openssl/err.h>
#include <stdio.h>

int main(int argc, char **argv) {
    if (argc != 2) {
        fprintf(stderr, "Usage: %s <CAfile>\n", argv[0]);
        return 1;
    }
    const char *cafile = argv[1];
    SSL_CTX *ctx;
    OPENSSL_init_ssl(0, NULL);
    ctx = SSL_CTX_new(TLS_method());
    if (!ctx) {
        fprintf(stderr, "Error creating SSL context\n");
        ERR_print_errors_fp(stderr);
        return 1;
    }
    if (!SSL_CTX_load_verify_locations(ctx, cafile, NULL)) {
        fprintf(stderr, "Error loading CA file\n");
        ERR_print_errors_fp(stderr);
        SSL_CTX_free(ctx);
        return 1;
    }
    printf("CA file loaded successfully\n");
    SSL_CTX_free(ctx);
    return 0;
}

I compiled it with 1.1.1w, 3.0.14 LTS, and 3.3.1 x64 respectively, all downloaded from here.

set PATH=C:\Users\ikena\mingw64\bin\;%PATH%
gcc -o openssl-1.1.1w\bin\loadca loadca.c "-Iopenssl-1.1.1w\include" "-Lopenssl-1.1.1w\lib" -lssl -lcrypto
gcc -o openssl-3.0.14\bin\loadca loadca.c "-Iopenssl-3.0.14\include" "-Lopenssl-3.0.14\lib" -lssl -lcrypto
gcc -o openssl-3.3.1\bin\loadca loadca.c "-Iopenssl-3.3.1\include" "-Lopenssl-3.3.1\lib" -lssl -lcrypto

I then test their speed against 288 KB cacert.pem from python/certifi (can download here: https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem) 50 times and calculate the average speed:

import subprocess
import time

def measure_command(cmd, iterations):
    start_time = time.time()
    start_time_cpu = time.process_time()
    for _ in range(iterations):
        subprocess.run(cmd, shell=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    end_time = time.time()
    end_time_cpu = time.process_time()
    ave_time = (end_time - start_time) / iterations
    ave_time_cpu = (end_time_cpu - start_time_cpu) / iterations
    print(f"Command: {cmd} Average time over {iterations} iterations: {ave_time:.6f}s, {ave_time_cpu:.6f}s CPU time")

ITER = 50
measure_command('openssl-1.1.1w\\bin\\loadca.exe cacert.pem', ITER)
measure_command('openssl-3.0.14\\bin\\loadca.exe cacert.pem', ITER)
measure_command('openssl-3.3.1\\bin\\loadca.exe cacert.pem ', ITER)

The result:

Command: openssl-1.1.1w\bin\loadca.exe cacert.pem Average time over 50 iterations: 0.036579s, 0.001563s CPU time
Command: openssl-3.0.14\bin\loadca.exe cacert.pem Average time over 50 iterations: 0.488777s, 0.000313s CPU time
Command: openssl-3.3.1\bin\loadca.exe cacert.pem  Average time over 50 iterations: 0.452512s, 0.000625s CPU time

Before 3.0 it was already possible to load keys of all known types, even without specifying which key type(s) to expect, in reasonable time. Even if the extra flexibility of handling further (provider-defined) key types is the main reason, with credential loading now taking 3-6 times longer, something must have gone pretty wrong here.

At least when loading cert/key material having any well-known key type, the generalization to potentially handle further key types should produce just negligible overhead. And even for new key types there should be rather efficient ways of determining which specific (sub-)parser to use.

For python, as mentioned here https://github.com/python/cpython/issues/95031#issuecomment-1748269062 , you could do:

  1. Do not verify certificates: verfiy = False
  2. Use Python 3.11
  3. Use a context manager, ie, cache the verified certificate

Please also note that on Windows (i.e., comment https://github.com/openssl/openssl/issues/18814#issuecomment-2194020984) the biggest part of the performance degradation comes from the use of unbuffered IO when reading PEM files which this PR resolves: https://github.com/openssl/openssl/pull/25716

This is still a major problem years later, especially now that 1.1.1 is all but dead. The performance degradation is significant. When will resources be applied to find a solution to this issue?

This comment still stands to reason and is well liked:

https://github.com/openssl/openssl/issues/18814#issuecomment-1187567135

See also #16791 and #17627