openssl: PKEY-method override no longer working with OpenSSL 3.0

I have an application that uses a PKEY-method override to perform HSM-based secure key signing. In OpenSSL 1.1.1. this works well, but in OpenSSL 3.0 it does not work anymore.

I am aware that all the PKEY method functions are deprecated in OpenSSL 3.0, but I guess they ‘should’ still work the same as before…

However, it might be that my usage is somewhat unusual, and thus not covered by the OpenSSL 3.0 compatibility code. So it might be that you immediately tell me that this is no supposed to work anymore with OpenSSL 3.0. That would be totally OK for me, I do have a provider based solution for that use case that works with OpenSSL 3.0. However, maybe this situation is unintentionally not covered by the compatibility code and is worth to fix.

Here is what I do:

I setup my own PKEY method as follows. Note that this is not within an engine, but just within the application code.

pkey_meth = EVP_PKEY_meth_new(EVP_PKEY_id(pkey), 0);
EVP_PKEY_meth_set_init(pkey_meth, my_pkey_meth_init);
EVP_PKEY_meth_set_cleanup(pkey_meth, my_pkey_meth_cleanup);
EVP_PKEY_meth_set_copy(pkey_meth, my_pkey_meth_copy);
EVP_PKEY_meth_set_ctrl(pkey_meth, my_pkey_meth_ctrl, NULL);
EVP_PKEY_meth_set_sign(pkey_meth, NULL, my_pkey_meth_sign);
EVP_PKEY_meth_add0(pkey_meth);

Note that my_pkey_meth_ctrl() supports some own private control commands, i.e. EVP_PKEY_CTRL_MY_COMMAND.

Also note that the PKEY used in this application is a legacy PKEY, that has been produced via EVP_PKEY_assign_EC_KEY() (public key only, the private key is known to my PKEY method by other ways - i.e. is set via EVP_PKEY_CTX_ctrl() with a private command later on).

I then setup a digest sign context as follows:

EVP_PKEY_CTX *pctx = NULL;
EVP_MD_CTX *ctx = EVP_MD_CTX_new();
EVP_DigestSignInit(ctx, &pctx, md, NULL, pkey);
EVP_PKEY_CTX_ctrl(pkey_ctx, -1, -1, EVP_PKEY_CTRL_MY_COMMAND, x, y);

The very last function (EVP_PKEY_CTX_ctrl()) fails.

I debugged this, and it seems that it is seeding the command to the (default) provider, and thus through the ctrl to parms translation, which of course fails to translate that command since ist my private one. So evp_pkey_ctx_ctrl_to_param() fails with EVP_R_COMMAND_NOT_SUPPORTED.

It goes that way, because in evp_pkey_ctx_ctrl_int() it calls evp_pkey_ctx_state() which returns EVP_PKEY_STATE_PROVIDER for that PKEY context.

I think that the error actually happend way before that. EVP_DigestSignInit() internally creates the PKEY context. do_sigver_init() creates a new PKEY context using EVP_PKEY_CTX_new_from_pkey() which calls int_ctx_new(). At the end of int_ctx_new() the created PKEY context looks as follows:

*ret = {operation = 0, libctx = 0x0, propquery = 0x0, keytype = 0x3fffddb1d68 "id-ecPublicKey", keymgmt = 0x1113690, op = {
    keymgmt = {genctx = 0x0}, kex = {exchange = 0x0, algctx = 0x0}, sig = {signature = 0x0, algctx = 0x0}, ciph = {cipher = 0x0,
      algctx = 0x0}, encap = {kem = 0x0, algctx = 0x0}}, cached_parameters = {dist_id_name = 0x0, dist_id = 0x0, dist_id_len = 0,
    dist_id_set = 0}, app_data = 0x0, pkey_gencb = 0x0, keygen_info = 0x0, keygen_info_count = 0, legacy_keytype = 408,
  pmeth = 0x10b9070, engine = 0x0, pkey = 0x10bba80, peerkey = 0x0, data = 0x10b9350, flag_call_digest_custom = 0, rsa_pubexp = 0x0}

Note that the pmeth field is set, and it is my own PKEY method, as expected. It got it via pmeth = evp_pkey_meth_find_added_by_application(id);

Later in do_sigver_init() it uses the created PKEY context and checks if it is a legacy context:

     if (evp_pkey_ctx_is_legacy(locpctx))
        goto legacy;

And here is the problem, evp_pkey_ctx_is_legacy() returns false, so it is treated as provider context, not legacy.

evp_pkey_ctx_is_legacy is a define as follows:

#define evp_pkey_ctx_is_legacy(ctx)                             \
    ((ctx)->engine != NULL || (ctx)->keytype == NULL)

I would assume that it should also check (ctx)->pmeth != NULL, because a context with an application PKEY method should also be treated as a legacy context.

The legacy code within do_sigver_init() would then use the PKEY method to perform the sign operation.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 26 (26 by maintainers)

Commits related to this issue

EVP: Move setting |keytype| in int_ctx_new() in the legacy section The code that set |keytype| in the legacy section of int_ctx_new() did so a little too early, before other variables that should be ... — committed to levitte/openssl by levitte 3 years ago
Fix custom EVP_PKEY_METHOD implementations where no engine is present It is possible to have a custom EVP_PKEY_METHOD implementation without having an engine. In those cases we were failing to use th... — committed to mattcaswell/openssl by mattcaswell 3 years ago

Most upvoted comments

To ease investigations into this issue, I wrote a fully standalone reproducer:

#include <openssl/evp.h>
#include <openssl/pem.h>

static EVP_PKEY_METHOD *custom_rsa;
static const EVP_PKEY_METHOD *rsa_orig;

#define EVP_PKEY_CTRL_MY_COMMAND 9999


static char data[] =
"-----BEGIN PRIVATE KEY-----\n"
"MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDVXWBq3/xh7kiq\n"
"jBFIQ6VttlJdqphJsWGSNbH8OgQlDG15/7TVyelcHDvgq7O4faPebb3g3ddavxRH\n"
"EUJepoLQYcF/3RNG5gmFBw7y1PwaZNIKrSCrIGuW8K3MxBlTVdwBHaSz74q0SVNd\n"
"igUc8dzhRL/F1+J3GVdclwt17ohDcQ/KbMG0slCnd0ZsWA8Rv/F2JFquOUK3UWcp\n"
"4dBVMG8X5JHqrfgowkNvomSp+52YkmJIPusNT4JKiv8/cu6Wta6hwZi6732QdW3/\n"
"WlKeq/XAftCHQ9uFBwcPfTh6/dHT7mUd0+o5aoc37krT4A1u9XCswr3xbvOSlV6p\n"
"8KFllZONAgMBAAECggEADLTt7A+A2Vg2jamf0dztejY0e42QWjstI2b9PZc67fXq\n"
"gyx+WYkX07t+uWegYWliG/oPJ9guXiIpE/5sJHToL37S5kmFP2CtynVcJ4wVo4DD\n"
"nY0n9+kLX0bgIuS+2V6wpoRcbbbjXM9NHrH8kfe5ftT4UtEDlLI2qLX6IcDd7p4u\n"
"OYjILChR8GSGTw96yIy2Ws/1Uq9PMw64JoT4RcK5QqnkcPMDFRH1SeLOL+zXP2c4\n"
"nEl9yOy3HauZKxwl/Ry/XK1s3DdjopIAU29ut+hAuMiTb06kzZnumL9NoplKoZtU\n"
"otw/gVcCKhT+Ep+p6i8InLF0XEME8A0qUR0niWebgQKBgQD6vkxR49B8ZZQrzjw4\n"
"XKs1lI9cP7cgPiuWlDHMNjYou3WbOaGrMeScvbB1Ldh9A8pjAhxlw8AaV/xs4qcA\n"
"trmVmSISVMVyc1wSGlJXWi2nUzTNs9OE3vj22SyStihf8UUZtWwX2b5Y4JrYhA/V\n"
"+ThGGqHR03oLNLShNLtJc2c7YQKBgQDZ1nkibEyrepexw/fnwkw61IJKq9wRIh1G\n"
"PREakhbe9wU5ie0knuf9razt7awzQiwFmlixmWqsM7UEtLuXNnNPciwdrKhhbvrd\n"
"vD/rkbIEHEPllIhFlDtOzn3hRBWTzWmXFjpou/2LvHTSbVis4IYVZymTp2jb1ZLs\n"
"7VbiG9JTrQKBgQDc6n75g1szzpdehQT/r33U5j/syeJBUSU8NPMu9fB/sLHsgjlT\n"
"SNEf2+y1QSBE/Or6kmiMrIv7advn30W+Vj9qc5HWTsPrk4HiHTjA553jl2alebN5\n"
"lK4LZspjtIQcC8mS3goPdXPEgJdM/gWpwzr2YQ6DfOxBJT2j7n64NyoT4QKBgH7/\n"
"yx+GhCx1DHtXBPDZFhg2TL+78lEK0oZgk9gp06up2CHzh44SFq6O0oLkTcCUk5Ww\n"
"poTkLIy4mJBlzfgahp+KsK2cO46SZS9g0ONFzcMXt33hWpE2Gl2XhUwPpYTF/QlY\n"
"rDTjZK5S8Mi9dzVSsNlJi7PJphiEK2R1+nFYRwcBAoGBANWoIG85jpXAOnq/Kcgx\n"
"Rl3YivR0Ke6r1tFlP58rT7X3EkiboXyQl5vLIFCAwUte6RGrLl1dy3Qyh80B9ySL\n"
"Jx6vj42CK7vgv6A96TuVYhnXTnEI6ZvwAQ2VGaw4BizhjALs/kdSE/og9aSCs3ws\n"
"KQypwAFz0tbHxaNag/bSAN0J\n"
"-----END PRIVATE KEY-----\n";


EVP_PKEY *loadkey(void)
{
    BIO* bio = BIO_new_mem_buf(data, -1);
    EVP_PKEY *pkey = PEM_read_bio_PrivateKey(bio, NULL, NULL, NULL);

    BIO_free(bio);
    return pkey;
}

static int custom_rsa_init(EVP_PKEY_CTX *ctx)
{
    static int (*pinit)(EVP_PKEY_CTX *ctx);

    if (pinit == NULL)
        EVP_PKEY_meth_get_init(rsa_orig, &pinit);
    return pinit(ctx);
}

static void custom_rsa_cleanup(EVP_PKEY_CTX *ctx)
{
    static void (*pcleanup)(EVP_PKEY_CTX *ctx);

    if (pcleanup == NULL)
        EVP_PKEY_meth_get_cleanup(rsa_orig, &pcleanup);
    pcleanup(ctx);
}

static int custom_rsa_decrypt_init(EVP_PKEY_CTX *ctx)
{
    static int (*pdecrypt_init)(EVP_PKEY_CTX *ctx);

    if (pdecrypt_init == NULL)
        EVP_PKEY_meth_get_decrypt(rsa_orig, &pdecrypt_init, NULL);
    return pdecrypt_init(ctx);
}

static int custom_rsa_sign(EVP_PKEY_CTX *ctx, unsigned char *out,
                              size_t *outlen, const unsigned char *in,
                              size_t inlen)
{
    static int (*psign)(EVP_PKEY_CTX *ctx, unsigned char *sig, size_t *siglen,
                        const unsigned char *tbs, size_t tbslen);

    if (psign == NULL)
        EVP_PKEY_meth_get_sign(rsa_orig, NULL, &psign);
    return psign(ctx, out, outlen, in, inlen);
}

static int custom_rsa_ctrl(EVP_PKEY_CTX *ctx, int type, int p1, void *p2)
{
    static int (*pctrl)(EVP_PKEY_CTX *ctx, int type, int p1, void *p2);

    if (pctrl == NULL)
        EVP_PKEY_meth_get_ctrl(rsa_orig, &pctrl, NULL);

    if (type == EVP_PKEY_CTRL_MY_COMMAND) {
        printf("Success!\n");
        return 1;
    }

    return pctrl(ctx, type, p1, p2);
}

static int custom_rsa_ctrl_str(EVP_PKEY_CTX *ctx, const char *type,
                               const char *value)
{
    static int (*pctrl_str)(EVP_PKEY_CTX *ctx, const char *type,
                            const char *value);

    if (pctrl_str == NULL)
        EVP_PKEY_meth_get_ctrl(rsa_orig, NULL, &pctrl_str);
    return pctrl_str(ctx, type, value);
}

int main(void)
{
    EVP_PKEY_CTX *pctx = NULL;
    EVP_MD_CTX *ctx = EVP_MD_CTX_new();
    EVP_PKEY *pkey = NULL;

    if ((rsa_orig = EVP_PKEY_meth_find(EVP_PKEY_RSA)) == NULL
        || (custom_rsa = EVP_PKEY_meth_new(EVP_PKEY_RSA, 0)) == NULL
        || ctx == NULL)
        return 0;
    EVP_PKEY_meth_set_init(custom_rsa, custom_rsa_init);
    EVP_PKEY_meth_set_cleanup(custom_rsa, custom_rsa_cleanup);
    EVP_PKEY_meth_set_sign(custom_rsa, NULL, custom_rsa_sign);
    EVP_PKEY_meth_set_ctrl(custom_rsa, custom_rsa_ctrl,
                           custom_rsa_ctrl_str);
    EVP_PKEY_meth_add0(custom_rsa);

    pkey = loadkey();
    EVP_DigestSignInit(ctx, &pctx, EVP_sha256(), NULL, pkey);
    EVP_PKEY_CTX_ctrl(pctx, -1, -1, EVP_PKEY_CTRL_MY_COMMAND, 0, NULL);  
}

When compiled against 1.1.1 this application prints “Success!”, but fails to do so against master.

mattcaswell on Jul 19, 2021

My alternative fix for this issue is in #16118. Please give it a try. This fixes both the legacy key case and the provided key case.

I have reverted Richards fix and applied yours (only the one commit), and tested my application again. Your fix also fixes the problem.

Please note that I have tested only the legacy-PKEY case, since my application always uses legacy PKEYs. So I can’t tell if it also works with provided PKEYs. However, your original reproducer should cover that case.

I was first a bit confused if your fix wouldn’t also need the change from Richards fix about only setting keytype for provided keys, but I then realized that you also changed evp_pkey_ctx_is_legacy() to check for keymgmt only. So you fix looks good from my point of view.

ifranzki on Jul 21, 2021

With a fix in my PKEY method sign function for the sig = NULL case it now works fine. So I can confirm that your fix in https://github.com/openssl/openssl/pull/16111 fixes the problem.

ifranzki on Jul 20, 2021

That should not be necessary. We just need to know that when we init the pctx, that it should be inited as legacy and therefore the pmethod will handle the ctrls.

The consequence is that we will have to do pkey downgrading under these conditions, since pmeths rightfully assume legacy keys. That’s something we have explicitly said we shouldn’t have to do.

It’s a bit late to change that perspective now!

levitte on Jul 19, 2021

Although the original producer should also work in master but doesn’t - so both are valid scenarios.

Why? With a provider key, ctrls of that sort are invalid. Remember, we do not automagically downgrade pkeys, and the ctrl<->params translation only handles ctrl cmds and param keys that libcrypto knows. Translating completely random cmds like EVP_PKEY_CTRL_MY_COMMAND are too big an ask.

levitte on Jul 19, 2021