openssl: PEM/DER decoding of PKCS8 RSA private keys are 80 times slower in 3.0
We have two sample programs to parse PKCS8 RSA private keys. First to parse one in PEM format:
#include <openssl/evp.h>
#include <openssl/x509.h>
#include <openssl/pem.h>
static char data[] = "-----BEGIN PRIVATE KEY-----\nMIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQCqD9qzDx5C7niP8lvd8uXwQNHL\nJjakRyEzfbV5FJtJujgOXjF5POm7WICAgtdrct+g5v/SfYvgNkbs06E7th1zFY7cmEimJOIVuDsG\nlMUySlYioxW9QvTERgq3e9NONKYPEabGwMzh6ZYjHYSaKZUDAMlV58XhXLIf4+AAyVoSF7jihbs5\nGb+noksQ4alKxiJDKr/9ijfC+HMj4VQEc41Mib78+ImXMRzPuCj2IPSorY4svMsWLgjm8+4FV/Ps\nSkd2zOeSffeUm6Oa4F0rWnmOyRk6WBKIv6OW2mj3OTh6f7VOBy3tn9xUkY+wFcY6OrFfwgbH2kJ+\n5aibj00mrL4FXLLVKbYp5YAZFvEvujKuAX2M7ZocS95XHwfbQK3ol6wCNK75NLOTroVQtlH/5gcN\niJtxTiW81hwY17IhPwVBhq1ynb+9YEzNoUuqfsEvhtHeRLdU/YYOiXXQVESCYXgQ5ApFRExfVu1t\n8a/MibNIPM+nJIcJh80BNbl2fL9ShMJXuBgRz/Yyp//1NNoC8/MkO/vAaZBCrStOZStZLRyi0Aye\nvvw0a7IIS82h4R6cjte1Vfnd+eX4MOd98a/AHJ/4ecp9lIySQyVf5NrcGTz1A5LtyZdi4teGwBl+\nX2qp4RMu3uJE5+T1xBGRO0wdqJeGLw9jCck9wHP4d6DxUR7J+wIDAQABAoICAA1ytIItY2C6l+kW\nKsKZ5yoBDjYI3xBNmagHPFcHVKJXagBk3XevY/JPNNY0wpE6I8oHClrcV7fSwvgOYjUlGR4VKddy\n6WfOCdza1TwXfqKT80zI5bqyNUXiHg3VamfITQtrA2u7KliBDsDXIDnKqQB0SveSnPjNyj4wWHHn\nekps+s9a8Ou6iAfbEyGSHr+NfH8gPc9wYyl1WTGQq4Kwmo9fYy2A/+xnU1Zvwzl3cLF4DAKoqUyn\nNkgBKSTeCCl61DzmRje042OqjRz9uhBoFq2+ZFLTTR/oO6j9u4g1S6yQNcemVLDyT8uWOS0dA7Bu\nHMGsR7n9Hf4H7jXi9qBkz5/e5mDUHp79ccDUW3gTHDz0Mw15A9JXJd7R1XNkCvNmrTMN9eCDdQRW\nrPMryIX9+ljVswxjrSnSnTOa69Xmc/z90LEgCgYmUhyiax+F1T0AWycMwCdz67BO+rLYuEbcOBo0\nh35uVUeNwrerpFt7kUHBMBRtxkozoAvy1eZEAKifIcceeEkZGQDlzbUjXSAXaSWa3XQFq+lCu5Kn\n88tgjEDSLU8jVIfxdCO3y7pSCqzf02T9f91tb4f3zi/nxc19Ksc5tbC1vj1hMeOKLdDT02r8ygkr\n4Mrfc6AaXQWYpRapx3OA8tNydCnUW1LXq7rQsC12JYn0IKAa5bFx7Ecmi/2JAoIBAQDIZ0es+2D/\nbLVBH6DOGP/eAZzeILwbMOvjm1hB/us6zwTY4MIKWGiA1dkWF4ILNSaGLIbZ+sO6tLzL0LtkynqG\nuWm/25YY8sccwCA6cy8AQ9adN2zJUyWlD+O6tJ0PRim6nV5lnKAmgzSBX6BcTT7GN3BaB/u2JlBq\ncf2F2gkz2eyyOHPKvmT/DoEeTaTSDyAuBAN6lrgTfFI8cszPkMky5dlFz+9kZEC/GKJ33cgYTWxi\nMhKxN+TwIBulghM/LgZ2atgQ0VdRTkqfcSrG+i3XkzhQHc8VAVHzfkHqe1kOfv4ZvFy5QhTOAF0L\nk+sM8P7MIy+PMexpcPqXY89maBQpAoIBAQDZPbXR6tFkTDt/QqGjytbk7CjDzt1KIZPZ4EmFkeXL\nSQjC7NfJu+w8tmpG9V8hN/dCQo2NNaSTNwnX/eikOC/OJuqf11mr3JV+81N82fOz0c6nqZaMBQ7Y\ncZcr9/77pzmACIEe3hY5g+oKO54ENt8h/FYEYPjtn01WyKpXciLFo1Lt/qjxYUwNPRVKrICvDjmN\nCeWhZmpvjl0fyFB6tXLlcAL3Ocp5rbMzhf2emisnnXwbrPmgzmkJ/Wn0wpgk+8SxKA3eeeWKXWgb\nSMs/bbBEIM6MOQ1un8Seg0fVpSGBvCnVtFkwT27nCv6j7COrYkE6BZxj0K5NzyH0aA4LD9GDAoIB\nADuD6JZnxUO1/hJMGU57wCknY3XYVOTiX3ul280lrqg1aOQbw6Sc4tQ4LhNQge9gJoO8X4QG4+/j\n0xnYcH6bX035bH1s8iOQni9co3WYVYIHo4nnNuiHR+vAT0pYbzhlBumD6M/Wdv1ZA9PUGWSwEA9/\n0V77dfZ/ZGxoU/lXalo6wv+eoky4xHe20AO23VcA5PalfH8AmcQ3rJiFI2wVPJtgBWmlOhwfZdca\nss1UUSNeguyaoFB/H/9sGanKenrN6V9rlaVQ9lSQIrs9OY4EKG8YKqYoZCKB1NuySFMhtK4IauAr\nv4HJLTKMixVwJWMfgxwO6wXktqgNxG4HV0W7bRkCggEAYzixS8BxhNrgrd5UD4h8oDBQ6iYYolw1\nuGSdj/k0OKYR713XrVc8rfovDlvR6E00jLnzBxUCJw8TWuiokiDrjL/vl7P3S+zDBynB7xtpGK9y\nMNffX/KLdkZjYnyxpGUbeSPpPZz4D6r1gVj7cjdRsKcc7oEQERAaddHPI4OI6DYRkYwnw5/J6Z4F\nlIa3e70GgimMDSzG3k7qr7KBN5qacLq5UAvAM9UnLRg832zQ2xYt8kIN/eloxlxNQbKDZRjtHHEL\n7JpGQe0puJSF6GGECYnmbNs+DFHCrxeM/sKeTDAR936Y4dzV7YbzCRG4tPV6jzKy3FAa3IUHoCbK\nizjdWwKCAQBc7bQBV7ERpzvd9Ur5Ra7lIxW+0X9gD2U+nIdc7kt5NtvegOFu+OxlAbx7vfm5mx/m\nU9ugBgoUsProO4H6dLfyBbNpydHJvfkycJ65KiBgtr11Upm+8deeZTlckL3Aj2wQGJBKzKyDjBOw\nIHPCCPEczAFc/J6J3JPGcVK9ZCOAUQ2ZiuUl6TItCOCZJ2rpRJS8mpD3Am8dan2fz2UZSN7Xhnvl\nMTip8/M7UmYn9W9IaOybqa8STXBetJExkI2UsdQgZWXyuYXddCcVqPsrHe43leHUZKJq+H8OBdUU\nRCQED9fnCZSloFH2pGE9tQzlbktWmFmqsJib+X70SfLtkpRe\n-----END PRIVATE KEY-----\n";
int main(void)
{
BIO* bio = BIO_new_mem_buf(data, -1);
for (int i = 0; i < 1000; ++i) {
BIO_reset(bio);
EVP_PKEY* pkey = PEM_read_bio_PrivateKey(bio, NULL, NULL, NULL);
EVP_PKEY_free(pkey);
}
BIO_free(bio);
}
With OpenSSL 1.1.1k, it takes about 0.017s to execute, while with master (6ef2f71ac70aff99da277be4a554e3b1fe739050) it takes about 1.4s.
Next one to decode a DER formatted private key:
#include <openssl/evp.h>
#include <openssl/x509.h>
#include <openssl/pem.h>
static char data[] = "\x30\x82\x09\x41\x02\x01\x00\x30\x0d\x06\x09\x2a\x86\x48\x86\xf7\x0d\x01\x01\x01\x05\x00\x04\x82\x09\x2b\x30\x82\x09\x27\x02\x01\x00\x02\x82\x02\x01\x00\xaa\x0f\xda\xb3\x0f\x1e\x42\xee\x78\x8f\xf2\x5b\xdd\xf2\xe5\xf0\x40\xd1\xcb\x26\x36\xa4\x47\x21\x33\x7d\xb5\x79\x14\x9b\x49\xba\x38\x0e\x5e\x31\x79\x3c\xe9\xbb\x58\x80\x80\x82\xd7\x6b\x72\xdf\xa0\xe6\xff\xd2\x7d\x8b\xe0\x36\x46\xec\xd3\xa1\x3b\xb6\x1d\x73\x15\x8e\xdc\x98\x48\xa6\x24\xe2\x15\xb8\x3b\x06\x94\xc5\x32\x4a\x56\x22\xa3\x15\xbd\x42\xf4\xc4\x46\x0a\xb7\x7b\xd3\x4e\x34\xa6\x0f\x11\xa6\xc6\xc0\xcc\xe1\xe9\x96\x23\x1d\x84\x9a\x29\x95\x03\x00\xc9\x55\xe7\xc5\xe1\x5c\xb2\x1f\xe3\xe0\x00\xc9\x5a\x12\x17\xb8\xe2\x85\xbb\x39\x19\xbf\xa7\xa2\x4b\x10\xe1\xa9\x4a\xc6\x22\x43\x2a\xbf\xfd\x8a\x37\xc2\xf8\x73\x23\xe1\x54\x04\x73\x8d\x4c\x89\xbe\xfc\xf8\x89\x97\x31\x1c\xcf\xb8\x28\xf6\x20\xf4\xa8\xad\x8e\x2c\xbc\xcb\x16\x2e\x08\xe6\xf3\xee\x05\x57\xf3\xec\x4a\x47\x76\xcc\xe7\x92\x7d\xf7\x94\x9b\xa3\x9a\xe0\x5d\x2b\x5a\x79\x8e\xc9\x19\x3a\x58\x12\x88\xbf\xa3\x96\xda\x68\xf7\x39\x38\x7a\x7f\xb5\x4e\x07\x2d\xed\x9f\xdc\x54\x91\x8f\xb0\x15\xc6\x3a\x3a\xb1\x5f\xc2\x06\xc7\xda\x42\x7e\xe5\xa8\x9b\x8f\x4d\x26\xac\xbe\x05\x5c\xb2\xd5\x29\xb6\x29\xe5\x80\x19\x16\xf1\x2f\xba\x32\xae\x01\x7d\x8c\xed\x9a\x1c\x4b\xde\x57\x1f\x07\xdb\x40\xad\xe8\x97\xac\x02\x34\xae\xf9\x34\xb3\x93\xae\x85\x50\xb6\x51\xff\xe6\x07\x0d\x88\x9b\x71\x4e\x25\xbc\xd6\x1c\x18\xd7\xb2\x21\x3f\x05\x41\x86\xad\x72\x9d\xbf\xbd\x60\x4c\xcd\xa1\x4b\xaa\x7e\xc1\x2f\x86\xd1\xde\x44\xb7\x54\xfd\x86\x0e\x89\x75\xd0\x54\x44\x82\x61\x78\x10\xe4\x0a\x45\x44\x4c\x5f\x56\xed\x6d\xf1\xaf\xcc\x89\xb3\x48\x3c\xcf\xa7\x24\x87\x09\x87\xcd\x01\x35\xb9\x76\x7c\xbf\x52\x84\xc2\x57\xb8\x18\x11\xcf\xf6\x32\xa7\xff\xf5\x34\xda\x02\xf3\xf3\x24\x3b\xfb\xc0\x69\x90\x42\xad\x2b\x4e\x65\x2b\x59\x2d\x1c\xa2\xd0\x0c\x9e\xbe\xfc\x34\x6b\xb2\x08\x4b\xcd\xa1\xe1\x1e\x9c\x8e\xd7\xb5\x55\xf9\xdd\xf9\xe5\xf8\x30\xe7\x7d\xf1\xaf\xc0\x1c\x9f\xf8\x79\xca\x7d\x94\x8c\x92\x43\x25\x5f\xe4\xda\xdc\x19\x3c\xf5\x03\x92\xed\xc9\x97\x62\xe2\xd7\x86\xc0\x19\x7e\x5f\x6a\xa9\xe1\x13\x2e\xde\xe2\x44\xe7\xe4\xf5\xc4\x11\x91\x3b\x4c\x1d\xa8\x97\x86\x2f\x0f\x63\x09\xc9\x3d\xc0\x73\xf8\x77\xa0\xf1\x51\x1e\xc9\xfb\x02\x03\x01\x00\x01\x02\x82\x02\x00\x0d\x72\xb4\x82\x2d\x63\x60\xba\x97\xe9\x16\x2a\xc2\x99\xe7\x2a\x01\x0e\x36\x08\xdf\x10\x4d\x99\xa8\x07\x3c\x57\x07\x54\xa2\x57\x6a\x00\x64\xdd\x77\xaf\x63\xf2\x4f\x34\xd6\x34\xc2\x91\x3a\x23\xca\x07\x0a\x5a\xdc\x57\xb7\xd2\xc2\xf8\x0e\x62\x35\x25\x19\x1e\x15\x29\xd7\x72\xe9\x67\xce\x09\xdc\xda\xd5\x3c\x17\x7e\xa2\x93\xf3\x4c\xc8\xe5\xba\xb2\x35\x45\xe2\x1e\x0d\xd5\x6a\x67\xc8\x4d\x0b\x6b\x03\x6b\xbb\x2a\x58\x81\x0e\xc0\xd7\x20\x39\xca\xa9\x00\x74\x4a\xf7\x92\x9c\xf8\xcd\xca\x3e\x30\x58\x71\xe7\x7a\x4a\x6c\xfa\xcf\x5a\xf0\xeb\xba\x88\x07\xdb\x13\x21\x92\x1e\xbf\x8d\x7c\x7f\x20\x3d\xcf\x70\x63\x29\x75\x59\x31\x90\xab\x82\xb0\x9a\x8f\x5f\x63\x2d\x80\xff\xec\x67\x53\x56\x6f\xc3\x39\x77\x70\xb1\x78\x0c\x02\xa8\xa9\x4c\xa7\x36\x48\x01\x29\x24\xde\x08\x29\x7a\xd4\x3c\xe6\x46\x37\xb4\xe3\x63\xaa\x8d\x1c\xfd\xba\x10\x68\x16\xad\xbe\x64\x52\xd3\x4d\x1f\xe8\x3b\xa8\xfd\xbb\x88\x35\x4b\xac\x90\x35\xc7\xa6\x54\xb0\xf2\x4f\xcb\x96\x39\x2d\x1d\x03\xb0\x6e\x1c\xc1\xac\x47\xb9\xfd\x1d\xfe\x07\xee\x35\xe2\xf6\xa0\x64\xcf\x9f\xde\xe6\x60\xd4\x1e\x9e\xfd\x71\xc0\xd4\x5b\x78\x13\x1c\x3c\xf4\x33\x0d\x79\x03\xd2\x57\x25\xde\xd1\xd5\x73\x64\x0a\xf3\x66\xad\x33\x0d\xf5\xe0\x83\x75\x04\x56\xac\xf3\x2b\xc8\x85\xfd\xfa\x58\xd5\xb3\x0c\x63\xad\x29\xd2\x9d\x33\x9a\xeb\xd5\xe6\x73\xfc\xfd\xd0\xb1\x20\x0a\x06\x26\x52\x1c\xa2\x6b\x1f\x85\xd5\x3d\x00\x5b\x27\x0c\xc0\x27\x73\xeb\xb0\x4e\xfa\xb2\xd8\xb8\x46\xdc\x38\x1a\x34\x87\x7e\x6e\x55\x47\x8d\xc2\xb7\xab\xa4\x5b\x7b\x91\x41\xc1\x30\x14\x6d\xc6\x4a\x33\xa0\x0b\xf2\xd5\xe6\x44\x00\xa8\x9f\x21\xc7\x1e\x78\x49\x19\x19\x00\xe5\xcd\xb5\x23\x5d\x20\x17\x69\x25\x9a\xdd\x74\x05\xab\xe9\x42\xbb\x92\xa7\xf3\xcb\x60\x8c\x40\xd2\x2d\x4f\x23\x54\x87\xf1\x74\x23\xb7\xcb\xba\x52\x0a\xac\xdf\xd3\x64\xfd\x7f\xdd\x6d\x6f\x87\xf7\xce\x2f\xe7\xc5\xcd\x7d\x2a\xc7\x39\xb5\xb0\xb5\xbe\x3d\x61\x31\xe3\x8a\x2d\xd0\xd3\xd3\x6a\xfc\xca\x09\x2b\xe0\xca\xdf\x73\xa0\x1a\x5d\x05\x98\xa5\x16\xa9\xc7\x73\x80\xf2\xd3\x72\x74\x29\xd4\x5b\x52\xd7\xab\xba\xd0\xb0\x2d\x76\x25\x89\xf4\x20\xa0\x1a\xe5\xb1\x71\xec\x47\x26\x8b\xfd\x89\x02\x82\x01\x01\x00\xc8\x67\x47\xac\xfb\x60\xff\x6c\xb5\x41\x1f\xa0\xce\x18\xff\xde\x01\x9c\xde\x20\xbc\x1b\x30\xeb\xe3\x9b\x58\x41\xfe\xeb\x3a\xcf\x04\xd8\xe0\xc2\x0a\x58\x68\x80\xd5\xd9\x16\x17\x82\x0b\x35\x26\x86\x2c\x86\xd9\xfa\xc3\xba\xb4\xbc\xcb\xd0\xbb\x64\xca\x7a\x86\xb9\x69\xbf\xdb\x96\x18\xf2\xc7\x1c\xc0\x20\x3a\x73\x2f\x00\x43\xd6\x9d\x37\x6c\xc9\x53\x25\xa5\x0f\xe3\xba\xb4\x9d\x0f\x46\x29\xba\x9d\x5e\x65\x9c\xa0\x26\x83\x34\x81\x5f\xa0\x5c\x4d\x3e\xc6\x37\x70\x5a\x07\xfb\xb6\x26\x50\x6a\x71\xfd\x85\xda\x09\x33\xd9\xec\xb2\x38\x73\xca\xbe\x64\xff\x0e\x81\x1e\x4d\xa4\xd2\x0f\x20\x2e\x04\x03\x7a\x96\xb8\x13\x7c\x52\x3c\x72\xcc\xcf\x90\xc9\x32\xe5\xd9\x45\xcf\xef\x64\x64\x40\xbf\x18\xa2\x77\xdd\xc8\x18\x4d\x6c\x62\x32\x12\xb1\x37\xe4\xf0\x20\x1b\xa5\x82\x13\x3f\x2e\x06\x76\x6a\xd8\x10\xd1\x57\x51\x4e\x4a\x9f\x71\x2a\xc6\xfa\x2d\xd7\x93\x38\x50\x1d\xcf\x15\x01\x51\xf3\x7e\x41\xea\x7b\x59\x0e\x7e\xfe\x19\xbc\x5c\xb9\x42\x14\xce\x00\x5d\x0b\x93\xeb\x0c\xf0\xfe\xcc\x23\x2f\x8f\x31\xec\x69\x70\xfa\x97\x63\xcf\x66\x68\x14\x29\x02\x82\x01\x01\x00\xd9\x3d\xb5\xd1\xea\xd1\x64\x4c\x3b\x7f\x42\xa1\xa3\xca\xd6\xe4\xec\x28\xc3\xce\xdd\x4a\x21\x93\xd9\xe0\x49\x85\x91\xe5\xcb\x49\x08\xc2\xec\xd7\xc9\xbb\xec\x3c\xb6\x6a\x46\xf5\x5f\x21\x37\xf7\x42\x42\x8d\x8d\x35\xa4\x93\x37\x09\xd7\xfd\xe8\xa4\x38\x2f\xce\x26\xea\x9f\xd7\x59\xab\xdc\x95\x7e\xf3\x53\x7c\xd9\xf3\xb3\xd1\xce\xa7\xa9\x96\x8c\x05\x0e\xd8\x71\x97\x2b\xf7\xfe\xfb\xa7\x39\x80\x08\x81\x1e\xde\x16\x39\x83\xea\x0a\x3b\x9e\x04\x36\xdf\x21\xfc\x56\x04\x60\xf8\xed\x9f\x4d\x56\xc8\xaa\x57\x72\x22\xc5\xa3\x52\xed\xfe\xa8\xf1\x61\x4c\x0d\x3d\x15\x4a\xac\x80\xaf\x0e\x39\x8d\x09\xe5\xa1\x66\x6a\x6f\x8e\x5d\x1f\xc8\x50\x7a\xb5\x72\xe5\x70\x02\xf7\x39\xca\x79\xad\xb3\x33\x85\xfd\x9e\x9a\x2b\x27\x9d\x7c\x1b\xac\xf9\xa0\xce\x69\x09\xfd\x69\xf4\xc2\x98\x24\xfb\xc4\xb1\x28\x0d\xde\x79\xe5\x8a\x5d\x68\x1b\x48\xcb\x3f\x6d\xb0\x44\x20\xce\x8c\x39\x0d\x6e\x9f\xc4\x9e\x83\x47\xd5\xa5\x21\x81\xbc\x29\xd5\xb4\x59\x30\x4f\x6e\xe7\x0a\xfe\xa3\xec\x23\xab\x62\x41\x3a\x05\x9c\x63\xd0\xae\x4d\xcf\x21\xf4\x68\x0e\x0b\x0f\xd1\x83\x02\x82\x01\x00\x3b\x83\xe8\x96\x67\xc5\x43\xb5\xfe\x12\x4c\x19\x4e\x7b\xc0\x29\x27\x63\x75\xd8\x54\xe4\xe2\x5f\x7b\xa5\xdb\xcd\x25\xae\xa8\x35\x68\xe4\x1b\xc3\xa4\x9c\xe2\xd4\x38\x2e\x13\x50\x81\xef\x60\x26\x83\xbc\x5f\x84\x06\xe3\xef\xe3\xd3\x19\xd8\x70\x7e\x9b\x5f\x4d\xf9\x6c\x7d\x6c\xf2\x23\x90\x9e\x2f\x5c\xa3\x75\x98\x55\x82\x07\xa3\x89\xe7\x36\xe8\x87\x47\xeb\xc0\x4f\x4a\x58\x6f\x38\x65\x06\xe9\x83\xe8\xcf\xd6\x76\xfd\x59\x03\xd3\xd4\x19\x64\xb0\x10\x0f\x7f\xd1\x5e\xfb\x75\xf6\x7f\x64\x6c\x68\x53\xf9\x57\x6a\x5a\x3a\xc2\xff\x9e\xa2\x4c\xb8\xc4\x77\xb6\xd0\x03\xb6\xdd\x57\x00\xe4\xf6\xa5\x7c\x7f\x00\x99\xc4\x37\xac\x98\x85\x23\x6c\x15\x3c\x9b\x60\x05\x69\xa5\x3a\x1c\x1f\x65\xd7\x1a\xb2\xcd\x54\x51\x23\x5e\x82\xec\x9a\xa0\x50\x7f\x1f\xff\x6c\x19\xa9\xca\x7a\x7a\xcd\xe9\x5f\x6b\x95\xa5\x50\xf6\x54\x90\x22\xbb\x3d\x39\x8e\x04\x28\x6f\x18\x2a\xa6\x28\x64\x22\x81\xd4\xdb\xb2\x48\x53\x21\xb4\xae\x08\x6a\xe0\x2b\xbf\x81\xc9\x2d\x32\x8c\x8b\x15\x70\x25\x63\x1f\x83\x1c\x0e\xeb\x05\xe4\xb6\xa8\x0d\xc4\x6e\x07\x57\x45\xbb\x6d\x19\x02\x82\x01\x00\x63\x38\xb1\x4b\xc0\x71\x84\xda\xe0\xad\xde\x54\x0f\x88\x7c\xa0\x30\x50\xea\x26\x18\xa2\x5c\x35\xb8\x64\x9d\x8f\xf9\x34\x38\xa6\x11\xef\x5d\xd7\xad\x57\x3c\xad\xfa\x2f\x0e\x5b\xd1\xe8\x4d\x34\x8c\xb9\xf3\x07\x15\x02\x27\x0f\x13\x5a\xe8\xa8\x92\x20\xeb\x8c\xbf\xef\x97\xb3\xf7\x4b\xec\xc3\x07\x29\xc1\xef\x1b\x69\x18\xaf\x72\x30\xd7\xdf\x5f\xf2\x8b\x76\x46\x63\x62\x7c\xb1\xa4\x65\x1b\x79\x23\xe9\x3d\x9c\xf8\x0f\xaa\xf5\x81\x58\xfb\x72\x37\x51\xb0\xa7\x1c\xee\x81\x10\x11\x10\x1a\x75\xd1\xcf\x23\x83\x88\xe8\x36\x11\x91\x8c\x27\xc3\x9f\xc9\xe9\x9e\x05\x94\x86\xb7\x7b\xbd\x06\x82\x29\x8c\x0d\x2c\xc6\xde\x4e\xea\xaf\xb2\x81\x37\x9a\x9a\x70\xba\xb9\x50\x0b\xc0\x33\xd5\x27\x2d\x18\x3c\xdf\x6c\xd0\xdb\x16\x2d\xf2\x42\x0d\xfd\xe9\x68\xc6\x5c\x4d\x41\xb2\x83\x65\x18\xed\x1c\x71\x0b\xec\x9a\x46\x41\xed\x29\xb8\x94\x85\xe8\x61\x84\x09\x89\xe6\x6c\xdb\x3e\x0c\x51\xc2\xaf\x17\x8c\xfe\xc2\x9e\x4c\x30\x11\xf7\x7e\x98\xe1\xdc\xd5\xed\x86\xf3\x09\x11\xb8\xb4\xf5\x7a\x8f\x32\xb2\xdc\x50\x1a\xdc\x85\x07\xa0\x26\xca\x8b\x38\xdd\x5b\x02\x82\x01\x00\x5c\xed\xb4\x01\x57\xb1\x11\xa7\x3b\xdd\xf5\x4a\xf9\x45\xae\xe5\x23\x15\xbe\xd1\x7f\x60\x0f\x65\x3e\x9c\x87\x5c\xee\x4b\x79\x36\xdb\xde\x80\xe1\x6e\xf8\xec\x65\x01\xbc\x7b\xbd\xf9\xb9\x9b\x1f\xe6\x53\xdb\xa0\x06\x0a\x14\xb0\xfa\xe8\x3b\x81\xfa\x74\xb7\xf2\x05\xb3\x69\xc9\xd1\xc9\xbd\xf9\x32\x70\x9e\xb9\x2a\x20\x60\xb6\xbd\x75\x52\x99\xbe\xf1\xd7\x9e\x65\x39\x5c\x90\xbd\xc0\x8f\x6c\x10\x18\x90\x4a\xcc\xac\x83\x8c\x13\xb0\x20\x73\xc2\x08\xf1\x1c\xcc\x01\x5c\xfc\x9e\x89\xdc\x93\xc6\x71\x52\xbd\x64\x23\x80\x51\x0d\x99\x8a\xe5\x25\xe9\x32\x2d\x08\xe0\x99\x27\x6a\xe9\x44\x94\xbc\x9a\x90\xf7\x02\x6f\x1d\x6a\x7d\x9f\xcf\x65\x19\x48\xde\xd7\x86\x7b\xe5\x31\x38\xa9\xf3\xf3\x3b\x52\x66\x27\xf5\x6f\x48\x68\xec\x9b\xa9\xaf\x12\x4d\x70\x5e\xb4\x91\x31\x90\x8d\x94\xb1\xd4\x20\x65\x65\xf2\xb9\x85\xdd\x74\x27\x15\xa8\xfb\x2b\x1d\xee\x37\x95\xe1\xd4\x64\xa2\x6a\xf8\x7f\x0e\x05\xd5\x14\x44\x24\x04\x0f\xd7\xe7\x09\x94\xa5\xa0\x51\xf6\xa4\x61\x3d\xb5\x0c\xe5\x6e\x4b\x56\x98\x59\xaa\xb0\x98\x9b\xf9\x7e\xf4\x49\xf2\xed\x92\x94\x5e";
int main(void)
{
BIO* bio = BIO_new_mem_buf(data, 2373);
PKCS8_PRIV_KEY_INFO* info = d2i_PKCS8_PRIV_KEY_INFO_bio(bio, NULL);
BIO_free(bio);
for (int i = 0; i < 1000; ++i) {
EVP_PKEY* privKey = EVP_PKCS82PKEY(info);
EVP_PKEY_free(privKey);
}
PKCS8_PRIV_KEY_INFO_free(info);
}
This takes about 0.005s with 1.1.1k, while it takes about 0.44s with master. I’m using x64 linux, with Core i7-8700K.
About this issue
- Original URL
- State: open
- Created 6 years ago
- Comments: 62 (58 by maintainers)
Commits related to this issue
- Decoding PKCS#8: separate decoding of encrypted and unencrypted PKCS#8 This has us switch from the 'structure' "pkcs8" to "PrivateKeyInfo", which is sensible considering we already have "SubjectPubli... — committed to levitte/openssl by levitte 5 years ago
- DECODER: Adapt addition of extra decoder implementations The new PKCS#8 decoder implementation decodes from DER to DER. OSSL_DECODER_CTX_add_extra() wasn't suited for this case; we had to modify it t... — committed to levitte/openssl by levitte 5 years ago
- Decoder resolution performance optimizations This refactors decoder functionality to reduce calls to OSSL_DECODER_is_a / EVP_KEYMGMT_is_a, which are substantial bottlenecks in the performance of repe... — committed to hlandau/openssl by hlandau 5 years ago
- Decoder resolution performance optimizations This refactors decoder functionality to reduce calls to OSSL_DECODER_is_a / EVP_KEYMGMT_is_a, which are substantial bottlenecks in the performance of repe... — committed to openssl/openssl by hlandau 5 years ago
- Decoder resolution performance optimizations This refactors decoder functionality to reduce calls to OSSL_DECODER_is_a / EVP_KEYMGMT_is_a, which are substantial bottlenecks in the performance of repe... — committed to t8m/openssl by hlandau 5 years ago
- Decoder resolution performance optimizations This refactors decoder functionality to reduce calls to OSSL_DECODER_is_a / EVP_KEYMGMT_is_a, which are substantial bottlenecks in the performance of repe... — committed to t8m/openssl by hlandau 5 years ago
- Decoder resolution performance optimizations This refactors decoder functionality to reduce calls to OSSL_DECODER_is_a / EVP_KEYMGMT_is_a, which are substantial bottlenecks in the performance of repe... — committed to t8m/openssl by hlandau 5 years ago
- Decoder resolution performance optimizations This refactors decoder functionality to reduce calls to OSSL_DECODER_is_a / EVP_KEYMGMT_is_a, which are substantial bottlenecks in the performance of repe... — committed to openssl/openssl by hlandau 5 years ago
- This refactors decoder functionality to reduce calls to OSSL_DECODER_is_a / EVP_KEYMGMT_is_a, which are substantial bottlenecks in the performance of repeated decode operations (see #15199). Reviewed... — committed to krk/openssl by krk 4 years ago
- Decoder resolution performance optimizations This refactors decoder functionality to reduce calls to OSSL_DECODER_is_a / EVP_KEYMGMT_is_a, which are substantial bottlenecks in the performance of repe... — committed to krk/openssl by krk 4 years ago
- Optimise PKEY decoders The most expensive part of using a PKEY decoder is the OSSL_DECODER_CTX_new_for_pkey() call. This builds up all of the decoder chains, which is a complex and time consuming ope... — committed to mattcaswell/openssl by mattcaswell 3 years ago
There may be some structural issues to address before doing that. This
collect_extra_decoderfunction seems to called 2000 times in a single private key parsing operation.Poking around the code in the profile, it seems OpenSSL is quadratic in the number of decoders you have, and there are quite a lot of them. I don’t fully follow what
OSSL_DECODER_CTX_add_extrais doing, but it seems OpenSSL now models generic “container” formats like PEM, PrivateKeyInfo, etc., as pluggable conversion functions with input/output formats? And then, before it does anything, it looks likeOSSL_DECODER_CTX_add_extratries to build all possible paths through these conversion functions, matching inputs to outputs?That first layer of this path-building starts with a set of leaf “decoder instances” and then, for each one (twice, actually), iterates over every “decoder” to find the ones that can chain backwards to that decoder instances input type. That means decoders are tapped on the head 2 * decoder_instances * decoders times. (And then the next layer adds another O(decoders) iterations. I suspect this would also go exponential if the wrong series of container formats were added to the system.)
I couldn’t find any documentation on why this path-builder design was chosen. It is a little odd to me because one shouldn’t need to explore the whole tree of paths to parse one input. I think this comes out of OpenSSL working backwards from all possible outputs, rather than working forwards from the one input you have. The way these container formats are designed (and generally when parsing things), the forwards is the natural direction:
PEM_read_PrivateKeybeing called, we know the input is PEM. PEM is a type string (BEGIN WHATEVER) followed by base64-encoded data.RSA PRIVATE KEY, so you find the provider(s) that supportRSA PRIVATE KEYand ask them to parse the data. They should parse an RSAPrivateKey ASN.1 structure and that’s the end of it.PRIVATE KEY. That means the type is PKCS#8 PrivateKeyInfo. That too is a generic, provider-independent format that, more-or-less, expands to another (type, data) pair. This time the type is an AlgorithmIdentifier.At no point do you need to explore every possible chain of decoders. The only operation you need is “I have an input of format X. Call the function that parses X”. Where there need only be a single implementation, that can be a direct function call. Where you wish it to be pluggable, that only needs a simple lookup. A container format decoder is just something that needs to do that dispatch a second time.
While saying “OpenSSL 3.x has regressions over 1.1.1” is indeed difficult to evaluate (but still useful as an overall tracking bug), saying “OpenSSL 3.x has regressions over 1.1.1 for private key parsing” is a very evaluatable goal because you have a concrete benchmark, or collection of benchmarks.
That’s precisely this issue. Like I said, bug organization strategies vary, but if I were organizing fixing all this, I would keep this open as a concrete goal. And then I would have an overall tracking bug linking all the benchmarks I know of where the new release showed a marked performance regression, because even if “OpenSSL 3.x has regressions over 1.1.1” is hard to evaluate, it is a real problem. The task from there is to work with the user to help identify the particular operation, so you can graduate to an evaluatable goal.
From there, in investigating these goals, you may identify particular bottlenecks and come up with concrete tasks that will fix things. Those could be one-off PRs if quick and easy, or perhaps an issues tracking those tasks.
That is not what I’m saying at all. There are certainly a bunch of small mistakes to fix, but when so many problems in a system are caused by architectural changes, it suggests the architecture was bad. In that case, the fix is not fixing 1,000 little things but replacing the bad architecture.
I ran through the start of this exercise for you all a year ago in https://github.com/openssl/openssl/issues/15199#issuecomment-1631038583. I think I’d also pointed out in various other threads how designing a system around format guessing was not only unperformant but also a security risk.
Another example is the provider lookup system and the lazy-loading and global caches. https://github.com/openssl/openssl/discussions/23369 actually came out of me reading through the provider code, because I wanted to see how easy it’d be to build a simpler and more efficient scheme where each provider object is immutable after load. Such a design is much more straightforward to implement correctly and without thread contention. Of course, other architectural flaws will still require you all keep the provider list mutable (and thus perhaps you need the RCU machinery), but there is no good reason for individual providers to be mutable. I would strongly recommend you all think through that, with an eye towards minimizing mutable shared state.
Indeed I think the caches are a symptom of this 1,000 small issues theory. When one sees a single performance issue, it’s tempting to add a targeted cache for it, because it’s easy and doesn’t require rethinking the whole design. And if it is indeed a singular issue, that may even be a good strategy. But when the design itself had problems, the result is caches everywhere. Compounded, this strategy has costs, from a little extra work that every operation needs to do, to thread contention from mutable state, to complexity. They also make benchmarks less representative, as single- and multi-threaded benchmarks no longer translate, and there is amortization hidden behind repeat runs.
Keep in mind the notes here. The current design is parsing things in the wrong direction and is doing piles and piles of unnecessary work. https://github.com/openssl/openssl/issues/15199#issuecomment-1631038583
Setting aside whether all the design goals were wise, OpenSSL could still have met its design goals with a better implementation. These are, at the end of the day, not very complex formats.
Regarding base64, keep in mind that PEM is often used to carry private keys. We actually switched our base64 decoder to a constant-time one.
Like I said, I would suggest this issue should be “done” when private key parsing, a concrete benchmark, is on par with 1.1.1. That’s what OpenSSL’s users presumably care about.
From a planning standpoint your concrete tasks here are:
Mmm. Sounds like you’re trying to navigate a tension between concrete, closable bugs vs not knowing all the 3.0 performance regressions that touch the private key parser. Because of the way 3.0 was developed (a whole slew of far-reaching, risky, and unperformant architectural changes that all landed in one go) as well as how much has been built atop the shaky foundations since, I think that tension is somewhat fundamental. It’ll be too difficult to go back and tease apart which architectural decisions impacted performance of which operations. You’ll have to instead discover the problems as you fix them.
I dunno how you all like to track bugs, but perhaps then this bug should track the overall performance goal (get 3.0 on par with 1.1.1), and only be closed when you meet that goal. And then you can file child bugs for concrete problems (such as the bad parser architecture) as you identify them, with those being closed as each concrete problem is fixed.
I am afraid this is as fast as it can be with the current design.
By wildcard decoding, I assume you mean where the caller forgets to tell you the format, and OpenSSL has to try between several overlapping (i.e. ambiguous) formats? Ambiguous parsers are a thoroughly well-studied and well-understood source of security vulnerabilities, not to mention unpredictable behavior (i.e backwards compatibility risks). Hopefully you all carefully analyzed all pairs of supported formats before exposing them to be part of a wildcard decode. E.g. all 32-byte strings are valid raw X25519 keys, but a 32-byte string may also be something else entirely, so raw X25519 can never participate in such a scheme.
Design problems aside, I see no reason why it requires this backwards parse. Even if the caller forgets to tell you the format, that problem only applies at the top level. Once you’ve decided to try, say, DER PKCS#8 PrivateKeyInfo, or PEM[*], all parsing from then on is unambiguous. You only need to try ambiguous cases and backtrack at the top-level. And, of course, even if you did need to backtrack on the inner layers (again, you do not), that’s still no reason to build the graph ahead of time.
You really don’t need to pre-explore all possible call graphs like this. It’s just overhead, not flexibility.
[*] PEM and DER are not analogous, so a good parser API design would take this into account. PEM contains a type header, so the caller can actually say “I have a PEM private key” and let the library dispatch between “BEGIN RSA PRIVATE KEY” and “BEGIN PRIVATE KEY”. Whereas a DER encoding is not necessarily sufficient to identify the structure, so the caller must say “I have a DER RSAPrivateKey” or “I have a DER PrivateKeyInfo”.
Unfortunately we will also have to support the wildcard decoding as regressing this is not an option.
Now that 3.2 is released and you all are working on 3.3, I may suggest taking some time to think about the structural issues in the encoder/decoder system. Working backwards and guessing formats is not a performant, secure, or well-defined way to decode an input. Start from the input type that the caller asked you to parse and work forwards.
Yes, I agree but it’s probably more of a 3.3 thing whereas I think we can get my patch into 3.2. It’s not completely clear to me if we can do what @davidben suggests without breaking changes/new API. It’s at the least a significant refactor. I do think we should do it though.
Regardless of my patch I think @davidben makes good points and we do need to look at the design of the decoder subsystem.
This reminds me that we had/have code that tries all possible formats and key types, even when we already know which format and key type it is. There are just too many calls to those functions.
Incorporating my recent performance fixes, as well as my libctx refactor, this drops from 1.315s in 3.0 to 0.982s (1.34x speedup). 1.1 is still 0.038s, so this remains an issue.
Yes the OSSL_LIB_CTX locking is particularly horrific (as mentioned in #17116) and
ossl_lib_ctx_get_datawill be a primary path for hitting that code - so its not a big surprise that we’re running into problems as a result.