openssl: Some old algorithms in PKCS#12 files don't work any more

Using alpha16-dev (1f3b58d8413cfa) compiled with ./config enable-md2 enable-rc5 no-shared enable-ec_nistp_64_gcc_128 -ggdb on Fedora 32.

Some files that are readable with 1.1.1 aren’t readable with 3.0.0-alpha16-dev, even with legacy provider loaded.

Algorithms affected:

• PBES2 with RC2-CBC-128
• PBES2 with RC2-CBC-40
• PBES2 with RC2-CBC-64
• PBES1 with pbeWithSHAAnd128BitRC4
• PBES1 with pbeWithSHAAnd40BitRC4

Reproducer for RC2-CBC-128:

# compile openssl with old algorithms enabled
# checkout the keyfile corpus in parallel directory:
git clone https://github.com/redhat-qe-security/keyfile-corpus.git
cd openssl
apps/openssl pkcs12 -provider-path providers/ -provider legacy -provider default -in ../'keyfile-corpus/rsa(2048,sha256),cert&key(PBES2(PBKDF2(salt(8),iter(2048),keyLen(16),prf(default)),rc2-cbc(keyBits(56=128bit),IV(8)))),mac(sha1,salt(8),iter(2048)),pass(ascii).p12' -out /dev/null -info -nodes -passin file:../keyfile-corpus/password-ascii.txt

OpenSSL output:

MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: PBES2, PBKDF2, RC2-CBC, Iteration 2048, PRF hmacWithSHA1
Error outputting keys and certificates
4057DE1C127F0000:error:0680006D:asn1 encoding routines:ASN1_TYPE_get_octetstring:data is wrong:crypto/asn1/evp_asn1.c:37:
4057DE1C127F0000:error:0300007A:digital envelope routines:evp_cipher_asn1_to_param_ex:cipher parameter error:crypto/evp/evp_lib.c:230:
4057DE1C127F0000:error:0300007A:digital envelope routines:PKCS5_v2_PBE_keyivgen:cipher parameter error:crypto/evp/p5_crpt2.c:149:

Reproducer for RC4:

apps/openssl pkcs12 -provider-path providers/ -provider legacy -provider default -in '../keyfile-corpus/rsa(2048,sha256),cert&key(pbeWithSHAAnd128BitRC4,salt(8),iter(2048)),mac(sha1,salt(8),iter(2048)),pass(ascii).p12' -out /dev/null -info -nodes -passin 'file:../keyfile-corpus/password-ascii.txt'

OpenSSL output:

MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: pbeWithSHA1And128BitRC4, Iteration 2048
Error outputting keys and certificates
40374D13F17F0000:error:1180006A:PKCS12 routines:PKCS12_PBE_keyivgen:iv gen error:crypto/pkcs12/p12_crpt.c:61:

Other reproducers:

apps/openssl pkcs12 -provider-path providers/ -provider legacy -provider default -in '../keyfile-corpus/rsa(2048,sha256),cert&key(PBES2(PBKDF2(salt(8),iter(2048),keyLen(5),prf(default)),rc2-cbc(keyBits(160=40bit),IV(8)))),mac(sha1,salt(8),iter(2048)),pass(ascii).p12' -out /dev/null -info -nodes -passin 'file:../keyfile-corpus/password-ascii.txt'
apps/openssl pkcs12 -provider-path providers/ -provider legacy -provider default -in '../keyfile-corpus/rsa(2048,sha256),cert&key(PBES2(PBKDF2(salt(8),iter(2048),keyLen(8),prf(default)),rc2-cbc(keyBits(120=64bit),IV(8)))),mac(sha1,salt(8),iter(2048)),pass(ascii).p12' -out /dev/null -info -nodes -passin 'file:../keyfile-corpus/password-ascii.txt'
apps/openssl pkcs12 -provider-path providers/ -provider legacy -provider default -in '../keyfile-corpus/rsa(2048,sha256),cert&key(pbeWithSHAAnd40BitRC4,salt(8),iter(2048)),mac(sha1,salt(8),iter(2048)),pass(ascii).p12'  -out /dev/null -info -nodes -passin 'file:../keyfile-corpus/password-ascii.txt'