openssl: X509_digest_sig reports unsupported algorithm with RSA-PSS

Meanwhile we got there the following interesting feedback regarding EdDSA:

I prefer to use SHA-512 for Ed25519 and SHAKE256(x, 64) for Ed448, as in RFC8419: 2.3. Message Digest Algorithm Identifiers When the signer includes signed attributes, a message digest algorithm is used to compute the message digest on the eContent value. When signing with Ed25519, the message digest algorithm MUST be SHA-512 [FIPS180]. Additional information on SHA-512 is available in [RFC6234]. When signing with Ed448, the message digest algorithm MUST be SHAKE256 [FIPS202] with a 512-bit output value.

SHA-512 is a natural fit for Ed25519 since SHA-512 is already used as the H(x) function in EdDSA; likewise SHAKE256 for Ed448. This is specified in RFC 8032, and is (IIUC) the motivation for the CMS recommendations in RFC 8419.

This is applicable in all cases where the AlgorithmIdentifier contains params to be passed to the operational (such as signature) algorithm.

But the structure of the AlgorithmIdentifier parameters is inherently specific to the algorithm. So I think we will need to add specific code for each such algorithm (starting with PSS), regardless of where we actually end up adding that code.

Sorry for responding slow - I am on vacation this week. Good that this issue has come up now, since @HBrock, further colleagues, and me are updating RFC 4210 and in particular are editing a new RFC on CMP algorithms: https://datatracker.ietf.org/doc/html/draft-ietf-lamps-cmp-algorithms My proposal would be to define SHA256 as the default certHash algorithm in case a cert signature alg does not imply any.

@t8m, the “intention” comes from RFC 4210, and is about certHash in this structure:

     CertStatus ::= SEQUENCE {
        certHash    OCTET STRING,
        -- the hash of the certificate, using the same hash algorithm
        -- as is used to create and verify the certificate signature
        certReqId   INTEGER,
        -- to match this confirmation with the corresponding req/rep
        statusInfo  PKIStatusInfo OPTIONAL
     }

I tried to find more details, but no, RFC 4210 is quite vague on the matter

You don’t need to be an RSA-PSS expert. This is applicable in all cases where the AlgorithmIdentifier contains params to be passed to the operational (such as signature) algorithm. This is solved in legacy code in a manner that doesn’t port to providers, period… and it’s solved for EVP_CIPHER specifically, through the EVP_CIPHER_param_to_asn1() and EVP_CIPHER_asn1_to_param() calls.

Currently, this means that code that need to rely on this sort of parameter being passed need to rely on legacy methods, or we need to put in specific calls for RSA-PSS. The latter is fairly easy, and while it’s a temporary solution, I think that at this stage, that’s the way to go. Look at #15527 to see how to do that.

Or, alternatively, we could for Eddsa algos just fallback to SHA256 or SHA512.

Either way we should decide what sensible behaviour is and document it.

Or, alternatively, we could for Eddsa algos just fallback to SHA256 or SHA512.

This will presumably also have problems with Ed25519/Ed448. They don’t use a digest alg to create the signature.

Well, for those the failure would be IMO legit as the hash is part of the signature algorithm and so the used digest is not meaningful by itself.