caddy: trusted_leaf_cert_file does not work as documented

I played around with TLS client certificates and found the behavior of trusted_leaf_cert_file really confusing and to be inconsistent with Caddy’s documentation.

Example

First, generate two self-signed certificates to use for our tests:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -keyout key1.pem -out crt1.pem -sha256 -days 42 -subj '/CN=subj1' -nodes
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -keyout key2.pem -out crt2.pem -sha256 -days 42 -subj '/CN=subj2' -nodes

Create the following Caddyfile:

{
	admin off
	local_certs
	skip_install_trust
	http_port 8000
}

localhost:8001 {
	tls {
		client_auth {
			trusted_ca_cert_file crt1.pem
		}
	}
}

localhost:8002 {
	tls {
		client_auth {
			trusted_leaf_cert_file crt2.pem
		}
	}
}

localhost:8003 {
	tls {
		client_auth {
			mode require
			trusted_leaf_cert_file crt2.pem
		}
	}
}

localhost:8004 {
	tls {
		client_auth {
			trusted_ca_cert_file crt1.pem
			trusted_leaf_cert_file crt2.pem
		}
	}
}

Note that in all cases where mode it is not explicitly set, it defaults to require_and_verify as at least one CA or leaf certificate was specified.

And start Caddy:

caddy run --config Caddyfile

Expected behavior

The documentation states the following:

Multiple trusted_* directives may be used to specify multiple CA or leaf certificates. Client certificates which are not listed as one of the leaf certificates or signed by any of the specified CAs will be rejected according to the mode.

Therefore, I’d expect crt1 to be accepted on each port that has it listed as a trusted CA and crt2 on each port that has it listed as a trusted leaf. In addition, when more require is set, I’d expect every certificate to be accepted.

Actual behavior

trusted_ca_cert_file crt1.pem

Only the specified crt1 works, as expected:

$ curl -ki --cert crt1.pem --key key1.pem https://localhost:8001 
HTTP/2 200 
server: Caddy
content-length: 0
date: Mon, 10 Jan 2022 20:25:25 GMT

$ curl -ki --cert crt2.pem --key key2.pem https://localhost:8001 
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0

trusted_leaf_cert_file crt2.pem

Neither certificate works, I’d expect crt2 to work:

$ curl -ki --cert crt1.pem --key key1.pem https://localhost:8002
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0
$ curl -ki --cert crt2.pem --key key2.pem https://localhost:8002
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0

mode require + trusted_leaf_cert_file crt2.pem

Only crt2 works, even though the configuration explicitly states to require but not verify a certificate:

$ curl -ki --cert crt1.pem --key key1.pem https://localhost:8003
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0
$ curl -ki --cert crt2.pem --key key2.pem https://localhost:8003
HTTP/2 200 
server: Caddy
content-length: 0
date: Mon, 10 Jan 2022 20:25:48 GMT

trusted_ca_cert_file crt1.pem + trusted_leaf_cert_file crt2.pem

Again, neither certificate works, I’d expect both to work:

$ curl -ki --cert crt1.pem --key key1.pem https://localhost:8004
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0
$ curl -ki --cert crt2.pem --key key2.pem https://localhost:8004
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0

Reading code and docs

For trusted_ca_cert_file, Caddy populates a x509.CertPool and sets it to the ClientCAs attribute of tls.Config: https://github.com/caddyserver/caddy/blob/c48fadc4a7655008d13076c7f757c36368e2ca13/modules/caddytls/connpolicy.go#L361-L379

For trusted_leaf_cert_file, Caddy sets the VerifyPeerCertificate attribute of tls.Config to a custom function checking the certificates against the list of specified leaf certificates: https://github.com/caddyserver/caddy/blob/c48fadc4a7655008d13076c7f757c36368e2ca13/modules/caddytls/connpolicy.go#L381-L394

According to the documentation, VerifyPeerCertificate is checked in addition to any other default verification:

VerifyPeerCertificate, if not nil, is called after normal certificate verification by either a TLS client or server. It receives the raw ASN.1 certificates provided by the peer and also any verified chains that normal processing found. If it returns a non-nil error, the handshake is aborted and that error results. If normal verification fails then the handshake will abort before considering this callback. If normal verification is disabled by setting InsecureSkipVerify, or (for a server) when ClientAuth is RequestClientCert or RequireAnyClientCert, then this callback will be considered but the verifiedChains argument will always be nil.

So if I understand correctly, when setting only trusted_leaf_cert_file but neither specifying trusted_ca_cert_file nor mode require, it is impossible to use any certificate to connect to the server.