caddy: Since 1.0.0 tls certificate selection does not work as expected

1. Which version of Caddy are you using (caddy -version)?

v1.0.0 compiled from source

2. What are you trying to do?

I am serving sites using tls directive, that for each site there is separate certificate.

Certificates are generated like:

openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout 02.pem -out 02.pem -subj '/CN=*.example.com/O=02/'
openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout 01.pem -out 01.pem -subj '/CN=*.example.com/O=01/'

3. What is your Caddyfile?

https://01.example.com:6443 {
  bind 127.0.0.1
  tls 01.pem 01.pem
}  
https://02.example.com:6443 {
  bind 127.0.0.1
  tls 02.pem 02.pem
}

4. How did you run Caddy (give the full command and describe the execution environment)?

./caddy -log stdout -pidfile caddy.pid

5. Please paste any relevant HTTP request(s) here.

curl -v -o /dev/null -k --resolve 01.example.com:6443:127.0.0.1 https://01.example.com:6443 2>&1 | grep subject
*       subject: O=01,CN=*.example.com
curl -v -o /dev/null -k --resolve 02.example.com:6443:127.0.0.1 https://02.example.com:6443 2>&1 | grep subject
*       subject: O=01,CN=*.example.com

6. What did you expect to see?

curl -v -o /dev/null -k --resolve 01.example.com:6443:127.0.0.1 https://01.example.com:6443 2>&1 | grep subject
*       subject: O=01,CN=*.example.com
curl -v -o /dev/null -k --resolve 02.example.com:6443:127.0.0.1 https://02.example.com:6443 2>&1 | grep subject
*       subject: O=02,CN=*.example.com

7. What did you see instead (give full error messages and/or log)?

curl -v -o /dev/null -k --resolve 01.example.com:6443:127.0.0.1 https://01.example.com:6443 2>&1 | grep subject
*       subject: O=01,CN=*.example.com
curl -v -o /dev/null -k --resolve 02.example.com:6443:127.0.0.1 https://02.example.com:6443 2>&1 | grep subject
*       subject: O=01,CN=*.example.com

Caddy log on startup:

2019/04/26 09:05:24 [INFO][cache:0xc000032280] Started certificate maintenance routine
2019/04/26 09:05:24 [WARNING] Stapling OCSP: no OCSP stapling for [*.example.com]: no OCSP server specified in certificate
2019/04/26 09:05:24 [INFO] Successfully loaded TLS assets from 01.pem and 01.pem
2019/04/26 09:05:24 [WARNING] Stapling OCSP: no OCSP stapling for [*.example.com]: no OCSP server specified in certificate
2019/04/26 09:05:24 [INFO] Successfully loaded TLS assets from 02.pem and 02.pem
Activating privacy features... done.

Serving HTTPS on port 6443
https://01.example.com:6443 (only accessible on this machine)
https://02.example.com:6443 (only accessible on this machine)

2019/04/26 09:05:24 [INFO] Serving https://01.example.com:6443 (only accessible on this machine)
2019/04/26 09:05:24 [INFO] Serving https://02.example.com:6443 (only accessible on this machine)
2019/04/26 09:05:35 [WARNING] Sending telemetry (attempt 1): telemetry server returned status code 403 - backing off and retrying
2019/04/26 09:05:45 [NOTICE] Sending telemetry: we were too early; waiting 1h1m16.266725991s before trying again

8. Why is this a bug, and how do you think this should be fixed?

As each site has different certificate, I would expect it to be served with different certificates, it was working pre 1.x.x

9. What are you doing to work around the problem in the meantime?

Downgrade to before 1.0.0-beta1, which does not include certmagic rewrite

10. Please link to any related issues, pull requests, and/or discussion.

I think this can be related to https://github.com/mholt/caddy/pull/2571

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 3
  • Comments: 20 (12 by maintainers)

Commits related to this issue

Most upvoted comments

The v2 Caddyfile now implements this behavior implicitly; certificates specified manually will be tagged so that the specific one you provide will be used to serve that hostname, even if multiple certs match. At least, that’s the hope. Please test it in your dev/staging environments and confirm!

I think we can get this to work (mostly) implicitly with the Caddyfile adapter, I’ll just need more time to figure that out. I’m not sure yet how/if this would work in Caddy 1, but it definitely will in Caddy Enterprise.

Ok, thank you, that gives me good info.

As it means that it’s unsure if it’d land in Caddy 1, I will then accept current Caddy 1 state. At least we know the situation and the limits, and we will know that we can obtain this functionality with Caddy 2 or Caddy Enterprise.

I see. So, yes, this is a weird edge case… here is what happened.

I expect that selecting certificate with tls cert.pem cert.pem would enforce using exact certificate, and that no more automatic handling happens.

When you specify a certificate manually like this, it is loaded into memory, into the “certificate cache” we call it. Certificates loaded manually aren’t handled automatically, as you suggest. But all certificates – both manual and managed – are loaded into the same cache. The only difference is a little flag that indicates whether they are manually or automatically managed. Certificates are simply placed there in the cache (regardless of where they came from) and indexed by their SANs for recall upon a TLS handshake where the SNI matches one of those SANs.

What happened is that a recent refactoring of CertMagic actually fixed a bug where certificate configurations were static, even if the underlying configuration changed from the time that it was loaded or obtained and the time that it was reloaded or renewed (about 60 days, or longer). This was due to certificate configs being designed to be short-lived, not long-lived, yet attaching them to certificates (which are long-lived) made them “leak” in memory in the sense that even if the config associated with a certificate was changed, it wouldn’t be replaced on the certificate.

That is all fixed now, but I suppose the reason this is affecting you is because we removed the buggy mapping between configs and certificates, but in doing so we just use the first matching certificate by SAN, since there’s no config mapping to require that a specific certificate for a given SAN is used; rather it just uses the first one that matches the SAN.

I will see about improving on this in Caddy 1 for a fix, as well as ensuring that Caddy 2 has this capability as well.