caddy: ERR_SSL_PROTOCOL_ERROR after few days of running smoothly

We have a configuration which consists of

Hardcoded SSL certificates for *.example.com
OnDemand TLS for all other sites

{
  email support@example.com
  on_demand_tls {
    ask https://api.example.com/domain-allowed
  }
}

https://*.example.com {
  tls /etc/caddy-ssl/tls.crt /etc/caddy-ssl/tls.key
  rewrite * {path}?{query}
  reverse_proxy https://page.example.com
}

https://example.com {
  tls /etc/caddy-ssl/tls.crt /etc/caddy-ssl/tls.key
  rewrite * {path}?{query}
  reverse_proxy https://page.example.com
}

https:// {
  tls {
    on_demand
  }
  rewrite * {path}?{query}
  reverse_proxy https://page.example.com
  log {
    level WARN
  }
}

We run it on 2 servers accessing a shared network volume via NFS. Everything runs smoothly but after a few days (lasted for 4 days now) we encounter a very weird behaviour. Chrome then only responds with ERR_SSL_PROTOCOL_ERROR after accessing the site with the hardcoded SSL certificate. The OnDemand TLS sites continue working without issues.

cURL verbose log:

% curl -H "Host: example.com" https://example.com -v
*   Trying 116.203.253.235...
* TCP_NODELAY set
* Connected to example.com (116.203.253.200) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS alert, internal error (592):
* error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error
* Closing connection 0
curl: (35) error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error

When I restart caddy afterwards, everything is back normal again.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 24 (14 by maintainers)

Most upvoted comments

Hi @mholt just got some logs for you:

{"log":"2021/09/24 21:45:55.643\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.cache\u0009cache full; evicting random certificate\u0009{\"removing_subjects\": [\"*.funnelcockpit.com\", \"funnelcockpit.com\"], \"removing_hash\": \"ad90470dc6f7de69aead289244a2943111627b170fd0b0ff39eb1cb01f434b48\", \"inserting_subjects\": [\"www.hochpreisvertriebler.de\"], \"inserting_hash\": \"d1b41385f61a6d9096e3ce6c3d65decd9037574c22228c2cca4d939a972126be\"}\r\n","stream":"stdout","time":"2021-09-24T21:45:55.64394605Z"}
{"log":"2021/09/24 21:45:55.643\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.cache\u0009removed certificate from cache\u0009{\"subjects\": [\"*.funnelcockpit.com\", \"funnelcockpit.com\"], \"expiration\": \"2022/09/27 23:59:59.000\", \"managed\": false, \"issuer_key\": \"\", \"hash\": \"ad90470dc6f7de69aead289244a2943111627b170fd0b0ff39eb1cb01f434b48\"}\r\n","stream":"stdout","time":"2021-09-24T21:45:55.643981676Z"}
{"log":"2021/09/24 21:46:39.289\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.handshake\u0009no matching certificate; will choose from all certificates\u0009{\"identifier\": \"funnelcockpit.com\"}\r\n","stream":"stdout","time":"2021-09-24T21:46:39.290172349Z"}
{"log":"2021/09/24 21:46:39.296\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.handshake\u0009choosing certificate\u0009{\"identifier\": \"funnelcockpit.com\", \"num_choices\": 9999}\r\n","stream":"stdout","time":"2021-09-24T21:46:39.296448641Z"}
{"log":"2021/09/24 21:46:39.298\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.handshake\u0009custom certificate selection results\u0009{\"error\": \"no certificates matched custom selection policy\", \"identifier\": \"funnelcockpit.com\", \"subjects\": [], \"managed\": false, \"issuer_key\": \"\", \"hash\": \"\"}\r\n","stream":"stdout","time":"2021-09-24T21:46:39.298942268Z"}
{"log":"2021/09/24 21:46:48.832\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.handshake\u0009no matching certificate; will choose from all certificates\u0009{\"identifier\": \"funnelcockpit.com\"}\r\n","stream":"stdout","time":"2021-09-24T21:46:48.833097147Z"}
{"log":"2021/09/24 21:46:48.836\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.handshake\u0009choosing certificate\u0009{\"identifier\": \"funnelcockpit.com\", \"num_choices\": 9999}\r\n","stream":"stdout","time":"2021-09-24T21:46:48.83708281Z"}
{"log":"2021/09/24 21:46:48.838\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.handshake\u0009custom certificate selection results\u0009{\"error\": \"no certificates matched custom selection policy\", \"identifier\": \"funnelcockpit.com\", \"subjects\": [], \"managed\": false, \"issuer_key\": \"\", \"hash\": \"\"}\r\n","stream":"stdout","time":"2021-09-24T21:46:48.839090369Z"}
{"log":"2021/09/24 21:46:50.622\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.handshake\u0009no matching certificate; will choose from all certificates\u0009{\"identifier\": \"funnelcockpit.com\"}\r\n","stream":"stdout","time":"2021-09-24T21:46:50.622825347Z"}
{"log":"2021/09/24 21:46:50.625\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.handshake\u0009choosing certificate\u0009{\"identifier\": \"funnelcockpit.com\", \"num_choices\": 9999}\r\n","stream":"stdout","time":"2021-09-24T21:46:50.626194709Z"}
{"log":"2021/09/24 21:46:50.627\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.handshake\u0009custom certificate selection results\u0009{\"error\": \"no certificates matched custom selection policy\", \"identifier\": \"funnelcockpit.com\", \"subjects\": [], \"managed\": false, \"issuer_key\": \"\", \"hash\": \"\"}\r\n","stream":"stdout","time":"2021-09-24T21:46:50.627920934Z"}
{"log":"2021/09/24 21:46:51.171\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.handshake\u0009no matching certificate; will choose from all certificates\u0009{\"identifier\": \"funnelcockpit.com\"}\r\n","stream":"stdout","time":"2021-09-24T21:46:51.17193989Z"}
{"log":"2021/09/24 21:46:51.174\u0009\u001b[35mDEBUG\u001b[0m\u0009tls.handshake\u0009choosing certificate\u0009{\"identifier\": \"funnelcockpit.com\", \"num_choices\": 9999}\r\n","stream":"stdout","time":"2021-09-24T21:46:51.174442915Z"}

They look same to me like the ones from the forum. So probably resolved by your new fix 👍

gbhrdt on Sep 24, 2021

@gbhrdt Thanks for the update. I’m pretty sure this is finally patched here: https://github.com/caddyserver/certmagic/commit/88b8609b4d8fb8c7b1865322602606b85240c80d

I might still refine this but it should be workable now. (For example, this won’t help the case of manually-managed certificates, only automatic, managed certs.)

mholt on Sep 22, 2021

I have an updated working theory as to what’s happening: https://caddy.community/t/error-tls-alert-internal-error-592-again/13272/29?u=matt

Your full logs can confirm this. But I probably won’t wait too long for them before patching. (Hopefully I’m right!)

mholt on Sep 21, 2021

@mholt Alright, global debug logs are enabled now. Let’s wait for some interesting logs 👍

gbhrdt on Sep 3, 2021

That’s not the same error @iandk.

Please open a topic on the forums, and please fill out the help topic template. https://caddy.community

francislavoie on Sep 1, 2021

@mholt Alright, thanks for your very quick replies, I’ll build the image myself, upgrade and keep you updated on the debug logs!

gbhrdt on Aug 31, 2021