caddy: "Context deadline exceeded" when creating certificate

Not looking to revive this, just confirming that this is also happening for me when I had a bulk import of a few thousand domains over a couple of days.

I think ZeroSSL just has some performance issues (maybe only from time to time, unsure as of yet).

Nothing I can think of. Thanks for reporting it to them. I’m not sure we can really do anything/much about it… open to ideas though.

That’s already an absurdly long amount of time to hold a TLS handshake open, and I’ve even doubled it to 180 on tip

100% agree. Also the issue here is the huge amplitud of issuance time.

*I’ve already contacted ZeroSSL support, let’s see what they have to say 🤞

EDIT: I’m still waiting for their reply, hope it arrives soon EDIT 2: They are now actively looking into this issue 🙌 🤞

Thanks for the extra info Matt. We stand available in case there’s something we can do to gather more info.

Hi guys! This is happening again and now we have the logs 💪 We’re running v2.4.2 h1:chB106RlsIaY4mVEyq9OQM5g/9lHYVputo/LAX2ndFg=, and this is our Caddy config:

{
    # This enables `debug` log
    debug
    on_demand_tls {
        ask https://ourapp-https-guard.com/caddy-check-blablablabla
    }

    storage dynamodb caddy_ssl_certificates
        storage_clean_interval 32d
}

:80 {
    respond /health "Im healthy!" 200
}

:443 {
   tls {
       issuer zerossl blablablabla {
           email my@email.com
           dir https://acme.zerossl.com/v2/DV90
           eab blablablabla blablablablablablablablablablablablablablablabla
       }
       issuer acme {
         email my@email.com
       }
       on_demand
   }

    reverse_proxy * https://ourapp.com {
        header_up Host ourapp.com
        header_up X-Forwarded-Host {host}
        header_up X-Forwarded-For {remote_host}
        header_up X-Forwarded-Port {server_port}
        header_up X-Forwarded-Proto {scheme}
        health_timeout 5s
    }
}

And these are the logs with the debug level enabled:

Oct 25 19:50:45 custom-domains-staging-br-01 systemd[1]: Reloading Caddy.
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[1088051]: {"level":"info","ts":1635191445.7659783,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[1088051]: 2021/10/25 19:50:45 [WARNING] Unnecessary header_up ('X-Forwarded-Proto' field): the reverse proxy's default behavior is to pass headers to the upstream
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[1088051]: 2021/10/25 19:50:45 [WARNING] Unnecessary header_up ('X-Forwarded-Proto' field): the reverse proxy's default behavior is to pass headers to the upstream
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[1088051]: {"level":"warn","ts":1635191445.7697878,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191445.772287,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:51494","headers":{"Accept-Encoding":["gzip"],"Content-Length":["1824"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191445.7734485,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191445.7746146,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00011ac40"}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191445.774674,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191445.774871,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191445.7748866,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191445.776187,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191445.7762077,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191445.7763228,"logger":"tls","msg":"cleaning storage unit","description":"&{caddy_ssl_certificates <nil>   false 0 0}"}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191445.845681,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000120770"}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191445.8468363,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191445.8468578,"logger":"admin.api","msg":"load complete"}
Oct 25 19:50:45 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191445.8489323,"logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
Oct 25 19:50:45 custom-domains-staging-br-01 systemd[1]: Reloaded Caddy.
Oct 25 19:50:50 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191450.55012,"logger":"tls","msg":"loading managed certificate","domain":"staging-farfalla.publica.la","expiration":1641599999,"issuer_key":"acme.zerossl.com-v2-DV90","storage":{"table":"caddy_ssl_certificates","lock_timeout":300000000000,"lock_polling_interval":5000000000}}
Oct 25 19:50:59 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191459.7764788,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"demofgiliotest20211025.publicala.me"}
Oct 25 19:51:00 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191460.7144182,"logger":"tls.obtain","msg":"acquiring lock","identifier":"demofgiliotest20211025.publicala.me"}
Oct 25 19:51:00 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191460.95853,"logger":"tls.obtain","msg":"lock acquired","identifier":"demofgiliotest20211025.publicala.me"}
Oct 25 19:51:01 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191461.434814,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme.zerossl.com-v2-DV90"}
Oct 25 19:51:01 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191461.6692853,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["demofgiliotest20211025.publicala.me"],"ca":"https://acme.zerossl.com/v2/DV90","account":"fgilio@publica.la"}
Oct 25 19:51:01 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191461.669537,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["demofgiliotest20211025.publicala.me"],"ca":"https://acme.zerossl.com/v2/DV90","account":"fgilio@publica.la"}
Oct 25 19:51:02 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191462.573778,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Type":["application/octet-stream"],"Date":["Mon, 25 Oct 2021 19:51:02 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["YDoTPRlh1-qHnmy21iTQfw2JtAHE71cZH4XHehfPk7Y"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:02 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191462.8212652,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["297"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:02 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/Nc8u6X2dsf4i12EFZxO7zQ"],"Replay-Nonce":["qdBDJ32c92XMn4iVk3y2VPHLTCLgwXN4UdXzBGufa60"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:03 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191463.024055,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["465"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:02 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["PigsaXE4kBUg-kaB6_NF5Ny7WoAXCVRIlT3QZAkxPJw"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:03 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191463.0242057,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
Oct 25 19:51:03 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191463.0242183,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"demofgiliotest20211025.publicala.me","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Oct 25 19:51:03 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191463.3946435,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/chall/z3Cw6JJZYHbrKiTBXEOjmA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["164"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:03 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\"","<https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg>;rel=\"up\""],"Replay-Nonce":["xaOOee_pwzgE3JT1USy2RvJirn_YfoAPnL2I4LrCO5E"],"Retry-After":["10"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:03 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191463.394733,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"demofgiliotest20211025.publicala.me","challenge_type":"http-01"}
Oct 25 19:51:03 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191463.9322329,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["468"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:03 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["UCpkKlUG8uEy-ZMz-oLaLp1GK3Xf8_cW44NOTDlFaG0"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:09 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191469.1185324,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["468"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:09 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["N3zMKUBCwDK-3ssj-9kVbr88JVDx8UzIPIxomqw_uZ8"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:14 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191474.3100483,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["468"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:14 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["iW_r_ZTntJXw50Z3JFGjjTCD00v3HVHGtSVEZh5jNFI"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:19 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191479.5178995,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["468"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:19 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["EIwWXh3JTlgqK8Gn2gcYS5bApJVo0DdJeILerZkEKOY"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:24 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191484.6796386,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["468"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:24 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["AwrHbfeB_kVkB8BzaGojzcIxQLh-vYno_ggAVNGXt8M"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:29 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191489.852036,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["468"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:29 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["VfEeBU8t0aOqQBHXrb0-3XmYupAO6_mGFxCc2oNu-T4"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:35 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191495.0641243,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["468"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:34 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["kEfmDi9PxaNLjvMXdX_WpAFfA2jPHAnuo19FPAdWDIY"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:40 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191500.244451,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["468"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:40 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["TX2H8SKfcOjkYl02_8jknWlGHO9JbXSFp84MXqVfLiE"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:45 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191505.4565187,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["468"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:45 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["u30DLaMqPG0E8oTx5_mPWWuVVc3c-VBaRwgT-QBkHME"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:50 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191510.6674209,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/11CxCt3R_tFKswKXaK_VWg","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["335"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:50 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["gnGzVGQ7SBQJxXtVfW4vCusBsvAxbwfrOR7lpwSQ4C0"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:51:50 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191510.7889047,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.zerossl.com/v2/DV90/order/Nc8u6X2dsf4i12EFZxO7zQ"}
Oct 25 19:51:51 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191511.1681097,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/order/Nc8u6X2dsf4i12EFZxO7zQ/finalize","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:51:51 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/Nc8u6X2dsf4i12EFZxO7zQ"],"Replay-Nonce":["ibglOVN7I9VI-4SgQo0MkXky56_ccsJ02QqbQ0ZdW9g"],"Retry-After":["15"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:52:06 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191526.4611456,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/order/Nc8u6X2dsf4i12EFZxO7zQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:52:06 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/Nc8u6X2dsf4i12EFZxO7zQ"],"Replay-Nonce":["V_7jyio-vb3FVdV4z2wlYzB2IoNIEc_cN810OtbvgdY"],"Retry-After":["15"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:52:21 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191541.714607,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/order/Nc8u6X2dsf4i12EFZxO7zQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:52:21 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/Nc8u6X2dsf4i12EFZxO7zQ"],"Replay-Nonce":["sUozd0B4QXXAtVuC0A5ehwBZfcK6ExteVZhL46iDGvk"],"Retry-After":["15"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:52:29 custom-domains-staging-br-01 caddy[326595]: {"level":"error","ts":1635191549.7770689,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"demofgiliotest20211025.publicala.me","issuer":"acme.zerossl.com-v2-DV90","error":"[demofgiliotest20211025.publicala.me] finalizing order https://acme.zerossl.com/v2/DV90/order/Nc8u6X2dsf4i12EFZxO7zQ: context deadline exceeded (ca=https://acme.zerossl.com/v2/DV90)"}
Oct 25 19:52:29 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191549.7771528,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
Oct 25 19:52:30 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191550.014238,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["demofgiliotest20211025.publicala.me"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"fgilio@publica.la"}
Oct 25 19:52:30 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191550.0142884,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["demofgiliotest20211025.publicala.me"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"fgilio@publica.la"}
Oct 25 19:52:30 custom-domains-staging-br-01 caddy[326595]: {"level":"warn","ts":1635191550.0143466,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme-v02.api.letsencrypt.org/directory","error":"performing request: Get \"https://acme-v02.api.letsencrypt.org/directory\": context deadline exceeded"}
Oct 25 19:52:30 custom-domains-staging-br-01 caddy[326595]: {"level":"error","ts":1635191550.0143867,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"demofgiliotest20211025.publicala.me","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[demofgiliotest20211025.publicala.me] creating new order: provisioning client: context deadline exceeded (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Oct 25 19:52:30 custom-domains-staging-br-01 caddy[326595]: {"level":"error","ts":1635191550.0144014,"logger":"tls.obtain","msg":"will retry","error":"[demofgiliotest20211025.publicala.me] Obtain: [demofgiliotest20211025.publicala.me] creating new order: provisioning client: context deadline exceeded (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":89.055843708,"max_duration":2592000}
Oct 25 19:52:30 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191550.014412,"logger":"tls.obtain","msg":"releasing lock","identifier":"demofgiliotest20211025.publicala.me"}
Oct 25 19:52:30 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191550.1329362,"logger":"http.stdlib","msg":"http: TLS handshake error from 190.245.59.70:55122: context canceled"}
Oct 25 19:53:06 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191586.7669406,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"demofgiliotest20211025.publicala.me"}
Oct 25 19:53:07 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191587.6330338,"logger":"tls.obtain","msg":"acquiring lock","identifier":"demofgiliotest20211025.publicala.me"}
Oct 25 19:53:07 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191587.871315,"logger":"tls.obtain","msg":"lock acquired","identifier":"demofgiliotest20211025.publicala.me"}
Oct 25 19:53:08 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191588.3419158,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme.zerossl.com-v2-DV90"}
Oct 25 19:53:08 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191588.5800972,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["demofgiliotest20211025.publicala.me"],"ca":"https://acme.zerossl.com/v2/DV90","account":"fgilio@publica.la"}
Oct 25 19:53:08 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191588.5812044,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["demofgiliotest20211025.publicala.me"],"ca":"https://acme.zerossl.com/v2/DV90","account":"fgilio@publica.la"}
Oct 25 19:53:09 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191589.1687744,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Type":["application/octet-stream"],"Date":["Mon, 25 Oct 2021 19:53:09 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["0dh8iykLRJEgdsog2XLslsiYq9drCPoJMiSsrYQ5kus"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:53:09 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191589.4605386,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["297"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:53:09 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA"],"Replay-Nonce":["4OIO4OAZKeTYOVgNJ3iSr6k1V9cygtbghKA51AFEMDY"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:53:09 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191589.689198,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/sC26awn34WxJOT0K8KH0cQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["465"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:53:09 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["chu11uxUzb3hc23EtDu8cK1brtOuoCQlfTAurdW2B2g"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:53:09 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191589.6893125,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
Oct 25 19:53:09 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191589.6893222,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"demofgiliotest20211025.publicala.me","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Oct 25 19:53:10 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191590.0838847,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/chall/4ThSVn5l_RTsZehqxbIFtA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["164"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:53:10 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\"","<https://acme.zerossl.com/v2/DV90/authz/sC26awn34WxJOT0K8KH0cQ>;rel=\"up\""],"Replay-Nonce":["SOMosWVXlBlfaliviulBbQDDcWUhqXvktLK3SKxFqKg"],"Retry-After":["10"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:53:10 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191590.0842097,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"demofgiliotest20211025.publicala.me","challenge_type":"http-01"}
Oct 25 19:53:10 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191590.6076076,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/sC26awn34WxJOT0K8KH0cQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["335"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:53:10 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["Qjl5WjHBK1nRfekMs74U1XMfmG3NRtXS0DTGZnn7cKA"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:53:10 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191590.72736,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA"}
Oct 25 19:53:11 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191591.1284513,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA/finalize","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:53:11 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA"],"Replay-Nonce":["vWl4qMBEOyq_WD6QQmwESuaep6PBO-WMj75erB6rAlE"],"Retry-After":["15"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:53:26 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191606.3751113,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:53:26 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA"],"Replay-Nonce":["ff17Uyd9N4mLvKCP_weztBWv6NgsrCNBUjVvtf9aSm4"],"Retry-After":["15"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:53:41 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191621.5647938,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:53:41 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA"],"Replay-Nonce":["VbYdCdeqSQlyJ1Hv_v54mh5DaFU16bVP_r4D6FEfLNE"],"Retry-After":["15"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:53:56 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191636.89529,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:53:56 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA"],"Replay-Nonce":["yFP7uOa_II2AP_bKIL4rjQUiCOOEObCmPCDWDIzAas0"],"Retry-After":["15"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:54:12 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191652.0751743,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:54:12 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA"],"Replay-Nonce":["zA9RLK9H4Zds2K_a4rWJTu7EgdGwssDJmD6ZaUpcKWM"],"Retry-After":["15"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:54:27 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191667.2728748,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:54:27 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA"],"Replay-Nonce":["m8iQffKLQR0V3xr9QapqBqwThpZ0IiHXgqoT2GK9oOg"],"Retry-After":["15"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:54:36 custom-domains-staging-br-01 caddy[326595]: {"level":"error","ts":1635191676.7675414,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"demofgiliotest20211025.publicala.me","issuer":"acme.zerossl.com-v2-DV90","error":"[demofgiliotest20211025.publicala.me] finalizing order https://acme.zerossl.com/v2/DV90/order/q1DjDnv8t0SndgnF9XbLJA: context deadline exceeded (ca=https://acme.zerossl.com/v2/DV90)"}
Oct 25 19:54:36 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191676.7681208,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
Oct 25 19:54:37 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191677.0064538,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["demofgiliotest20211025.publicala.me"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"fgilio@publica.la"}
Oct 25 19:54:37 custom-domains-staging-br-01 caddy[326595]: {"level":"error","ts":1635191677.006508,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"demofgiliotest20211025.publicala.me","issuer":"acme-v02.api.letsencrypt.org-directory","error":"context canceled"}
Oct 25 19:54:37 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191677.006521,"logger":"tls.obtain","msg":"releasing lock","identifier":"demofgiliotest20211025.publicala.me"}
Oct 25 19:54:37 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191677.1253417,"logger":"http.stdlib","msg":"http: TLS handshake error from 190.245.59.70:55130: [demofgiliotest20211025.publicala.me] Obtain: context canceled"}
Oct 25 19:54:37 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191677.5960405,"logger":"http.stdlib","msg":"http: TLS handshake error from 190.245.59.70:55137: no certificate available for 'demofgiliotest20211025.publicala.me'"}
Oct 25 19:55:05 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191705.225738,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"demofgiliotest20211025.publicala.me"}
Oct 25 19:55:06 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191706.053277,"logger":"tls.obtain","msg":"acquiring lock","identifier":"demofgiliotest20211025.publicala.me"}
Oct 25 19:56:07 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191767.717552,"logger":"tls.obtain","msg":"lock acquired","identifier":"demofgiliotest20211025.publicala.me"}
Oct 25 19:56:08 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191768.187909,"logger":"tls.obtain","msg":"trying issuer 1/2","issuer":"acme.zerossl.com-v2-DV90"}
Oct 25 19:56:08 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191768.4232187,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["demofgiliotest20211025.publicala.me"],"ca":"https://acme.zerossl.com/v2/DV90","account":"fgilio@publica.la"}
Oct 25 19:56:08 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191768.423264,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["demofgiliotest20211025.publicala.me"],"ca":"https://acme.zerossl.com/v2/DV90","account":"fgilio@publica.la"}
Oct 25 19:56:09 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191769.081165,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme.zerossl.com/v2/DV90/newNonce","headers":{"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Type":["application/octet-stream"],"Date":["Mon, 25 Oct 2021 19:56:09 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["aUBEL5uDebF2_j8V7OWauOgISt025bl8xiknaomxmzk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:56:09 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191769.2536356,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/newOrder","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=0, no-cache, no-store","max-age=-1"],"Content-Length":["297"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:56:09 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/HR0WouyxpXxQa5CmqaqllQ"],"Replay-Nonce":["x3XpWbLQ7HVpUjhIe4uAAqY2QPdDGZCw9qXCuovoJAY"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:56:09 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191769.3988042,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/m-cOpAQ_GEtU7gfFHJq1Hw","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["465"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:56:09 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["qeKkPlh2Gt5sMok6so3Ik4S_j1Fcob6zPvCSuIB6xfE"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:56:09 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191769.3989742,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
Oct 25 19:56:09 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191769.398987,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"demofgiliotest20211025.publicala.me","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Oct 25 19:56:09 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191769.7203524,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/chall/0a-lCl-MMqXOXoZLmYcRgA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["164"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:56:09 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\"","<https://acme.zerossl.com/v2/DV90/authz/m-cOpAQ_GEtU7gfFHJq1Hw>;rel=\"up\""],"Replay-Nonce":["nWiya1skWh5cU3ErXOHv_FSEx-oFq19mc8KnNBXpRUo"],"Retry-After":["10"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:56:09 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191769.7204297,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"demofgiliotest20211025.publicala.me","challenge_type":"http-01"}
Oct 25 19:56:10 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191770.1756642,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/m-cOpAQ_GEtU7gfFHJq1Hw","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["335"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:56:10 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["vc83s2uVDcb6OYR1I_ZuPK-iorN9uRQ7QoDqPukgwqo"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:56:10 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191770.2961998,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.zerossl.com/v2/DV90/order/HR0WouyxpXxQa5CmqaqllQ"}
Oct 25 19:56:10 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191770.5669835,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/order/HR0WouyxpXxQa5CmqaqllQ/finalize","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:56:10 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/HR0WouyxpXxQa5CmqaqllQ"],"Replay-Nonce":["iQybeRTqNuUGcLD_YyYlEklJ9aVDXUaa0wXlSkK_Ys4"],"Retry-After":["15"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
^[Oct 25 19:56:25 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191785.7490797,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/order/HR0WouyxpXxQa5CmqaqllQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.2 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["300"],"Content-Type":["application/json"],"Date":["Mon, 25 Oct 2021 19:56:25 GMT"],"Location":["https://acme.zerossl.com/v2/DV90/order/HR0WouyxpXxQa5CmqaqllQ"],"Replay-Nonce":["iSFMy-lDgwM3rH-lExumF6azYevHrzst8tfvzWvIBcI"],"Retry-After":["15"],"Server":["nginx"],"Status":[""],"Strict-Transport-Security":["max-age=15552000"]}}
Oct 25 19:56:35 custom-domains-staging-br-01 caddy[326595]: {"level":"error","ts":1635191795.2265968,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"demofgiliotest20211025.publicala.me","issuer":"acme.zerossl.com-v2-DV90","error":"[demofgiliotest20211025.publicala.me] finalizing order https://acme.zerossl.com/v2/DV90/order/HR0WouyxpXxQa5CmqaqllQ: context deadline exceeded (ca=https://acme.zerossl.com/v2/DV90)"}
Oct 25 19:56:35 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191795.227279,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme-v02.api.letsencrypt.org-directory"}
Oct 25 19:56:35 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191795.4636214,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["demofgiliotest20211025.publicala.me"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"fgilio@publica.la"}
Oct 25 19:56:35 custom-domains-staging-br-01 caddy[326595]: {"level":"error","ts":1635191795.4636722,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"demofgiliotest20211025.publicala.me","issuer":"acme-v02.api.letsencrypt.org-directory","error":"context canceled"}
Oct 25 19:56:35 custom-domains-staging-br-01 caddy[326595]: {"level":"info","ts":1635191795.4636865,"logger":"tls.obtain","msg":"releasing lock","identifier":"demofgiliotest20211025.publicala.me"}
Oct 25 19:56:35 custom-domains-staging-br-01 caddy[326595]: {"level":"debug","ts":1635191795.583506,"logger":"http.stdlib","msg":"http: TLS handshake error from 190.245.59.70:55143: [demofgiliotest20211025.publicala.me] Obtain: context canceled"}

I hope this helps, please let me know if you need more information.

Hi @francislavoie! No updates, this has not happened again. We’ll reopen and update in case it does happen again, and we’ll use the debug flag straight away.

Hi @mholt!

When you enable debug logs, what do they show?

Never did it. According to the docs it’s just a matter of adding debug to the caddyfile at the same level as the on_demand_tls or storage directives. I’m going to try this and let you know.

UPDATE: I tested the debug flag and can now see a ton more info. But the issue is no longer happening right now, I’ll keep this in mind for the next time something weird happens with our Caddy instances. I’m leaving the issue open in case the questions below can point us in any direction.

Edit: Also, it’s weird that it happens with both ZeroSSL and Let’s Encrypt, very different providers.

Yes, it’s very strange.

How many operations are taking place at the time?

What do you mean by “operations”? If it’s only certificates generation, then it’s usually about 1 to 4 per hour.

How are we sure that it’s not a hiccup in your network? Just trying to narrow down possibilities.

Good question, I’d love to give you a solid answer but I’ve no idea. These servers are 100% dedicated to Caddy, nothing else is running on them. And both servers usually stay below 10% CPU usage.