caddy: Reverse Proxy Working on 2.0 No Longer Working on 2.1.1 - TLS Auth Issues?
1. Environment
1a. Operating system and version
Ubuntu 20.04
1b. Caddy version (run caddy version or paste commit SHA)
v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=
1c. Go version (if building Caddy from source; run go version)
N/A
2. Description
2a. What happens (briefly explain what is wrong)
A Caddyfile setting up a reverse proxy for five different services was previously working with 2.0, and is now no longer functional with 2.1.1. The reverse-proxied addresses no longer load, and there appears to be some sort of SSL-related error on the Caddy side.
Text of Caddyfile (actual subdomains used are different, but those listed here are consistent with the logs below, which were adjusted to match):
service1.emperialservices.com {
reverse_proxy localhost:32400
}
service2.emperialservices.com {
reverse_proxy localhost:8080
}
service3.emperialservices.com {
reverse_proxy localhost:8081
}
service4.emperialservices.com {
reverse_proxy localhost:9090
}
service5.emperialservices.com {
reverse_proxy 10.0.1.4:9091
}
2b. Why it’s a bug (if it’s not obvious)
The file worked previously, has not changed at all, and now no longer works. Likewise, network configuration and port forwarding has not changed—HTTP and HTTPS ports all forward to the machine running Caddy.
2c. Log output
Here’s the output just from trying to run Caddy while specifying the Caddyfile—about 3 minutes of elapsed time, with repeat TLS auth errors.
sudo caddy run --config /mnt/services/settings/caddy/Caddyfile
2020/07/19 18:33:47.137 INFO using provided configuration {"config_file": "/mnt/services/settings/caddy/Caddyfile", "config_adapter": ""}
2020/07/19 18:33:47.138 INFO admin admin endpoint started {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2020/07/19 18:33:47.139 INFO http server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2020/07/19 18:33:47.139 INFO http enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2020/07/19 13:33:47 [INFO][cache:0xc0001869c0] Started certificate maintenance routine
2020/07/19 18:33:47.140 INFO tls cleaned up storage units
2020/07/19 18:33:47.140 INFO http enabling automatic TLS certificate management {"domains": ["service5.emperialservices.com", "service4.emperialservices.com", "service2.emperialservices.com", "service3.emperialservices.com", "service1.emperialservices.com"]}
2020/07/19 18:33:47.141 INFO autosaved config {"file": "/root/.config/caddy/autosave.json"}
2020/07/19 18:33:47.141 INFO serving initial configuration
2020/07/19 13:33:47 [INFO][service4.emperialservices.com] Obtain certificate; acquiring lock...
2020/07/19 13:33:47 [INFO][service1.emperialservices.com] Obtain certificate; acquiring lock...
2020/07/19 13:33:47 [INFO][service4.emperialservices.com] Obtain: Lock acquired; proceeding...
2020/07/19 13:33:47 [INFO][service1.emperialservices.com] Obtain: Lock acquired; proceeding...
2020/07/19 13:33:47 [INFO][service2.emperialservices.com] Obtain certificate; acquiring lock...
2020/07/19 13:33:47 [INFO][service2.emperialservices.com] Obtain: Lock acquired; proceeding...
2020/07/19 13:33:47 [INFO][service3.emperialservices.com] Obtain certificate; acquiring lock...
2020/07/19 13:33:47 [INFO][service3.emperialservices.com] Obtain: Lock acquired; proceeding...
2020/07/19 13:33:47 [INFO][service5.emperialservices.com] Obtain certificate; acquiring lock...
2020/07/19 13:33:47 [INFO][service5.emperialservices.com] Obtain: Lock acquired; proceeding...
2020/07/19 13:33:47 [INFO][service4.emperialservices.com] Waiting on rate limiter...
2020/07/19 13:33:47 [INFO][service4.emperialservices.com] Done waiting
2020/07/19 13:33:47 [INFO] [service4.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:33:47 [INFO][service1.emperialservices.com] Waiting on rate limiter...
2020/07/19 13:33:47 [INFO][service1.emperialservices.com] Done waiting
2020/07/19 13:33:47 [INFO] [service1.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:33:47 [INFO][service2.emperialservices.com] Waiting on rate limiter...
2020/07/19 13:33:47 [INFO][service2.emperialservices.com] Done waiting
2020/07/19 13:33:47 [INFO] [service2.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:33:47 [INFO][service3.emperialservices.com] Waiting on rate limiter...
2020/07/19 13:33:47 [INFO][service3.emperialservices.com] Done waiting
2020/07/19 13:33:47 [INFO] [service3.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:33:47 [INFO][service5.emperialservices.com] Waiting on rate limiter...
2020/07/19 13:33:47 [INFO][service5.emperialservices.com] Done waiting
2020/07/19 13:33:47 [INFO] [service5.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:33:47 [INFO] [service2.emperialservices.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304027
2020/07/19 13:33:47 [INFO] [service2.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:33:47 [INFO] [service2.emperialservices.com] acme: use http-01 solver
2020/07/19 13:33:47 [INFO] [service2.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:33:47 [INFO] [service5.emperialservices.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304029
2020/07/19 13:33:47 [INFO] [service5.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:33:47 [INFO] [service5.emperialservices.com] acme: use http-01 solver
2020/07/19 13:33:47 [INFO] [service5.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:33:47 [INFO] [service3.emperialservices.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304024
2020/07/19 13:33:47 [INFO] [service3.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:33:47 [INFO] [service3.emperialservices.com] acme: use http-01 solver
2020/07/19 13:33:47 [INFO] [service3.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:33:48 [INFO] [service1.emperialservices.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304041
2020/07/19 13:33:48 [INFO] [service1.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:33:48 [INFO] [service1.emperialservices.com] acme: use http-01 solver
2020/07/19 13:33:48 [INFO] [service1.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:33:48 [INFO] [service4.emperialservices.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304075
2020/07/19 13:33:48 [INFO] [service4.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:33:48 [INFO] [service4.emperialservices.com] acme: use http-01 solver
2020/07/19 13:33:48 [INFO] [service4.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:33:59 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304027
2020/07/19 13:33:59 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304027
2020/07/19 13:33:59 [ERROR] error: one or more domains had a problem:
[service2.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service2.emperialservices.com/.well-known/acme-challenge/j9eCbdmy8wrmQaSgzWkO3rEcu8zU2Lup4onjF4ZA1mE: Timeout during connect (likely firewall problem), url:
(challenge=http-01 remaining=[tls-alpn-01])
2020/07/19 13:34:00 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304075
2020/07/19 13:34:00 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304075
2020/07/19 13:34:00 [ERROR] error: one or more domains had a problem:
[service4.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service4.emperialservices.com/.well-known/acme-challenge/7eNb75sRj5wYpbaGgPQRATRTiTjuKowOWXlMH3JBb5w: Timeout during connect (likely firewall problem), url:
(challenge=tls-alpn-01 remaining=[http-01])
2020/07/19 13:34:01 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304041
2020/07/19 13:34:01 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304041
2020/07/19 13:34:01 [ERROR] error: one or more domains had a problem:
[service1.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service1.emperialservices.com/.well-known/acme-challenge/kvY4Lm7kNme8obK3YPht0govHHlaCW_G0dpMG5p7A-E: Timeout during connect (likely firewall problem), url:
(challenge=tls-alpn-01 remaining=[http-01])
2020/07/19 13:34:01 [INFO] [service2.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:34:01 [INFO] [service2.emperialservices.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981306586
2020/07/19 13:34:01 [INFO] [service2.emperialservices.com] acme: use tls-alpn-01 solver
2020/07/19 13:34:01 [INFO] [service2.emperialservices.com] acme: Trying to solve TLS-ALPN-01
2020/07/19 13:34:02 [INFO] [service4.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:34:02 http: TLS handshake error from 127.0.0.1:50850: EOF
2020/07/19 13:34:02 [INFO] [service4.emperialservices.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981306758
2020/07/19 13:34:02 [INFO] [service4.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:34:02 [INFO] [service4.emperialservices.com] acme: use http-01 solver
2020/07/19 13:34:02 [INFO] [service4.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:34:03 [INFO] [service1.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:34:03 [INFO] [service1.emperialservices.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981307341
2020/07/19 13:34:03 [INFO] [service1.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:34:03 [INFO] [service1.emperialservices.com] acme: use http-01 solver
2020/07/19 13:34:03 [INFO] [service1.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:34:11 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304024
2020/07/19 13:34:11 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304024
2020/07/19 13:34:11 [ERROR] error: one or more domains had a problem:
[service3.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service3.emperialservices.com/.well-known/acme-challenge/6_fR-JeW_NaIDbEAgg3fGvCy32gbfwi4clP0xxc6dvI: Timeout during connect (likely firewall problem), url:
(challenge=tls-alpn-01 remaining=[http-01])
2020/07/19 13:34:12 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304029
2020/07/19 13:34:12 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981304029
2020/07/19 13:34:12 [ERROR] error: one or more domains had a problem:
[service5.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service5.emperialservices.com/.well-known/acme-challenge/KriX_OKg-0saO-GJNqDIRbUJmgnzO8DtEOVnZiR3CR4: Timeout during connect (likely firewall problem), url:
(challenge=http-01 remaining=[tls-alpn-01])
2020/07/19 13:34:13 [INFO] [service3.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:34:14 [INFO] [service3.emperialservices.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981309727
2020/07/19 13:34:14 [INFO] [service3.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:34:14 [INFO] [service3.emperialservices.com] acme: use http-01 solver
2020/07/19 13:34:14 [INFO] [service3.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:34:14 [INFO] [service5.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:34:15 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981306586
2020/07/19 13:34:15 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981306586
2020/07/19 13:34:15 [ERROR] error: one or more domains had a problem:
[service2.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url:
(challenge=tls-alpn-01 remaining=[])
2020/07/19 13:34:15 [INFO] [service5.emperialservices.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981310043
2020/07/19 13:34:15 [INFO] [service5.emperialservices.com] acme: use tls-alpn-01 solver
2020/07/19 13:34:15 [INFO] [service5.emperialservices.com] acme: Trying to solve TLS-ALPN-01
2020/07/19 13:34:15 http: TLS handshake error from 127.0.0.1:50858: EOF
2020/07/19 13:34:17 [ERROR] attempt 1: [service2.emperialservices.com] Obtain: [service2.emperialservices.com] error: one or more domains had a problem:
[service2.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url:
- retrying in 1m0s (29.958335485s/720h0m0s elapsed)...
2020/07/19 13:34:17 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981307341
2020/07/19 13:34:17 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981307341
2020/07/19 13:34:17 [ERROR] error: one or more domains had a problem:
[service1.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service1.emperialservices.com/.well-known/acme-challenge/EJCrZFskLnodei4aDOb5c1h1qAuZ-X5E6qoipR9nEKo: Timeout during connect (likely firewall problem), url:
(challenge=http-01 remaining=[])
2020/07/19 13:34:18 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981306758
2020/07/19 13:34:18 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981306758
2020/07/19 13:34:18 [ERROR] error: one or more domains had a problem:
[service4.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service4.emperialservices.com/.well-known/acme-challenge/8-c7ajXmClTo4Qb448TJ47XNfNS7SjwvWSH4nUotpHk: Timeout during connect (likely firewall problem), url:
(challenge=http-01 remaining=[])
2020/07/19 13:34:19 [ERROR] attempt 1: [service1.emperialservices.com] Obtain: [service1.emperialservices.com] error: one or more domains had a problem:
[service1.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service1.emperialservices.com/.well-known/acme-challenge/EJCrZFskLnodei4aDOb5c1h1qAuZ-X5E6qoipR9nEKo: Timeout during connect (likely firewall problem), url:
- retrying in 1m0s (32.713605928s/720h0m0s elapsed)...
2020/07/19 13:34:20 [ERROR] attempt 1: [service4.emperialservices.com] Obtain: [service4.emperialservices.com] error: one or more domains had a problem:
[service4.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service4.emperialservices.com/.well-known/acme-challenge/8-c7ajXmClTo4Qb448TJ47XNfNS7SjwvWSH4nUotpHk: Timeout during connect (likely firewall problem), url:
- retrying in 1m0s (33.388427966s/720h0m0s elapsed)...
2020/07/19 13:34:27 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981310043
2020/07/19 13:34:27 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981310043
2020/07/19 13:34:27 [ERROR] error: one or more domains had a problem:
[service5.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url:
(challenge=tls-alpn-01 remaining=[])
2020/07/19 13:34:29 [ERROR] attempt 1: [service5.emperialservices.com] Obtain: [service5.emperialservices.com] error: one or more domains had a problem:
[service5.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url:
- retrying in 1m0s (42.012744135s/720h0m0s elapsed)...
2020/07/19 13:34:29 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981309727
2020/07/19 13:34:29 [INFO] Unable to deactivate the authorization: https://acme-v02.api.letsencrypt.org/acme/authz-v3/5981309727
2020/07/19 13:34:29 [ERROR] error: one or more domains had a problem:
[service3.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service3.emperialservices.com/.well-known/acme-challenge/iVW0k70yatHIdZztkrqiA3JBkpxXSB1RUqEbRvPc5sg: Timeout during connect (likely firewall problem), url:
(challenge=http-01 remaining=[])
2020/07/19 13:34:31 [ERROR] attempt 1: [service3.emperialservices.com] Obtain: [service3.emperialservices.com] error: one or more domains had a problem:
[service3.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service3.emperialservices.com/.well-known/acme-challenge/iVW0k70yatHIdZztkrqiA3JBkpxXSB1RUqEbRvPc5sg: Timeout during connect (likely firewall problem), url:
- retrying in 1m0s (44.149078668s/720h0m0s elapsed)...
2020/07/19 13:35:17 [INFO] [service2.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:35:17 [INFO] [service2.emperialservices.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319692
2020/07/19 13:35:17 [INFO] [service2.emperialservices.com] acme: use tls-alpn-01 solver
2020/07/19 13:35:17 [INFO] [service2.emperialservices.com] acme: Trying to solve TLS-ALPN-01
2020/07/19 13:35:17 http: TLS handshake error from 127.0.0.1:50876: EOF
2020/07/19 13:35:19 [INFO] [service1.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:35:20 [INFO] [service1.emperialservices.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319700
2020/07/19 13:35:20 [INFO] [service1.emperialservices.com] acme: use tls-alpn-01 solver
2020/07/19 13:35:20 [INFO] [service1.emperialservices.com] acme: Trying to solve TLS-ALPN-01
2020/07/19 13:35:20 http: TLS handshake error from 127.0.0.1:50878: EOF
2020/07/19 13:35:20 [INFO] [service4.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:35:20 [INFO] [service4.emperialservices.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319703
2020/07/19 13:35:20 [INFO] [service4.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:35:20 [INFO] [service4.emperialservices.com] acme: use http-01 solver
2020/07/19 13:35:20 [INFO] [service4.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:35:29 [INFO] [service5.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:35:29 [INFO] [service5.emperialservices.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319724
2020/07/19 13:35:29 [INFO] [service5.emperialservices.com] acme: use tls-alpn-01 solver
2020/07/19 13:35:29 [INFO] [service5.emperialservices.com] acme: Trying to solve TLS-ALPN-01
2020/07/19 13:35:29 http: TLS handshake error from 127.0.0.1:50888: EOF
2020/07/19 13:35:31 [INFO] [service3.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:35:31 [INFO] [service3.emperialservices.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319728
2020/07/19 13:35:31 [INFO] [service3.emperialservices.com] acme: use tls-alpn-01 solver
2020/07/19 13:35:31 [INFO] [service3.emperialservices.com] acme: Trying to solve TLS-ALPN-01
2020/07/19 13:35:31 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319692
2020/07/19 13:35:32 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319692
2020/07/19 13:35:32 [ERROR] error: one or more domains had a problem:
[service2.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url:
(challenge=tls-alpn-01 remaining=[http-01])
2020/07/19 13:35:32 http: TLS handshake error from 127.0.0.1:50890: EOF
2020/07/19 13:35:34 [INFO] [service2.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:35:34 [INFO] [service2.emperialservices.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319735
2020/07/19 13:35:34 [INFO] [service2.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:35:34 [INFO] [service2.emperialservices.com] acme: use http-01 solver
2020/07/19 13:35:34 [INFO] [service2.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:35:41 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319700
2020/07/19 13:35:41 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319700
2020/07/19 13:35:41 [ERROR] error: one or more domains had a problem:
[service1.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url:
(challenge=tls-alpn-01 remaining=[http-01])
2020/07/19 13:35:43 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319724
2020/07/19 13:35:43 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319724
2020/07/19 13:35:43 [ERROR] error: one or more domains had a problem:
[service5.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url:
(challenge=tls-alpn-01 remaining=[http-01])
2020/07/19 13:35:43 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319728
2020/07/19 13:35:43 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319728
2020/07/19 13:35:43 [ERROR] error: one or more domains had a problem:
[service3.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url:
(challenge=tls-alpn-01 remaining=[http-01])
2020/07/19 13:35:43 [INFO] [service1.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:35:43 [INFO] [service1.emperialservices.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319756
2020/07/19 13:35:43 [INFO] [service1.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:35:43 [INFO] [service1.emperialservices.com] acme: use http-01 solver
2020/07/19 13:35:43 [INFO] [service1.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:35:44 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319735
2020/07/19 13:35:44 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319735
2020/07/19 13:35:44 [ERROR] error: one or more domains had a problem:
[service2.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service2.emperialservices.com/.well-known/acme-challenge/_PlMY6Kjs4Y17cypYEpSe1jBZzWzI7bjrcvi6E0SAr4: Timeout during connect (likely firewall problem), url:
(challenge=http-01 remaining=[])
2020/07/19 13:35:45 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319703
2020/07/19 13:35:45 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319703
2020/07/19 13:35:45 [ERROR] error: one or more domains had a problem:
[service4.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service4.emperialservices.com/.well-known/acme-challenge/55QL2jwL9FzQCiIy_to0OhZNOwyXIdFYQMv5YEG49LE: Timeout during connect (likely firewall problem), url:
(challenge=http-01 remaining=[tls-alpn-01])
2020/07/19 13:35:45 [INFO] [service5.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:35:45 [INFO] [service5.emperialservices.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319761
2020/07/19 13:35:45 [INFO] [service5.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:35:45 [INFO] [service5.emperialservices.com] acme: use http-01 solver
2020/07/19 13:35:45 [INFO] [service5.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:35:45 [INFO] [service3.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:35:45 [INFO] [service3.emperialservices.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319764
2020/07/19 13:35:45 [INFO] [service3.emperialservices.com] acme: Could not find solver for: tls-alpn-01
2020/07/19 13:35:45 [INFO] [service3.emperialservices.com] acme: use http-01 solver
2020/07/19 13:35:45 [INFO] [service3.emperialservices.com] acme: Trying to solve HTTP-01
2020/07/19 13:35:46 [ERROR] attempt 2: [service2.emperialservices.com] Obtain: [service2.emperialservices.com] error: one or more domains had a problem:
[service2.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service2.emperialservices.com/.well-known/acme-challenge/_PlMY6Kjs4Y17cypYEpSe1jBZzWzI7bjrcvi6E0SAr4: Timeout during connect (likely firewall problem), url:
- retrying in 2m0s (1m59.65502561s/720h0m0s elapsed)...
2020/07/19 13:35:47 [INFO] [service4.emperialservices.com] acme: Obtaining bundled SAN certificate given a CSR
2020/07/19 13:35:47 [INFO] [service4.emperialservices.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319769
2020/07/19 13:35:47 [INFO] [service4.emperialservices.com] acme: use tls-alpn-01 solver
2020/07/19 13:35:47 [INFO] [service4.emperialservices.com] acme: Trying to solve TLS-ALPN-01
2020/07/19 13:35:47 http: TLS handshake error from 127.0.0.1:50900: EOF
2020/07/19 13:35:57 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319756
2020/07/19 13:35:57 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319756
2020/07/19 13:35:57 [ERROR] error: one or more domains had a problem:
[service1.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service1.emperialservices.com/.well-known/acme-challenge/sApk_k_i7UL-HiIMbG9SkFFKwq16Y3a677bB80eb62s: Timeout during connect (likely firewall problem), url:
(challenge=http-01 remaining=[])
2020/07/19 13:35:57 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319761
2020/07/19 13:35:57 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319761
2020/07/19 13:35:57 [ERROR] error: one or more domains had a problem:
[service5.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service5.emperialservices.com/.well-known/acme-challenge/mKdO-TD6aM4NuJvjnuEv4xvcLWfXYaNYZTiLO8r2eng: Timeout during connect (likely firewall problem), url:
(challenge=http-01 remaining=[])
2020/07/19 13:35:59 [ERROR] attempt 2: [service1.emperialservices.com] Obtain: [service1.emperialservices.com] error: one or more domains had a problem:
[service1.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service1.emperialservices.com/.well-known/acme-challenge/sApk_k_i7UL-HiIMbG9SkFFKwq16Y3a677bB80eb62s: Timeout during connect (likely firewall problem), url:
- retrying in 2m0s (2m12.038133897s/720h0m0s elapsed)...
2020/07/19 13:35:59 [ERROR] attempt 2: [service5.emperialservices.com] Obtain: [service5.emperialservices.com] error: one or more domains had a problem:
[service5.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service5.emperialservices.com/.well-known/acme-challenge/mKdO-TD6aM4NuJvjnuEv4xvcLWfXYaNYZTiLO8r2eng: Timeout during connect (likely firewall problem), url:
- retrying in 2m0s (2m12.361308731s/720h0m0s elapsed)...
2020/07/19 13:36:01 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319764
2020/07/19 13:36:01 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319764
2020/07/19 13:36:01 [ERROR] error: one or more domains had a problem:
[service3.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service3.emperialservices.com/.well-known/acme-challenge/wk7Ym7Zi8CjnOthI4WOXv9umm0I5nKNgocyWxI8yet8: Timeout during connect (likely firewall problem), url:
(challenge=http-01 remaining=[])
2020/07/19 13:36:01 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319769
2020/07/19 13:36:01 [INFO] Unable to deactivate the authorization: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/79319769
2020/07/19 13:36:01 [ERROR] error: one or more domains had a problem:
[service4.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url:
(challenge=tls-alpn-01 remaining=[])
2020/07/19 13:36:03 [ERROR] attempt 2: [service3.emperialservices.com] Obtain: [service3.emperialservices.com] error: one or more domains had a problem:
[service3.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching http://service3.emperialservices.com/.well-known/acme-challenge/wk7Ym7Zi8CjnOthI4WOXv9umm0I5nKNgocyWxI8yet8: Timeout during connect (likely firewall problem), url:
- retrying in 2m0s (2m16.537004343s/720h0m0s elapsed)...
2020/07/19 13:36:03 [ERROR] attempt 2: [service4.emperialservices.com] Obtain: [service4.emperialservices.com] error: one or more domains had a problem:
[service4.emperialservices.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Timeout during connect (likely firewall problem), url:
- retrying in 2m0s (2m16.656090983s/720h0m0s elapsed)...
^X^C2020/07/19 18:36:37.434 INFO shutting down {"signal": "SIGINT"}
2020/07/19 13:36:37 [INFO][service3.emperialservices.com] Obtain: Releasing lock
2020/07/19 13:36:37 [INFO][cache:0xc0001869c0] Stopped certificate maintenance routine
2020/07/19 13:36:37 [INFO][service5.emperialservices.com] Obtain: Releasing lock
2020/07/19 13:36:37 [INFO][service1.emperialservices.com] Obtain: Releasing lock
2020/07/19 13:36:37 [INFO][service2.emperialservices.com] Obtain: Releasing lock
2020/07/19 13:36:37 [INFO][service4.emperialservices.com] Obtain: Releasing lock
2020/07/19 13:36:37 [ERROR][service4.emperialservices.com] Obtain: Unable to unlock 'cert_acme_service4.emperialservices.com_acme-v02.api.letsencrypt.org-directory': remove /root/.local/share/caddy/locks/cert_acme_service4.emperialservices.com_acme-v02.api.letsencrypt.org-directory.lock: no such file or directory
2020/07/19 13:36:37 [ERROR] service4.emperialservices.com: obtaining certificate: context canceled
2020/07/19 13:36:37 [ERROR] Unable to clean up lock: remove /root/.local/share/caddy/locks/cert_acme_service3.emperialservices.com_acme-v02.api.letsencrypt.org-directory.lock: no such file or directory (lock=cert_acme_service3.emperialservices.com_acme-v02.api.letsencrypt.org-directory storage=FileStorage:/root/.local/share/caddy)
2020/07/19 13:36:37 [ERROR] Unable to clean up lock: remove /root/.local/share/caddy/locks/cert_acme_service5.emperialservices.com_acme-v02.api.letsencrypt.org-directory.lock: no such file or directory (lock=cert_acme_service5.emperialservices.com_acme-v02.api.letsencrypt.org-directory storage=FileStorage:/root/.local/share/caddy)
2020/07/19 13:36:37 [ERROR] service3.emperialservices.com: obtaining certificate: context canceled
2020/07/19 18:36:37.435 INFO admin stopped previous server
2020/07/19 13:36:37 [ERROR] service5.emperialservices.com: obtaining certificate: context canceled
2020/07/19 18:36:37.435 INFO shutdown done {"signal": "SIGINT"}
2d. Workaround(s)
None that I am aware of.
2e. Relevant links
None that I am aware of.
3. Tutorial (minimal steps to reproduce the bug)
- Install Caddy 2.1.1.
- Run Caddy with the Caddyfile above.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 22 (13 by maintainers)
Okay, great! No idea what happened either. But glad to see that it’s working again nonetheless.
I wonder if the ACME challenges will succeed. I’m still skeptical of the system’s networking stack/configuration to some extent, but I’m feeling more confident that it’s not a bug in Caddy.
Good news!
The fury.io server error resolved. So, I removed Caddy, cleaned up my original config, and rebooted. I then installed 2.0.0, and I configured the Caddyfile and service from scratch the way I had previously. I rebooted again, and then tested the config. It… worked. To verify that it isn’t an issue with 2.1.1, I updated Caddy to 2.1.1, and rebooted again just for good measure. And… it still works.
I have no clue what happened here. I’ve rebooted the server multiple times in trying to diagnose the issue. So, that seems unlikely. The only thing I can think of is that somehow something related to package install/setup on my server failed when updating Caddy most recently, and somehow, wiping and starting fresh fixed it.
The only other guess I have is that my dynamic DNS settings weren’t working for some reason, and suddenly came back online this afternoon after a couple weeks of not working. That seems unlikely, but so does a magic re-install. ¯_(ツ)_/¯
Interestingly, IPv6 is still what’s being bound for Caddy. Here’s what it shows in netstat:
I’m still a little put off by that since it seems like I might run into trouble at some point related to that and my DNS provider. But… I’m not going to look a gift-working-reverse-proxy in the mouth 😉.
Thanks for all your time and effort in helping me try to figure out my issue. I really appreciate it, and Caddy is a super rad project 😃.