traefik: ACME HTTP-01 challenge fails by timeout

Do you want to request a feature or report a bug?

Bug

What did you do?

I am trying to fetch automatic certificates from Let’s Encrypt with HTTP-01.

What did you expect to see?

Fetching certificates like before TLS-SNI problems.

What did you see instead?

No new certificates.

Possible problems / fixes

It looks like it has something to do with adding the http route to each domain (domain.com/.well-known/acme-challenge/[token]). When visiting the same route over https I receive an 404 directly. But via http timeouts.

https://github.com/containous/traefik/blob/5140bbe99a79b45f98c27fbb8e9b6833194af4cb/acme/challenge_http_provider.go#L22

Via Slack someone (maverick) tried my same configuration but with a consul backend. Maybe it has something to do with that?

When checking de debug logs it seems it “CleansUp” token for that domain before hitting the timeout. Maybe it has something to do with that?

Output of traefik version: (What version of Traefik are you using?)

Traefik version v1.5.0 built on 2018-01-23_04:42:32PM

What is your environment & configuration (arguments, toml, provider, platform, …)?

defaultEntryPoints = ["http", "https"]
debug = true
logLevel = "DEBUG"

[entryPoints]
  [entryPoints.http]
  address = ":80"
#    [entryPoints.http.redirect]
#    entryPoint = "https"
  compress = true
  [entryPoints.https]
    address = ":443"
    compress = true
    [entryPoints.https.tls]

[acme]
  email = "email@address.com"
  caServer = "https://acme-staging.api.letsencrypt.org/directory"
  # Tried it on production as well
  storage = "/etc/traefik/acme/acme.json"
  entryPoint = "https"
  OnHostRule = true
  acmeLogging = true
  [acme.httpChallenge]
    entryPoint = "http"

# Enable Docker configuration backend
[docker]
  endpoint = "unix:///var/run/docker.sock"
  domain = "sandbox.domain.com"
  watch = true
  swarmmode = true
  exposedbydefault = true

[api]
  entryPoint = "traefik"
  dashboard = true
  address = ":8080"

  [api.statistics]
    recentErrors = 10

docker-compose.yml

version: '3'
services:
  nginx:
    image: nginx:1.13
    volumes:
      - "../workspace:/srv"
      - "./nginx/default.conf:/etc/nginx/conf.d/default.conf"
    deploy:
      labels:
        - "traefik.backend=rest-api"
        - "traefik.port=80"
        - "traefik.frontend.rule=Host:rest-api.sandbox.domain.com"
        - "traefik.docker.network=frontend"
        - "traefik.backend.loadbalancer.method=drr"
    networks:
      - frontend
      - backend

  php:
    image: php-fpm:7.1
    volumes:
      - "../workspace:/srv"
    networks:
      - backend

networks:
  backend:
    external:
      name: rest-api
  frontend:
    external:
      name: frontend

If applicable, please paste the log output in debug mode (--debug switch)

logs 
time="2018-01-25T10:05:56Z" level=debug msg="LoadCertificateForDomains [rest-api.sandbox.domain.com]..." 
time="2018-01-25T10:05:56Z" level=debug msg="Looking for provided certificate to validate [rest-api.sandbox.domain.com]..." 
time="2018-01-25T10:05:56Z" level=debug msg="No provided certificate found for domains [rest-api.sandbox.domain.com], get ACME certificate." 
time="2018-01-25T10:05:56Z" level=debug msg="Loading ACME certificates [rest-api.sandbox.domain.com]..." 
legolog: 2018/01/25 10:05:56 [INFO][rest-api.sandbox.domain.com] acme: Obtaining bundled SAN certificate
legolog: 2018/01/25 10:05:56 [INFO][rest-api.sandbox.domain.com] AuthURL: https://acme-staging.api.letsencrypt.org/acme/authz/w3M__oDqozE[...]T_SPCiF7p5CYLFI
legolog: 2018/01/25 10:05:56 [INFO][rest-api.sandbox.domain.com] acme: Could not find solver for: dns-01
legolog: 2018/01/25 10:05:56 [INFO][rest-api.sandbox.domain.com] acme: Trying to solve HTTP-01
time="2018-01-25T10:05:56Z" level=debug msg="Challenge Present rest-api.sandbox.domain.com" 
time="2018-01-25T10:06:07Z" level=debug msg="Challenge CleanUp rest-api.sandbox.domain.com" 
time="2018-01-25T10:06:07Z" level=error msg="map[rest-api.sandbox.domain.com:acme: Error 400 - urn:acme:error:connection - Fetching http://rest-api.sandbox.domain.com/.well-known/acme-challenge/GECQ9JRWb4pA[...]Bc3rmeveJd611YowU: Timeout
Error Detail:
	Validation for rest-api.sandbox.domain.com:80
	Resolved to:
		***.***.***.***
		***:*:*:*::*
	Used: ***:*:*:*::*

]" 
time="2018-01-25T10:06:07Z" level=error msg="Error getting ACME certificates [rest-api.sandbox.domain.com] : cannot obtain certificates map[rest-api.sandbox.domain.com:acme: Error 400 - urn:acme:error:connection - Fetching http://rest-api.sandbox.domain.com/.well-known/acme-challenge/GECQ9JRWb4pA0OlC[...]eJd611YowU: Timeout
Error Detail:
	Validation for rest-api.sandbox.domain.com:80
	Resolved to:
		***.***.***.***
		***:*:*:*::*
	Used: ***:*:*:*::*

]" 
time="2018-01-25T10:06:07Z" level=debug msg="LoadCertificateForDomains []..." 
legolog: 2018/01/25 10:06:07 [INFO][exceptions.sandbox.domain.com] acme: Obtaining bundled SAN certificate
time="2018-01-25T10:06:07Z" level=debug msg="LoadCertificateForDomains [exceptions.sandbox.domain.com]..." 
time="2018-01-25T10:06:07Z" level=debug msg="Looking for provided certificate to validate [exceptions.sandbox.domain.com]..." 
time="2018-01-25T10:06:07Z" level=debug msg="No provided certificate found for domains [exceptions.sandbox.domain.com], get ACME certificate." 
time="2018-01-25T10:06:07Z" level=debug msg="Loading ACME certificates [exceptions.sandbox.domain.com]..." 
legolog: 2018/01/25 10:06:07 [INFO][exceptions.sandbox.domain.com] AuthURL: https://acme-staging.api.letsencrypt.org/acme/authz/oUlowLzxA9hKGib[...]MpTqEWA4ksu345xc
legolog: 2018/01/25 10:06:07 [INFO][exceptions.sandbox.domain.com] acme: Could not find solver for: dns-01
legolog: 2018/01/25 10:06:07 [INFO][exceptions.sandbox.domain.com] acme: Trying to solve HTTP-01
time="2018-01-25T10:06:07Z" level=debug msg="Challenge Present exceptions.sandbox.domain.com" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label traefik.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label payment_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label my_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label webfrontend_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label rest-api_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label order_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label catalog_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label price_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label notifications_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label exceptions_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels"

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 3
  • Comments: 50 (12 by maintainers)

Most upvoted comments

I ran into this also today, now I’m not 100% sure the last change I made resolved it as I changed the domain due to reaching the failed request limit. However the subsequent request did work, and the change was indentation - in the docs (https://docs.traefik.io/configuration/acme/) nothing under [acme] is indented, previously I had [acme.httpChallenge] indented as you do.

So try changing:

[acme]
  email = "email@address.com"
  caServer = "https://acme-staging.api.letsencrypt.org/directory"
  # Tried it on production as well
  storage = "/etc/traefik/acme/acme.json"
  entryPoint = "https"
  OnHostRule = true
  acmeLogging = true
  [acme.httpChallenge]
    entryPoint = "http"

to

[acme]
email = "email@address.com"
caServer = "https://acme-staging.api.letsencrypt.org/directory"
# Tried it on production as well
storage = "/etc/traefik/acme/acme.json"
entryPoint = "https"
OnHostRule = true
acmeLogging = true
[acme.httpChallenge]
  entryPoint = "http"

and see if it helps.

+8
cheeseweasel on Jan 2, 2019

This issue still persist. Is it possible to ignore IPv6?

+5
deargonaut on Apr 3, 2018

I had this issue with the tag 2.0-alpine (I know that is an alpha version yet), and the way I solved this was replacing /etc/resolv.conf with a custom resolv.conf file, with ‘nameserver 1.1.1.1’

After this, traefik works like a charm.

+3
richsanram on Apr 17, 2019

While debugging with Juliens we found a fix for this error.

It seemed that while trying to reach the .well-known/acme-challange url it always wanted to go via IPv6. When we removed the IPv6-interface and cleared it from DNS it got authenticated and I received my certificates.

Issue will remain open for Julien to come up how to reproduce and maybe fix this.

+3
deargonaut on Jan 31, 2018

Hi, I ran into the same issue and I am intrested in the fix which @deargonaut described. I have two questions, though.

it always wanted to go via IPv6

What is it? The Let’s Encrypt client trying to reach .well-known/acme-challenge url?

we removed the IPv6-interface

From where did you remove the IPv6-interface? Did you remove it from the host?

+2
schasse on Feb 7, 2018

Same here. With IPv6 assigned to server, HTTP validation of domains with AAAA records ends with timeout. After disabling IPv6 and restart Traefik, all domains (even those with AAAA records) validated OK.

Traefik version 1.5.3

+1
SniperCZE on Mar 21, 2018

Hello @deargonaut .

Is it possible for you to continue the discussion with the team in our Slack. @Juliens created a thread.

I guess thanks to this more interactive way it should be easier to help you.

Thanks in advance

+1
nmengin on Jan 26, 2018

Hello @deargonaut. Tthanks for your interest in the project.

This kind of timeout is generated by LEGO (the Let’s Encrypt GO library used by Træfik). It happens when LE cannot access to Træfik in the way to do a HTTP challenge.

Even if the log appears after the CleanUp log, it’s generated before during the challenge step as you can see in the Træfik code.

Can you check if :

  • The subdomain rest-api.sandbox.domain.com is mapped to the host where Træfik is deployed
  • The port 80 of the host where Træfik is deployed is reachable by LE in the port 80.

Thanks in advance.

+1
nmengin on Jan 25, 2018
Read more comments on GitHub
← traefik: ACME cannot select preferred chain in v1.7 which generates broken certificates (expired root CA in chain)
traefik: Acme process cannot find TLS key on Etcd3 →