traefik: Traefik ACME DNS-01 gcloud duplicate records.

Do you want to request a feature or report a bug?

Bug

What did you do?

I’m not sure if this is a traefik or upstream bug but changing my acme provider from route53 to gcloud my configuration generates 409 duplicate resource errors from google when generating certificates for two subdomains. Credentials are fine in both cases, the error appears isolated to the acme gcloud provider.

What did you expect to see?

Successful certificate generation with gcloud provider. The configuration works with route53 set as the provider.

What did you see instead?

Error 409. It looks like the acme code expects to be able to upsert TXT records but googles API doesn’t allow it. See log output below.

Output of `traefik version`: (What version of Traefik are you using?)

Version:      v1.7.4
Codename:     maroilles
Go version:   go1.11.1
Built:        2018-10-30_10:44:30AM
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, …)?

[acme]
  email = "email@mydomain.com"
  storage = "/acme-certificates/acme.json"
  acmeLogging = true
  entryPoint = "https"

[acme.dnsChallenge]
  provider = "gcloud"

[[acme.domains]]
  main = "*.development.mydomain.com"
  sans = ["development.mydomain.com"]

[[acme.domains]]
  main = "*.qa.mydomain.com"
  sans = ["qa.mydomain.com"]

If applicable, please paste the log output in DEBUG level (`--logLevel=DEBUG` switch)

time="2018-11-01T11:46:41Z" level=error msg="Error obtaining certificate: acme: Error -> One or more domains had a problem:\n[qa.mydomain.com] error presenting token: googlecloud: googleapi: Error 409: The resource 'entity.change.additions[0].rrdata[1]' named '\"VuOLZS3tfLfrOswFtV3mayqOYaz5PvQK16nJdgMdMez\"' already exists, alreadyExists\n"
time="2018-11-01T11:46:41Z" level=error msg="Unable to obtain ACME certificate for domains \"*.qa.mydomain.com,qa.mydomain.com\" : unable to generate a certificate for the domains [*.qa.mydomain.com qa.mydomain.com]: acme: Error -> One or more domains had a problem:\n[qa.mydomain.com] error presenting token: googlecloud: googleapi: Error 409: The resource 'entity.change.additions[0].rrdata[1]' named '\"VuOLZS3tfLfrOswFtV3mayqOYaz5PvQK16nJdgMdMez\"' already exists, alreadyExists\n"
time="2018-11-01T11:46:41Z" level=error msg="Error obtaining certificate: acme: Error -> One or more domains had a problem:\n[development.mydomain.com] error presenting token: googlecloud: googleapi: Error 409: The resource 'entity.change.additions[0].rrdata[1]' named '\"Jfv6orVfbwi0IdQbmpvG3qnIltMX5x56vGUpHjoBzOa\"' already exists, alreadyExists\n"
time="2018-11-01T11:46:41Z" level=error msg="Unable to obtain ACME certificate for domains \"*.development.mydomain.com,development.mydomain.com\" : unable to generate a certificate for the domains [*.development.mydomain.com development.mydomain.com]: acme: Error -> One or more domains had a problem:\n[development.mydomain.com] error presenting token: googlecloud: googleapi: Error 409: The resource 'entity.change.additions[0].rrdata[1]' named '\"Jfv6orVfbwi0IdQbmpvG3qnIltMX5x56vGUpHjoBzOa\"' already exists, alreadyExists\n"

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 42 (19 by maintainers)

Most upvoted comments

I finally found the root of the bug, a PR is coming really soon.

ldez on Dec 21, 2018

but as @shanna suggested, the best solution would be to adapt the provider implementation so that it checks if the required TXT record already exists … maybe that is something I could implement an create a PR

tompson on Dec 21, 2018

i have had this same kind of issue recently on gcloud, my current workaround is adding --acme.dnschallenge.delaybeforecheck=180" or if you use traefik.toml

[acme.dnsChallenge]
delayBeforeCheck = 180

it seems to try and set the TXT record multiple times, By setting a 3 minute delay, it seems to fix this.

By default the TXT record’s TTL on google is set to 2 minutes, by delaying the dns check it seems to solve this issue.

My guess would be it’s attempting to do the dns certificate mutliple times at startup and TTL caching the incorrect token but having the correct token in DNS.

HotaruBlaze on Dec 21, 2018