traefik: acme.json not being created in pod (running on Kubernetes): no such file or directory

Welcome!

• Yes, I’ve searched similar issues on GitHub and didn’t find any.
• Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.

What did you do?

Github Issue at the traefik-helm-chart repository: https://github.com/traefik/traefik-helm-chart/issues/564

As this does not look like it is caused by the helm chart, I am opening up this bug report.

I am running traefik in a k3s kubernetes cluster, installed via helm (not using the traefik preinstalled by k3s). I tried to configure a certificateresolver using the following content in the values.yaml:

additionalArguments:
  - --certificatesresolvers.hetzner-staging.acme.caserver="https://acme-staging-v02.api.letsencrypt.org/directory"
  - --certificatesresolvers.hetzner-staging.acme.email="le-ojkastl@ojkastl.de"
  - --certificatesresolvers.hetzner-staging.acme.storage="/data/acme.json"

What did you see instead?

I get the following error during startup:

time="2022-02-25T07:29:41Z" level=error msg="The ACME resolver \"hetzner-staging\" is skipped from the resolvers list because: unable to get ACME account: open \"/data/acme.json\": no such file or directory"

This happens:

• whether or not I am enabling persistence (so the acme.json would be in a volume and would be kept and not recreated)
• whether or not I use a sidecar container to fix the permissions (see #6972)
• whether or not I am using the annotations or fsGroup settings that were recommended in earlier bug reports for file permission problems

What version of Traefik are you using?

Traefik 2.6.1 from helm chart version traefik-10.14.2

What is your environment & configuration?

values.yaml:

additionalArguments:
  - --certificatesresolvers.hetzner-staging.acme.caserver="https://acme-staging-v02.api.letsencrypt.org/directory"
  - --certificatesresolvers.hetzner-staging.acme.email="le-ojkastl@ojkastl.de"
  - --certificatesresolvers.hetzner-staging.acme.storage="/data/acme.json"

If applicable, please paste the log output in DEBUG level

time="2022-02-24T21:32:56Z" level=info msg="Configuration loaded from flags."
time="2022-02-24T21:32:56Z" level=info msg="Traefik version 2.6.1 built on 2022-02-14T16:50:25Z"
time="2022-02-24T21:32:56Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true,\"sendAnonymousUsage\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"metrics\":{\"address\":\":9100/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}},\"traefik\":{\"address\":\":9000/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":8000/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":8443/tcp\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"kubernetesIngress\":{\"ingressEndpoint\":{\"publishedService\":\"traefik-system/traefik\"}},\"kubernetesCRD\":{\"allowCrossNamespace\":true}},\"api\":{\"dashboard\":true},\"metrics\":{\"prometheus\":{\"buckets\":[0.1,0.3,1.2,5],\"addEntryPointsLabels\":true,\"addServicesLabels\":true,\"entryPoint\":\"metrics\"}},\"ping\":{\"entryPoint\":\"traefik\",\"terminatingStatusCode\":503},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"certificatesResolvers\":{\"hetzner-staging\":{\"acme\":{\"email\":\"\\\"le-ojkastl@ojkastl.de\\\"\",\"caServer\":\"\\\"https://acme-staging-v02.api.letsencrypt.org/directory\\\"\",\"storage\":\"\\\"/data/acme.json\\\"\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160}}},\"pilot\":{\"dashboard\":true}}"
time="2022-02-24T21:32:56Z" level=info msg="Stats collection is enabled."
time="2022-02-24T21:32:56Z" level=info msg="Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration."
time="2022-02-24T21:32:56Z" level=info msg="Help us improve Traefik by leaving this feature on :)"
time="2022-02-24T21:32:56Z" level=info msg="More details on: https://doc.traefik.io/traefik/contributing/data-collection/"
time="2022-02-24T21:32:56Z" level=error msg="The ACME resolver \"hetzner-staging\" is skipped from the resolvers list because: unable to get ACME account: open \"/data/acme.json\": no such file or directory"
time="2022-02-24T21:32:56Z" level=debug msg="Configured Prometheus metrics" metricsProviderName=prometheus
time="2022-02-24T21:32:56Z" level=info msg="Starting provider aggregator.ProviderAggregator"
time="2022-02-24T21:32:56Z" level=debug msg="Start TCP Server" entryPointName=web
time="2022-02-24T21:32:56Z" level=debug msg="Start TCP Server" entryPointName=websecure
time="2022-02-24T21:32:56Z" level=debug msg="Start TCP Server" entryPointName=traefik
time="2022-02-24T21:32:56Z" level=debug msg="Start TCP Server" entryPointName=metrics
time="2022-02-24T21:32:56Z" level=info msg="Starting provider *traefik.Provider"
time="2022-02-24T21:32:56Z" level=debug msg="*traefik.Provider provider configuration: {}"
time="2022-02-24T21:32:56Z" level=info msg="Starting provider *ingress.Provider"
time="2022-02-24T21:32:56Z" level=debug msg="*ingress.Provider provider configuration: {\"ingressEndpoint\":{\"publishedService\":\"traefik-system/traefik\"}}"
time="2022-02-24T21:32:56Z" level=info msg="Starting provider *crd.Provider"
time="2022-02-24T21:32:56Z" level=info msg="ingress label selector is: \"\"" providerName=kubernetes
time="2022-02-24T21:32:56Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetes
time="2022-02-24T21:32:56Z" level=debug msg="*crd.Provider provider configuration: {\"allowCrossNamespace\":true}"
time="2022-02-24T21:32:56Z" level=info msg="label selector is: \"\"" providerName=kubernetescrd
time="2022-02-24T21:32:56Z" level=info msg="Creating in-cluster Provider client" providerName=kubernetescrd
time="2022-02-24T21:32:56Z" level=warning msg="Cross-namespace reference between IngressRoutes and resources is enabled, please ensure that this is expected (see AllowCrossNamespace option)" providerName=kubernetescrd
time="2022-02-24T21:32:56Z" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2022-02-24T21:32:56Z" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {\"Timeout\":4000000000}"
time="2022-02-24T21:32:56Z" level=debug msg="Configuration received from provider internal: {\"http\":{\"routers\":{\"ping\":{\"entryPoints\":[\"traefik\"],\"service\":\"ping@internal\",\"rule\":\"PathPrefix(`/ping`)\",\"priority\":2147483647},\"prometheus\":{\"entryPoints\":[\"metrics\"],\"service\":\"prometheus@internal\",\"rule\":\"PathPrefix(`/metrics`)\",\"priority\":2147483647}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{},\"ping\":{},\"prometheus\":{}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"tls\":{}}" providerName=internal
time="2022-02-24T21:32:56Z" level=debug msg="No default certificate, generating one" tlsStoreName=default