traefik: Acme process cannot find TLS key on Etcd3

Do you want to request a feature or report a bug?

bug

What did you do?

I created a docker-compose with Træfik and Etcd3. Since Træfik v1.5.0-rc1, I got a Key not found in store when I enable Acme. It works on acme disabled and with a storage file as a store instead of etcd too.

PS: It worked on Træfik v1.4 and Etcd2

What did you expect to see?

Træfik to start

What did you see instead?

etcd_1     | 2018-01-16 18:48:49.672552 I | etcdmain: etcd Version: 3.2.13
etcd_1     | 2018-01-16 18:48:49.672696 I | etcdmain: Git SHA: 95a726a
traefik_1  | time="2018-01-16T18:48:51Z" level=info msg="Using TOML configuration file /etc/traefik/traefik.toml"
etcd_1     | 2018-01-16 18:48:49.672728 I | etcdmain: Go Version: go1.8.5
etcd_1     | 2018-01-16 18:48:49.672754 I | etcdmain: Go OS/Arch: linux/amd64
etcd_1     | 2018-01-16 18:48:49.672769 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
etcd_1     | 2018-01-16 18:48:49.672928 I | embed: listening for peers on http://0.0.0.0:2380
traefik_1  | time="2018-01-16T18:48:51Z" level=warning msg="web provider configuration is deprecated, you should use these options : api, rest provider, ping and metrics"
traefik_1  | time="2018-01-16T18:48:51Z" level=info msg="Traefik version v1.5.0-rc5 built on 2018-01-15_03:59:03PM"
etcd_1     | 2018-01-16 18:48:49.673032 I | embed: listening for client requests on 0.0.0.0:2379
etcd_1     | 2018-01-16 18:48:49.675664 I | pkg/netutil: resolving etcd:2380 to 172.22.0.2:2380
traefik_1  | time="2018-01-16T18:48:51Z" level=info msg="
traefik_1  | Stats collection is disabled.
traefik_1  | Help us improve Traefik by turning this feature on :)
traefik_1  | More details on: https://docs.traefik.io/basic/#collected-data
traefik_1  | "
traefik_1  | time="2018-01-16T18:48:51Z" level=debug msg="Global configuration loaded {"LifeCycle":{"RequestAcceptGraceTimeout":0,"GraceTimeOut":0},"GraceTimeOut":0,"Debug":true,"CheckNewVersion":true,"SendAnonymousUsage":false,"AccessLogsFile":"","AccessLog":null,"TraefikLogsFile":"","TraefikLog":null,"LogLevel":"DEBUG","EntryPoints":{"http":{"Network":"","Address":":80","TLS":null,"Redirect":{"entryPoint":"https"},"Auth":null,"WhitelistSourceRange":null,"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}},"https":{"Network":"","Address":":443","TLS":{"MinVersion":"","CipherSuites":null,"Certificates":null,"ClientCAFiles":null,"ClientCA":{"Files":null,"Optional":false}},"Redirect":null,"Auth":null,"WhitelistSourceRange":null,"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}},"traefik":{"Network":"","Address":":8080","TLS":null,"Redirect":null,"Auth":null,"WhitelistSourceRange":null,"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}}},"Cluster":{"Node":"4cd0e250-14b2-41a7-ac33-e4d8089e7e2b","Store":{"Store":{},"Prefix":"/traefik"}},"Constraints":[],"ACME":{"Email":"acme@yoannboyer.com","Domains":null,"Storage":"/traefik/acme/account","StorageFile":"","OnDemand":false,"OnHostRule":true,"CAServer":"https://acme-staging.api.letsencrypt.org/directory","EntryPoint":"https","DNSChallenge":null,"HTTPChallenge":{"EntryPoint":"http"},"DNSProvider":"","DelayDontCheckDNS":0,"ACMELogging":true,"TLSConfig":null},"DefaultEntryPoints":["http","https"],"ProvidersThrottleDuration":2000000000,"MaxIdleConnsPerHost":200,"IdleTimeout":0,"InsecureSkipVerify":false,"RootCAs":null,"Retry":null,"HealthCheck":{"Interval":30000000000},"RespondingTimeouts":null,"ForwardingTimeouts":null,"Web":{"Address":":8080","CertFile":"","KeyFile":"","ReadOnly":false,"Statistics":null,"Metrics":null,"Path":"/","Auth":null,"Debug":false},"Docker":null,"File":null,"Marathon":null,"Consul":null,"ConsulCatalog":null,"Etcd":{"Watch":true,"Filename":"","Constraints":[],"Trace":false,"DebugLogGeneratedTemplate":false,"Endpoint":"etcd:2379","Prefix":"/traefik","TLS":null,"Username":"","Password":"","UseAPIV3":true},"Zookeeper":null,"Boltdb":null,"Kubernetes":null,"Mesos":null,"Eureka":null,"ECS":null,"Rancher":null,"DynamoDB":null,"ServiceFabric":null,"Rest":null,"API":{"EntryPoint":"traefik","Dashboard":true,"Debug":true,"CurrentConfigurations":null,"Statistics":null},"Metrics":null,"Ping":{"EntryPoint":"traefik"}}"
etcd_1     | 2018-01-16 18:48:49.676077 I | pkg/netutil: resolving etcd:2380 to 172.22.0.2:2380
etcd_1     | 2018-01-16 18:48:49.676170 I | etcdserver: name = etcd
etcd_1     | 2018-01-16 18:48:49.676178 I | etcdserver: data dir = /etcd-data
etcd_1     | 2018-01-16 18:48:49.676183 I | etcdserver: member dir = /etcd-data/member
etcd_1     | 2018-01-16 18:48:49.676187 I | etcdserver: heartbeat = 100ms
traefik_1  | time="2018-01-16T18:48:51Z" level=info msg="Preparing server https &{Network: Address::443 TLS:0xc420634680 Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420691580} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
etcd_1     | 2018-01-16 18:48:49.676191 I | etcdserver: election = 1000ms
etcd_1     | 2018-01-16 18:48:49.676195 I | etcdserver: snapshot count = 100000
etcd_1     | 2018-01-16 18:48:49.676301 I | etcdserver: advertise client URLs = http://etcd:2379
etcd_1     | 2018-01-16 18:48:49.676345 I | etcdserver: initial advertise peer URLs = http://etcd:2380
etcd_1     | 2018-01-16 18:48:49.676355 I | etcdserver: initial cluster = etcd=http://etcd:2380
etcd_1     | 2018-01-16 18:48:49.682542 I | etcdserver: starting member f24244a5c413e9f5 in cluster f72b19096fb8574b
etcd_1     | 2018-01-16 18:48:49.682581 I | raft: f24244a5c413e9f5 became follower at term 0
etcd_1     | 2018-01-16 18:48:49.682591 I | raft: newRaft f24244a5c413e9f5 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
etcd_1     | 2018-01-16 18:48:49.682594 I | raft: f24244a5c413e9f5 became follower at term 1
etcd_1     | 2018-01-16 18:48:49.688108 W | auth: simple token is not cryptographically signed
etcd_1     | 2018-01-16 18:48:49.690688 I | etcdserver: starting server... [version: 3.2.13, cluster version: to_be_decided]
etcd_1     | 2018-01-16 18:48:49.691550 I | etcdserver/membership: added member f24244a5c413e9f5 [http://etcd:2380] to cluster f72b19096fb8574b
etcd_1     | 2018-01-16 18:48:49.983030 I | raft: f24244a5c413e9f5 is starting a new election at term 1
etcd_1     | 2018-01-16 18:48:49.983092 I | raft: f24244a5c413e9f5 became candidate at term 2
etcd_1     | 2018-01-16 18:48:49.983156 I | raft: f24244a5c413e9f5 received MsgVoteResp from f24244a5c413e9f5 at term 2
etcd_1     | 2018-01-16 18:48:49.983214 I | raft: f24244a5c413e9f5 became leader at term 2
etcd_1     | 2018-01-16 18:48:49.983220 I | raft: raft.node: f24244a5c413e9f5 elected leader f24244a5c413e9f5 at term 2
etcd_1     | 2018-01-16 18:48:49.983664 I | etcdserver: published {Name:etcd ClientURLs:[http://etcd:2379]} to cluster f72b19096fb8574b
etcd_1     | 2018-01-16 18:48:49.983750 I | etcdserver: setting up the initial cluster version to 3.2
etcd_1     | 2018-01-16 18:48:49.983819 I | embed: ready to serve client requests
etcd_1     | 2018-01-16 18:48:49.984149 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!
etcd_1     | 2018-01-16 18:48:49.984517 N | etcdserver/membership: set the initial cluster version to 3.2
etcd_1     | 2018-01-16 18:48:49.984821 I | etcdserver/api: enabled capabilities for version 3.2
traefik_1  | time="2018-01-16T18:48:51Z" level=error msg="Error creating TLS config: Key not found in store"
traefik_1  | time="2018-01-16T18:48:51Z" level=fatal msg="Error preparing server: Key not found in store"
test_traefik_1 exited with code 1

Output of `traefik version`: (What version of Traefik are you using?)

Traefik version v1.5.0-rc5 built on 2018-01-15_03:59:03PM

What is your environment & configuration (arguments, toml, provider, platform, …)?

docker-compose.yml

version: '3'

services:
  etcd:
    image: gcr.io/etcd-development/etcd:v3.2.13
    restart: on-failure
    ports:
      - 2380:2380
      - 2379:2379
    command:
      - /usr/local/bin/etcd
      - --enable-v2=false
      - --data-dir=/etcd-data
      - --name=etcd
      - --initial-cluster-token=etcd-cluster-1
      - --initial-advertise-peer-urls=http://etcd:2380
      - --listen-client-urls=http://0.0.0.0:2379
      - --advertise-client-urls=http://etcd:2379
      - --listen-peer-urls=http://0.0.0.0:2380
      - --initial-cluster=etcd=http://etcd:2380
      - --initial-cluster-state=new

  traefik:
    image: traefik:v1.5.0-rc5
    restart: on-failure
    links:
      - etcd
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./traefik.toml:/etc/traefik/traefik.toml

traefik.toml:

debug = true
defaultEntryPoints = ["http", "https"]

[acme]
email = "acme@yoannboyer.com"
storage = "/traefik/acme/account"
entryPoint = "https"
acmeLogging = true
onHostRule = true
caServer = "https://acme-staging.api.letsencrypt.org/directory"
  [acme.httpChallenge]
  entryPoint = "http"

[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]

[etcd]
endpoint = "etcd:2379"
useAPIV3 = true

About this issue

Original URL
State: open
Created 6 years ago
Reactions: 2
Comments: 15 (4 by maintainers)

Commits related to this issue

Include missing key in error message for KV store Improves the operator experience when debugging issues like https://github.com/containous/traefik/issues/2712 — committed to camelpunch/traefik by camelpunch 6 years ago

Most upvoted comments

well, that is really sad, maybe we can add extra flag like acme.initStorageOnEmpty No? Name can be changed=)

tpoxa on Nov 20, 2018

I’ve spent the whole day trying to make it work with etcd without luck. There’s definitely room for improvements. I’m tried the docker version a.7-alpine.

guilhermeaiolfi on May 10, 2019

Hey @nmengin.

I use Etcd as replicated store for certificates and Træfik rules too 👍

Ok, now I see why I had these errors… Thanks for your help ! I use a variant of your first solution and it works.

I have some questions / suggestions then:

Why Træfik do not create this keys if they do not exist on the store ? ;
The solutions that you gave should not be mentioned somewhere on the ACME doc ? ;
The error Error creating TLS config: Key not found in store means everything and nothing… it might be useful to be more explicit I guess ;

yboyer on Jan 18, 2018

Hello @yboyer.

Tell me if I’m wrong but I guess you are only using your ETCD service to store your ACME certificates?

You have an error because, in the way to manage the ACME certificates in a KV store, Træfik needs two keys : /traefik/acme/account/lock and /traefik/acme/account/object and these keys are missing in your ETCD when you are starting Træfik.

There are two solutions :

Add these keys manually thanks to the etcdctl command. You can do it thanks to Docker services too :

  etcdctl-put-lock:
      image: tenstartups/etcdctl
      command: --endpoints=[${ETCD_IP}:2379] put "/traefik/acme/account/lock" ""
      environment:
          ETCDCTL_DIAL_: "TIMEOUT 10s"
          ETCDCTL_API : "3"
  etcdctl-put-object:
      image: tenstartups/etcdctl
      command: --endpoints=[${ETCD_IP}:2379] put "/traefik/acme/account/object" ""
      environment:
          ETCDCTL_DIAL_: "TIMEOUT 10s"
          ETCDCTL_API : "3"

With ${ETCD_IP} your etcd service IP (in docker network)

You can use the Træfik storeconfig command to copy all you configuration from the file to the etcd service :

  traefik-storeconfig:
      image: traefik:v1.5.0-rc5
      volumes:
        - "./traefik.toml:/etc/traefik/traefik.toml:ro"
      command: storeconfig

The keys will be automatically created.

Note : If you use the second solution, all your Træfik configuration will be copy into etcd. You’ll be able to specify to Træfik to use the KV store instead of the configuration file in the Træfik service.

Are these solutions OK for you?

nmengin on Jan 18, 2018

@yboyer try adding storageFile to the acme section:

[acme]
email = "acme@yoannboyer.com"
storage = "/traefik/acme/account"
storageFile = "/acme/acme.json"
#....

See https://github.com/containous/traefik/issues/927 for details.

igoratencompass on Jan 16, 2018