traefik: Traefik cannot find kubernetes secrets

Do you want to request a feature or report a bug?

Bug

What did you do?

Deployed traefik with kubernetes backend, aiming to use certificates stored in kubernetes secrets

What did you expect to see?

Traefik using the certificate from kubernetes secret

What did you see instead?

Traefik using it’s own default certificate and logging that the kubernetes secret doesn’t exist. I let it run overnight to allow plenty of time for the secrets sync, but without luck

Output of traefik version: (What version of Traefik are you using?)

Version:      v1.6.0-rc3
Codename:     tetedemoine
Go version:   go1.10
Built:        2018-03-28_04:04:00PM
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, …)?

apiVersion: v1
kind: Service
metadata:
  name: external-ingress-proxy
  namespace: sys-ingress-pub
spec:
  selector:
    app: external-ingress-proxy
  type: NodePort
  ports:
  - name: http
    port: 80
    nodePort: 30180
  - name: https-proto
    port: 443
    nodePort: 30143
  selector:
    k8s-app: external-ingress-proxy
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-ingress-proxy
  namespace: sys-ingress-pub
  labels:
    k8s-app: external-ingress-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: external-ingress-proxy
  revisionHistoryLimit: 8
  template:
    metadata:
      labels:
        k8s-app: external-ingress-proxy
        name: external-ingress-proxy
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik:1.6.0-rc3-alpine
        name: traefik
        ports:
        - containerPort: 80
        - containerPort: 443
        - containerPort: 8080
        args:
        - --web
        - --web.metrics
        - --web.metrics.prometheus
        - --logLevel=INFO
        - --kubernetes
        - --kubernetes.labelselector=kubernetes.io/ingress.class=traefik-public
        - --entryPoints=Name:http-redirect Address::80 Redirect.EntryPoint:https-proto
        - --entryPoints=Name:https-proto Address::443 TLS
        - --defaultentrypoints=http-redirect,https-proto

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
  namespace: sys-mon
  annotations:
    kubernetes.io/tls-acme: "true"
    kubernetes.io/ingress.class: traefik
  labels:
    kubernetes.io/ingress.class: traefik-public
spec:
  tls:
  - secretName: traefik-test-tls-cert
  rules:
  - host: myservice.io
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx
          servicePort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: sys-mon
  labels:
    name: nginx
spec:
  selector:
    app: nginx
  ports:
  - name: web
    port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: sys-mon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      name: nginx
      namespace: sys-mon
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - name: web
          containerPort: 80

RBAC clusterrolebinding:

Name:         traefik-test
Labels:       <none>
Annotations:  <none>
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind            Name     Namespace
  ----            ----     ---------
  ServiceAccount  default  sys-ingress-pub

The secret with the cert:

Name:         traefik-test-tls-cert
Namespace:    sys-mon
Labels:       <none>
Annotations:  <none>
Type:  kubernetes.io/tls
Data
====
tls.crt:  1034 bytes
tls.key:  1704 bytes

If applicable, please paste the log output at DEBUG level (--logLevel=DEBUG switch)

time="2018-04-04T08:57:49Z" level=warning msg="web provider configuration is deprecated, you should use these options : api, rest provider, ping and metrics"
time="2018-04-04T08:57:49Z" level=info msg="Traefik version v1.6.0-rc3 built on 2018-03-28_04:04:00PM"
time="2018-04-04T08:57:49Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data\n"
time="2018-04-04T08:57:49Z" level=info msg="Preparing server https-proto &{Address::443 TLS:0xc420402280 Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420485b80} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-04-04T08:57:51Z" level=info msg="Preparing server traefik &{Address::8080 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420485f40} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-04-04T08:57:51Z" level=info msg="Starting server on :443"
time="2018-04-04T08:57:51Z" level=info msg="Preparing server http-redirect &{Address::80 TLS:<nil> Redirect:0xc4203ca1c0 Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420485a40} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-04-04T08:57:51Z" level=info msg="Starting server on :8080"
time="2018-04-04T08:57:51Z" level=info msg="Starting provider *kubernetes.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"kubernetes.io/ingress.class=traefik-public\",\"IngressClass\":\"\"}"
time="2018-04-04T08:57:51Z" level=info msg="Starting server on :80"
time="2018-04-04T08:57:51Z" level=info msg="Creating in-cluster Provider client"
time="2018-04-04T08:57:52Z" level=error msg="Error configuring TLS for ingress sys-mon/nginx: secret sys-mon/traefik-test-tls-cert does not exist"
time="2018-04-04T08:57:52Z" level=info msg="Server configuration reloaded on :80"
time="2018-04-04T08:57:52Z" level=info msg="Server configuration reloaded on :443"
time="2018-04-04T08:57:52Z" level=info msg="Server configuration reloaded on :8080"
time="2018-04-04T09:07:51Z" level=warning msg="Error checking new version: GET https://api.github.com/repos/containous/traefik/releases: 403 API rate limit exceeded for 52.18.166.166. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.); rate reset in 20m5.628041774s"
time="2018-04-04T09:07:52Z" level=error msg="Error configuring TLS for ingress sys-mon/nginx: secret sys-mon/traefik-test-tls-cert does not exist"
time="2018-04-04T09:17:52Z" level=error msg="Error configuring TLS for ingress sys-mon/nginx: secret sys-mon/traefik-test-tls-cert does not exist" W0404 09:20:39.482673       1 reflector.go:341] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: watch of *v1.Endpoints ended with: too old resource version: 5095910 (5096509)
time="2018-04-04T09:27:52Z" level=error msg="Error configuring TLS for ingress sys-mon/nginx: secret sys-mon/traefik-test-tls-cert does not exist"
time="2018-04-04T09:37:52Z" level=error msg="Error configuring TLS for ingress sys-mon/nginx: secret sys-mon/traefik-test-tls-cert does not exist" W0404 09:45:27.582778       1 reflector.go:341] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: watch of *v1.Endpoints ended with: too old resource version: 5098162 (5098951)
time="2018-04-04T09:47:52Z" level=error msg="Error configuring TLS for ingress sys-mon/nginx: secret sys-mon/traefik-test-tls-cert does not exist"
time="2018-04-04T09:57:52Z" level=error msg="Error configuring TLS for ingress sys-mon/nginx: secret sys-mon/traefik-test-tls-cert does not exist"
time="2018-04-04T10:07:52Z" level=error msg="Error configuring TLS for ingress sys-mon/nginx: secret sys-mon/traefik-test-tls-cert does not exist"