traefik: IngressRoute subset not found for kube-system/kubernetes-dashboard

Do you want to request a feature or report a bug?

Bug

What did you do?

• installed traefik2 with helm from here traefik-helm-chart (namespace:kube-system)
• check: IngressRoute successful installed
• check: IngressRoute traefik-dashboard is routable
• try to install kubernetes dashboard with helm from here kubernetes-dashboard chart (namespace:kube-system)
• apply this ingressRoute

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name:  kubernetes-dashboard
spec:
  entryPoints:
    - web
  routes:
  - match: PathPrefix(`/ui`)
    kind: Rule
    middlewares:
    - name: strip-ui
    services:
    - name:  kubernetes-dashboard
      port: 443

----
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: strip-ui
spec:
  stripPrefix:
    prefixes:
      - "/ui"
      - "/ui/"

Try browsing of ‘MY_IP/ui’

# kubectl logs traefik-pod -f
level=error msg="subset not found for kube-system/kubernetes-dashboard" providerName=kubernetescrd ingress=kubernetes-dashboard namespace=kube-system

What did you expect to see?

Dashboard UI in browser

What did you see instead?

• Internal Server Error in browser

Output of traefik version: version:2

# traefik pod log
level=error msg="subset not found for kube-system/kubernetes-dashboard" providerName=kubernetescrd ingress=kubernetes-dashboard namespace=kube-system` in the traefik pod log

What is your environment & configuration (arguments, toml, provider, platform, …)?

# No toml 
# - thats my helm command `helm upgrade --install kubernetes-dashboard stable/kubernetes-dashboard -n kube-system --set enableInsecureLogin=true`
# - check Vagrant sample below for details

• k3s as single host kubernetes without default traefik1
• helm3 installed
• host system: debian10

# -*- mode: ruby -*-
Vagrant.configure("2") do |config|
  #config.vm.box = "ubuntu/bionic64"

  config.vm.box = "debian/buster64"
  config.vm.box_version = "10.3.0"

  config.vm.network "private_network", ip: "172.28.129.23"
  #config.vm.network "private_network", type: "dhcp"
  #config.vm.network "public_network", bridge: "en0: WLAN (AirPort)"

  config.vm.synced_folder ".", "/usr/share/whoami-charts"

  config.vm.provider "virtualbox" do |vb|
     vb.memory = "1024"
     vb.cpus = 1
   end

  config.vm.provision "shell", inline: <<SHELL
  # To prevent `k3s check-config`
  # /usr/sbin iptables v1.8.2 (nf_tables): should be older than v1.8.0 or in legacy mode (fail)
  update-alternatives --set iptables /usr/sbin/iptables-legacy

  apt-get update

  # install curl - if not exist
  # - https://www.cyberciti.biz/faq/howto-install-curl-command-on-debian-linux-using-apt-get/
  if ! curl --version > /dev/null ; then
    apt --yes install curl
  else
    echo ">> Command 'curl' already installed."
  fi

  # install k3s - if not exist (without default  traefik v1)
  # - https://rancher.com/docs/k3s/latest/en/installation/install-options/
  # - without default traefik - https://github.com/rancher/k3s/issues/1160
  if ! k3s -v > /dev/null ; then
    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --no-deploy traefik" sh -
  else
    echo ">> Command 'k3s' already installed."
  fi

  # install helm - if not exist
  if ! helm version > /dev/null ; then
    curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
    export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
    echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /home/vagrant/.bashrc
    echo "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> /root/.bashrc
  else
    echo ">> Command 'helm' already installed."
  fi

  # Ensure that traefik(v2) is deployed
  # - https://github.com/containous/traefik-helm-chart/tree/master
  # - https://docs.traefik.io/user-guides/crd-acme/ (traefik-doc: +1)
  helm repo add traefik https://containous.github.io/traefik-helm-chart
  helm repo update
  helm upgrade --install traefik traefik/traefik --namespace kube-system --set additionalArguments={--log.level=DEBUG}

  # Bind traefik /dashboard and /api on entryPoint:web
  cat << 'EOF' >> /tmp/traefik-dashboard.yaml
   # traefik-dashboard.yaml
   apiVersion: traefik.containo.us/v1alpha1
   kind: IngressRoute
   metadata:
    name: traefik-dashboard
   spec:
    entryPoints:
      - web
    routes:
      - match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
        kind: Rule
        services:
          - name: api@internal
            kind: TraefikService
EOF

  kubectl delete IngressRoute -n kube-system traefik-dashboard
  kubectl apply -f /tmp/traefik-dashboard.yaml

  helm repo add stable https://kubernetes-charts.storage.googleapis.com
  helm repo update
  helm upgrade --install kubernetes-dashboard stable/kubernetes-dashboard -n kube-system --set enableInsecureLogin=true
  
  cat << 'EOF' >> /tmp/kubernetes-dashboard-ingressroute.yaml
    apiVersion: traefik.containo.us/v1alpha1
    kind: IngressRoute
    metadata:
      name:  kubernetes-dashboard
    spec:
      entryPoints:
        - web
      routes:
      - match: PathPrefix(`/ui`)
        kind: Rule
        middlewares:
        - name: strip-ui
        services:
        - name:  kubernetes-dashboard
          port: 443
EOF

  kubectl apply -f /tmp/kubernetes-dashboard-ingressroute.yaml -n kube-system

  cat << 'EOF' >> /tmp/kubernetes-dashboard-middleware.yaml
    apiVersion: traefik.containo.us/v1alpha1
    kind: Middleware
    metadata:
      name: strip-ui
    spec:
      stripPrefix:
        prefixes:
          - "/ui"
          - "/ui/"
EOF

  kubectl apply -f /tmp/kubernetes-dashboard-middleware.yaml -n kube-system
SHELL
end

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

time="2020-05-19T09:46:44Z" level=debug msg="Skipping Kubernetes event kind *v1.Endpoints" providerName=kubernetes
time="2020-05-19T09:46:44Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36\"],\"X-Forwarded-Host\":[\"172.28.129.23\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Prefix\":[\"/ui\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-7697b4bb9c-qxfv8\"],\"X-Real-Ip\":[\"10.42.0.14\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"172.28.129.23\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.14:56382\",\"RequestURI\":\"/\",\"TLS\":null}"
time="2020-05-19T09:46:44Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36\"],\"X-Forwarded-Host\":[\"172.28.129.23\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Prefix\":[\"/ui\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-7697b4bb9c-qxfv8\"],\"X-Real-Ip\":[\"10.42.0.14\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"172.28.129.23\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.14:56382\",\"RequestURI\":\"/\",\"TLS\":null}" ForwardURL="https://10.42.0.18:9090"
time="2020-05-19T09:46:44Z" level=debug msg="'500 Internal Server Error' caused by: tls: first record does not look like a TLS handshake"
time="2020-05-19T09:46:44Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\"],\"Accept-Encoding\":[\"gzip, deflate\"],\"Accept-Language\":[\"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7\"],\"Cache-Control\":[\"max-age=0\"],\"Connection\":[\"keep-alive\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36\"],\"X-Forwarded-Host\":[\"172.28.129.23\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Prefix\":[\"/ui\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"traefik-7697b4bb9c-qxfv8\"],\"X-Real-Ip\":[\"10.42.0.14\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"172.28.129.23\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"10.42.0.14:56382\",\"RequestURI\":\"/\",\"TLS\":null}"