dashboard: 503 ServiceUnavailable

Issue details

I am new to kubernetes and I am trying to set up a basic 2 machine cluster.

The kubernetes-dashboard pod status is listed as pending with 0/1 ready, which is likely the underlying issue. I have no clue where to begin solving this problem. The ultimate result is that I receive 503 ServiceUnavailable in the browser.

Please do not:

recommend using a hosted solution.
recommend minikube.
recommend reading https://github.com/kubernetes/dashboard/issues/971 - I have already been through it, and my problem persists.

Environment

Dashboard version: 

kubernetes-dashboard-amd64:v1.4.0 per https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml

Kubernetes version: 

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Operating system: CentOS 7

Steps to reproduce

Followed getting started at: http://kubernetes.io/docs/getting-started-guides/kubeadm/
a) Created 2 Centos 7.0 VMs on XenServer 7.0 (kube-node-0 [master] and kube-node-1)
installed xauth, xhost, firefox on kube-node-0
ran kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml on kube-node-0
ran kubectl proxy & on kube-node-0
ran firefox & on kube-node-0
navigated to https://localhost/ui
navigated to http://localhost:8001/ui
a) 301 redirected to http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kubernertes-dashboard

Observed result

http response message to https://localhost/ui

401 Unauthorized

http response message to http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kubernertes-dashboard

{
  "kind": "Status",
  "apiVersion": "V1",
  "metadata": {},
  "status": "Failure",
  "message": "no endpoints available for service \"kubernetes-dashboard\"",
  "reason": "ServiceUnavalable",
  "code": 503
}

Expected result

Dashboard UI

Comments

Below is the output of various commands that showed up in https://github.com/kubernetes/dashboard/issues/971

$ kubectl cluster-info
Kubernetes master is running at http://localhost:8080
kube-dns is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

dump file at: https://gist.github.com/kking124/85efed41c107cd84224204435d512632

$ kubectl get nodes
NAME          STATUS     AGE
kube-node-0   Ready      1h
kube-node-1   Ready      1h

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
kube-system   etcd-kube-node-0                        1/1       Running   0          1h
kube-system   kube-apiserver-kube-node-0              1/1       Running   0          1h
kube-system   kube-controller-manager-kube-node-0     1/1       Running   0          1h
kube-system   kube-discovery-982812725-273x4          1/1       Running   0          1h
kube-system   kube-dns-2247936740-qymse               2/3       Running   1          1h
kube-system   kube-proxy-amd64-8o5rr                  1/1       Running   0          1h
kube-system   kube-proxy-amd64-mg0ic                  1/1       Running   0          1h
kube-system   kube-scheduler-kube-node-0              1/1       Running   0          1h
kube-system   kubernetes-dashboard-1655269645-ekxxs   0/1       Pending   0          52m
kube-system   weave-net-9i5ce                         2/2       Running   0          59m
kube-system   weave-net-y9iu2                         2/2       Running   0          59m

$ kubectl get pods

$ kubectl describe service kubernetes-dashboard --namespace=kube-system
Name:                   kubernetes-dashboard
Namespace:              kube-system
Labels:                 app=kubernetes-dashboard
Selector:               app=kubernetes-dashboard
Type:                   NodePort
IP:                     100.73.132.23
Port:                   <unset> 80/TCP
NodePort:               <unset> 30526/TCP
Endpoints:              <none>
Session Affinity:       None

$ kubectl --namespace=kube-system get ep kubernetes-dashboard
NAME                   ENDPOINTS   AGE
kubernetes-dashboard   <none>      1h

$ kubectl get svc kubernetes-dashboard --namespace=kube-system
NAME                   CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes-dashboard   100.73.132.23   <nodes>       80/TCP    1h

$ kubectl get deployment kubernetes-dashboard --namespace=kube-system
NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kubernetes-dashboard   1         1         1            0           1h

$ kubectl logs kubernetes-dashboard-1655269645-ekxxs --namespace kube-system

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 30 (12 by maintainers)

Most upvoted comments

I see this issue too with kubernetes 1.5.4 and kubernetes-dashboard image version gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0.

I installed kubeadm referring https://kubernetes.io/docs/getting-started-guides/kubeadm/, and then installed kubernetes-dashboard by doing

kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.6.0/src/deploy/kubernetes-dashboard.yaml

I see the kubernetes-dashboard in CrashLoopBackOff status and the k8s_kubernetes-dashboard.* container on the worker is in Exited state.

Below are the errors. Has anyone successfully installed kubernetes-dashboard on kubeadm?

# kubectl --namespace=kube-system get all
NAME                                                          READY     STATUS             RESTARTS   AGE
po/calico-policy-controller-mqsmh                             1/1       Running            0          4h
po/canal-etcd-tm2rv                                           1/1       Running            0          4h
po/canal-node-3nv2t                                           3/3       Running            0          4h
po/canal-node-5fckh                                           3/3       Running            1          4h
po/canal-node-6zgq8                                           3/3       Running            0          4h
po/canal-node-rtjl8                                           3/3       Running            0          4h
po/dummy-2088944543-09w8n                                     1/1       Running            0          4h
po/etcd-vhosakot-kolla-kube1.localdomain                      1/1       Running            0          4h
po/kube-apiserver-vhosakot-kolla-kube1.localdomain            1/1       Running            2          4h
po/kube-controller-manager-vhosakot-kolla-kube1.localdomain   1/1       Running            0          4h
po/kube-discovery-1769846148-pftx5                            1/1       Running            0          4h
po/kube-dns-2924299975-9m2cp                                  4/4       Running            0          4h
po/kube-proxy-0ndsb                                           1/1       Running            0          4h
po/kube-proxy-h7qrd                                           1/1       Running            1          4h
po/kube-proxy-k6168                                           1/1       Running            0          4h
po/kube-proxy-lhn0k                                           1/1       Running            0          4h
po/kube-scheduler-vhosakot-kolla-kube1.localdomain            1/1       Running            0          4h
po/kubernetes-dashboard-3203962772-mw26t                      0/1       CrashLoopBackOff   11         41m
NAME                       CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
svc/canal-etcd             10.96.232.136    <none>        6666/TCP        4h
svc/kube-dns               10.96.0.10       <none>        53/UDP,53/TCP   4h
svc/kubernetes-dashboard   10.100.254.77    <nodes>       80:30085/TCP    41m
NAME                   DESIRED   SUCCESSFUL   AGE
jobs/configure-canal   1         1            4h
NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/kube-discovery         1         1         1            1           4h
deploy/kube-dns               1         1         1            1           4h
deploy/kubernetes-dashboard   1         1         1            0           41m
NAME                                 DESIRED   CURRENT   READY     AGE
rs/calico-policy-controller          1         1         1         4h
rs/dummy-2088944543                  1         1         1         4h
rs/kube-discovery-1769846148         1         1         1         4h
rs/kube-dns-2924299975               1         1         1         4h
rs/kubernetes-dashboard-3203962772   1         1         0         41m

# kubectl --namespace=kube-system describe pod kubernetes-dashboard-3203962772-mw26t
  20m    5s    89    {kubelet vhosakot-kolla-kube2.localdomain}                        Warning    FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "kubernetes-dashboard" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=kubernetes-dashboard pod=kubernetes-dashboard-3203962772-mw26t_kube-system(67b0d69b-0b47-11e7-8c97-7a2ed4192438)"

# kubectl --namespace=kube-system logs kubernetes-dashboard-3203962772-mw26t
Using HTTP port: 9090
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

# docker ps -a | grep -i dash
3c33cf43d5e4        gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.0   "/dashboard --port=90"   54 seconds ago      Exited (1) 22 seconds ago                       k8s_kubernetes-dashboard.9eb4d80e_kubernetes-dashboard-3203962772-mw26t_kube-system_67b0d69b-0b47-11e7-8c97-7a2ed4192438_93520bd4

# docker logs k8s_kubernetes-dashboard.9eb4d80e_kubernetes-dashboard-3203962772-mw26t_kube-system_67b0d69b-0b47-11e7-8c97-7a2ed4192438_93520bd4
Using HTTP port: 9090
Creating API server client for https://10.96.0.1:443
Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout
Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

vhosakot on Mar 20, 2017

@justinsb you’ve been super helpful.

Please never help me again. I promise I’ll never help you.

kking124 on Oct 8, 2016