kubeflow: Cannot access the kubeflow dashboard

@bsamadi New version also have the same problem. kubeflow is not enable when I type microk8s enable.

i wrote a gist for install KF manually in microk8s, working perfectly https://gist.github.com/etheleon/80414516c7fbc7147a5718b9897b1518

@kosehy, I still get the “Invalid Email Address and password.” error message. I am not sure why it acts differently.

@bsamadi Can you type the below command?

microk8s juju config dex-auth static-username
microk8s juju config dex-auth static-password

And what is the output? If the output is empty, then you need to set the username and password as below

microk8s juju config dex-auth static-username=admin
microk8s juju config dex-auth static-password=1234

After that, you need to reopen the brower and go to “http://10.64.140.44.xip.io” If you cannot login, then try to open the “http://10.64.140.44.xip.io” as private window.

It seems that I had a similar problem. Does this solution help? I had to configure the kubeflow hostname so all pods are playing nicely together: https://github.com/ubuntu/microk8s/issues/1366#issuecomment-654411307

referring to: KUBEFLOW_HOSTNAME=your-dns-name-here microk8s.enable kubeflow

For running pipelines on microk8s you have to patch ‘workflow-controller’ to use the pns executor or use the agnostic installation. Not sure if this is already included when using microk8s enable kubeflow: https://github.com/kubeflow/pipelines/issues/4302#issuecomment-672819792

Hi @floleuerer and @kosehy, Could you try setting the KUBECONFIG env?

$ sudo microk8s.kubectl config view --raw > $HOME/.kube/config

$ export KUBECONFIG=$HOME/.kube/config

btw, for pipelines to work you’ll need to change your kubelet args too /var/snap/microk8s/current/args/kubelet

# --container-runtime=remote
# --container-runtime-endpoint=${SNAP_COMMON}/run/containerd.sock
--container-runtime=docker

@etheleon and @kosehy Thanks for writing a procedure for installing kubeflow on microk8s.

The installation works and I can access the dashboard, pipeline, notebook. But when I change kubelet args with what is above. Kubeflow no longer starts when I restart it. I tried with microk8s version 1.18/latest, 1.19/latest, 1.20/latest.

btw, for pipelines to work you’ll need to change your kubelet args too /var/snap/microk8s/current/args/kubelet

# --container-runtime=remote
# --container-runtime-endpoint=${SNAP_COMMON}/run/containerd.sock
--container-runtime=docker

If anyone is struggled with uninstall microk8s and kubeflow and install fresh microk8s, I found a good tutorial from @tomalbrecht’s comment https://github.com/kubeflow/manifests/issues/959#issue-572061859

Step to install fresh microk8s

sudo snap remove microk8s --purge
sudo rm -rf $HOME/.kube

sudo snap install microk8s --classic --channel=1.19/stable

@etheleon I figured the reason why I cannot log in as admin.

I have to use the kfctl_istio_dex.v1.2.0.yaml instead of kfctl_k8s_istio.v1.2.0.yaml After changing the kfctl_k8s_istio.v1.2.0.yaml to dex version, I can log in as admin

I am sorry to make you confuse.

Anyway, thank you for your awesome feedback @etheleon!!

@etheleon I got the same error as @kosehy - microk8s api running on port 16443 and kfctl is trying to reach port 8080. how did you manage to access the api?

I am trying to setup Kubeflow on a bare metal workstation and managed to access the dashboard and pipelines with Microk8s and MetalLB. Unfortunately I couldn’t access the notebooks too.

sudo snap install microk8s --classic --channel=latest/edge
microk8s enable dns dashboard gpu metallb
microk8s enable kubeflow

i wrote a gist for install KF manually in microk8s, working perfectly https://gist.github.com/etheleon/80414516c7fbc7147a5718b9897b1518

@etheleon Thank you for porviding how to install KF manually in microk8s!

however, when I run the commend below:

kfctl apply -V -f $CONFIG_URI

I have face error below

INFO[0003] namespace: kubeflow                           filename="utils/k8utils.go:433"
INFO[0003] Creating namespace: kubeflow                  filename="utils/k8utils.go:438"
Error: failed to apply:  (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize:  (kubeflow.error): Code 400 with message: couldn't create namespace kubeflow Error: Post "http://localhost:8080/api/v1/namespaces": dial tcp 127.0.0.1:8080: connect: connection refused
Usage:
  kfctl apply -f ${CONFIG} [flags]

Flags:
      --context string   Optional kubernetes context to use when applying resources. Currently not used by KFDef resources.
  -f, --file string      Static config file to use. Can be either a local path:
                         		export CONFIG=./kfctl_gcp_iap.yaml
                         	or a URL:
                         		export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.0.yaml
                         		export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_istio_dex.v1.2.0.yaml
                         		export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_aws.v1.2.0.yaml
                         		export CONFIG=https://raw.githubusercontent.com/kubeflow/manifests/v1.2-branch/kfdef/kfctl_k8s_istio.v1.2.0.yaml
                         	kfctl apply -V --file=${CONFIG}
  -h, --help             help for apply
  -V, --verbose          verbose output default is false

kfctl exited with error: failed to apply:  (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize:  (kubeflow.error): Code 400 with message: couldn't create namespace kubeflow Error: Post "http://localhost:8080/api/v1/namespaces": dial tcp 127.0.0.1:8080: connect: connection refused

Have you face this error before?

Here are my kubectl informations:

mlops@mlops-desktop:~/kf-test$ microk8s kubectl get namespaces
NAME              STATUS   AGE
kube-system       Active   36m
kube-public       Active   36m
kube-node-lease   Active   36m
default           Active   36m
mlops@mlops-desktop:~/kf-test$ microk8s kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:16443
CoreDNS is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
mlops@mlops-desktop:~/kf-test$ curl http://localhost:8080/api/
curl: (7) Failed to connect to localhost port 8080: Connection refused

@kosehy, I still get the “Invalid Email Address and password.” error message. I am not sure why it acts differently.