kubeflow: kfctl_istio_dex manifest fails

/kind bug

What steps did you take and what happened: Install kubeflow using kfctl_istio_dex manifest from here: https://raw.githubusercontent.com/kubeflow/manifests/master/kfdef/kfctl_istio_dex.yaml

kfctl runs successfully but pods in istio-system namespace won’t start with:

MountVolume.SetUp failed for volume "istio-certs" : couldn't propagate object cache: timed out waiting for the condition

What did you expect to happen: Kubeflow installed successfully

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): 1.0
kfctl version: (use kfctl version): 1.0.0-rc3
Kubernetes platform: (e.g. minikube) AWS kops
Kubernetes version: (use kubectl version): 1.15.1
OS (e.g. from /etc/os-release): Ubuntu 18.04

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 16 (7 by maintainers)

Most upvoted comments

@yanniszark well, that sounds like a lot of work. I hoped I would just install kubeflow without having to do any plumbing. I understand it is a complex system and I respect that. The only thing I wanted was to show sandbox vanilla kubeflow to my colleagues. And I am struggling bad with just that. On the other hand, I am adaptable person and I can improvise. I am running out of options though:

Installing kubeflow on Docker for Desktop - does not work due to memory limitations. ok.
Installing kubeflow using kfctl_istio - useless because of #4731 3 installing kubeflow using kfctl_istio_dex - istio screwed up and they will fix it in upcoming version. What I don’t uderstand is why kubeflow had to choose the very screwed up version and become hostage of it.

Are there any other options for me?

mfojtak on Feb 6, 2020

@mfojtak your concerns are valid. We might look to upgrade Istio in the near future; i.e. subsequent releases which will not require us to set these flags.

krishnadurai on Mar 4, 2020

I managed to solve this by following steps:

installed Istio 1.4.3
installed kfctl_istio_dex.yaml with istio-crds and istio-install removed and set clusterRbacConfig to OFF

Now all works and it also solves #4721 Should upgrade to Istio 1.4.3 be taken into consideration for 1.0 release?

mfojtak on Feb 9, 2020

@mfojtak I should mention here that since all config files are using the same Istio version, you will encounter this issue on all of them.

I can see many possible solutions:

Use MiniKF. This is one of the easiest ways to get started and explore Kubeflow without caring too much about the installation procedure.
Use v0.7 kfctl_existing_arrikto which uses an older Istio version and doesn’t require setting up trusted JWTs. (https://www.kubeflow.org/docs/started/k8s/kfctl-existing-arrikto/)
Use a Kubernetes cluster that allows you to have access to the trusted JWTs feature.

At this point, I would also like to point out that there isn’t a v1.0 release yet, so if you try RC branches you can expect that something might be broken. For a stable release, try v0.7 for now.

yanniszark on Feb 6, 2020