kubeflow: Kubeflow on K8s - "kfctl apply" throws "Unknown variable: x" error ; ksonnet doesn't support K8s 1.15 or later

I tried to install kubeflow v0.5.1 on an existing K8s “cluster” (it is a single node microk8s cluster on an Ubuntu Host) by following the instructions from this page: https://www.kubeflow.org/docs/started/getting-started-k8s/. At least for me the instructions seem to not work.

I am trying to deploy kubeflow from my laptop onto the microk8s “clsuter” which runs on a dedicated host.

However, the last command from the instructions - kfctl apply all -V - outputs just a list of errors. Except for the namespace creation nothing is deployed/created. Below an excerpt of the error which repeats itself and you can find the full output here.

INFO[0000] deploying kubeflow application                filename="cmd/apply.go:35"
INFO[0000] reading from /Users/.../cluster-init/kubeflow/kfapp/app.yaml  filename="coordinator/coordinator.go:341"
INFO[0000] reading from /Users/.../cluster-init/kubeflow/kfapp/app.yaml  filename="coordinator/coordinator.go:341"
INFO[0000] namespace: kubeflow                           filename="ksonnet/ksonnet.go:109"
ERRO[0004] (Will retry) Component ambassador apply failed; Error: find objects: /Users/.../cluster-init/kubeflow/kfapp/ks_app/lib/ksonnet-lib/v1.15.0/k8s.libsonnet:17681:93-94 Unknown variable: x

          "withX-Kubernetes-Embedded-Resource":: self + { "x-kubernetes-embedded-resource": x-kubernetes-embedded-resource },
  filename="ksonnet/ksonnet.go:174"
ERRO[0007] (Will retry) Component argo apply failed; Error: find objects: /Users/.../cluster-init/kubeflow/kfapp/ks_app/lib/ksonnet-lib/v1.15.0/k8s.libsonnet:17681:93-94 Unknown variable: x


[...]

Used kfctl version

$ kfctl version
v20181207-4e7f4ed-198-gaeea303e-dirty-03e65e

Used ks version

$ ks version
ksonnet version: 0.13.1
jsonnet version: v0.11.2
client-go version: kubernetes-1.10.4

I slightly changed the commands so i do not have to modify the PATH/set the KFAPP variable:

./kfctl init mykfapp
cd mykfapp
../kfctl generate all -V
../kfctl apply all -V

Additionally I am on macOS Mojave using zsh as shell.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 11
Comments: 15 (5 by maintainers)

Most upvoted comments

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.87. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

+14

issue-label-bot[bot] on Jun 25, 2019

@rky0930 for now you can use an older kubernetes version such as 1.14.3 or 1.14.1 (that are the versions I have tested successfully). For the rest you can just follow the official installation instructions and use the latest version of kubeflow and ksonnet/ks.

sven-petersen on Jul 1, 2019

Is there a timeline on when that will be happening? Because I’ve been trying to deploy Kubeflow on Minikube for the last 2 days and I’m running into issue after issue, this being one of them.

porterdarby on Jun 26, 2019

#3559 is the tracking bug for 0.6.

We are working on cutting the RC now.

If you are willing to live on the edge and be a beta tester, we would really appreciate help trying out master and reporting bugs.

We are using #release in kubeflow.slack.com to coordinate.

To try on master check out master and then

cd bootstrap
make build-kfctl

jlewi on Jul 1, 2019

Are you using kubernetes 1.15?

It looks like the error

ERRO[0007] (Will retry) Component argo apply failed; Error: find objects: /Users/.../cluster-init/kubeflow/kfapp/ks_app/lib/ksonnet-lib/v1.15.0/k8s.libsonnet:17681:93-94 Unknown variable: x

Is coming from the libsonnet file that ksonnet is auto-generating. I suspect there is an error specific to the 1.15 APis.

Unfortunately since ksonnet is deprecated the fix is probably going to be to wait for 0.6 which replaces ksonnet with kustomize.

jlewi on Jun 26, 2019