kubeflow: error during apply filename="kustomize/kustomize.go", leads to Code 500 with message: kfApp Apply failed for kustomize
/kind bug
What steps did you take and what happened: attempted to install kubeflow per these instructions -> https://www.kubeflow.org/docs/started/k8s/kfctl-k8s-istio/
I am trying to install v0.7.0-rc.5 because I ran in to this bug: https://github.com/kubeflow/kubeflow/issues/4199 and the fix for that only seems to be available in v0.7.0 release candidates.
After I run
kfctl apply -V -f ${CONFIG_URI}
I see continued errors and attempts to retry, as shown in the section ERROR AND RETRY. After about 10 retry attempts the installation finally fails with this message:
(kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize — Error unable to recognize “/tmp/kout500167443”: no matches for kind “Deployment” in version “apps/v1beta2”
Any guidance most gratefully appreciated.
ERROR AND RETRY
WARN[0339] Encountered error during apply: <nil> filename=“kustomize/kustomize.go:183” WARN[0339] Will retry in 21 seconds. filename=“kustomize/kustomize.go:184” serviceaccount/metadata-ui unchanged role.rbac.authorization.k8s.io/metadata-ui unchanged rolebinding.rbac.authorization.k8s.io/metadata-ui unchanged configmap/metadata-ui-parameters-b6c8ghff7c unchanged secret/metadata-db-secrets unchanged service/metadata-db unchanged service/metadata-envoy-service unchanged service/metadata-grpc-service unchanged service/metadata-service unchanged service/metadata-ui unchanged deployment.apps/metadata-db unchanged deployment.apps/metadata-deployment unchanged deployment.apps/metadata-grpc-deployment unchanged deployment.apps/metadata-ui unchanged virtualservice.networking.istio.io/metadata-grpc unchanged virtualservice.networking.istio.io/metadata-ui unchanged persistentvolumeclaim/metadata-mysql unchanged WARN[0366] Encountered error during apply: <nil> filename=“kustomize/kustomize.go:183” WARN[0366] Will retry in 41 seconds. filename=“kustomize/kustomize.go:184”
What did you expect to happen: was hoping for a clean install… didn’t succeed ;^(
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
Environment:
-
Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): v0.7.0-rc.5
-
kfctl version: (use
kfctl version): kfctl v0.7.0-rc.5-0-g18fba77a -
Kubernetes platform: (e.g.
minikube) minikube version: v1.4.0 commit: 7969c25a98a018b94ea87d949350f3271e9d64b6 -
Kubernetes version: (use
kubectl version): Client Version: version.Info{Major:“1”, Minor:“16”, GitVersion:“v1.16.2”, GitCommit:“c97fe5036ef3df2967d086711e6c0c405941e14b”, GitTreeState:“clean”, BuildDate:“2019-10-15T19:18:23Z”, GoVersion:“go1.12.10”, Compiler:“gc”, Platform:“darwin/amd64”} Server Version: version.Info{Major:“1”, Minor:“16”, GitVersion:“v1.16.0”, GitCommit:“2bd9643cee5b3b3a5ecbd3af49d09018f0773c77”, GitTreeState:“clean”, BuildDate:“2019-09-18T14:27:17Z”, GoVersion:“go1.12.9”, Compiler:“gc”, Platform:“linux/amd64”} -
OS (e.g. from
/etc/os-release):
MacOS mojave 10.14.5
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 6
- Comments: 27 (3 by maintainers)
The retry never succeeds.
I did only downgrade kubectl, and nothing to do with my cluster since I’m not deploying to an existing one.
I’m just using the config file mentioned in the Deploy using CLI https://www.kubeflow.org/docs/gke/deploy/deploy-cli/ and that creates a new cluster for me. So shouldn’t it be compatible with my versions of
kubectlandkfctl? Or do I need to do something else?This is the config file I am using:
"https://raw.githubusercontent.com/kubeflow/manifests/v0.7-branch/kfdef/kfctl_gcp_iap.0.7.0.yaml"I’m a novice to Kubeflow/K8s so I may be doing something wrong without realising it, but really I’m only following the instructions in the Kubeflow docs and it has never worked thus far. Any guidance would be much appreciated.
Same here on
INFO[0000] Deleting cachedir /opt/kubeflow/kubeflow-poc/.cache/manifests because Status.ReposCache is out of date filename="kfconfig/types.go:464" INFO[0000] Fetching https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz to /opt/kubeflow/kubeflow-poc/.cache/manifests filename="kfconfig/types.go:485" Error: failed to build kfApp from URI /opt/kubeflow/kubeflow-poc/kfctl_existing_arrikto.yaml: couldn't generate KfApp: (kubeflow.error): Code 500 with message: could not sync cache. Error: (kubeflow.error): Code 400 with message: couldn't download URI https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gz Error Error opening a gzip reader for /tmp/getter084554824/archive: EOFI have no Idea about Kubernetes and kubeflow either. I just followed the install instructions for 0.7.1. But I found a workaround, so it will deploy on my machine.
cd ${KF_DIR}vi kfctl_existing_arrikto.yaml--> replacehttps://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gzwith a local directory/tmp/manifests-0.7-branchcd /tmpwget https://github.com/kubeflow/manifests/archive/v0.7-branch.tar.gztar -xvf v0.7-branch.tar.gzcd ${KF_DIR}kfctl apply -V -f ${CONFIG_FILE}that’s just a workaround, but works for me.
not resolved in kfctl v1.0.2 using it with Kubeadm
This error should be solved in kubeflow v1.0.1. I was able to install kubeflow 1.0.1 without this error on microk8s v1.15.
Issue-Label Bot is automatically applying the label
kind/bugto this issue, with a confidence of 0.99. Please mark this comment with 👍 or 👎 to give our bot feedback!Links: app homepage, dashboard and code for this bot.
Same error as above^ but the work around did not help. I am simply trying to follow the same instructions as the original post with microk8s
The error I am getting is