kubeflow: Kubeflow deployment fails on kfctl apply

kind bug

What steps did you take and what happened: i am using the kubeflow deployment guide here https://www.kubeflow.org/docs/started/k8s/kfctl-k8s-istio/ for deploying kubeflow on Mac.

The step : echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/go/bin:/Users/xxxx/Desktop/know/ml/minikube/ echo $KFAPP demo echo $CONFIG https://raw.githubusercontent.com/kubeflow/kubeflow/v0.6.1/bootstrap/config/kfctl_k8s_istio.yaml kfctl init ${KFAPP} --config=${CONFIG} -V cd ${KFAPP} kfctl generate all -V kfctl apply all -V

Generates the below error

INFO[0031] creating StatefulSet/application-controller-stateful-set filename=“kustomize/kustomize.go:440” Error: couldn’t apply KfApp: (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize: (kubeflow.error): Code 500 with message: couldn’t create resources from application Error: no matches for kind “Application” in version “app.k8s.io/v1beta1” Usage: kfctl apply [all(=default)|k8s|platform] [flags]

Flags: -h, --help help for apply -V, --verbose verbose output default is false

couldn’t apply KfApp: (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize: (kubeflow.error): Code 500 with message: couldn’t create resources from application Error: no matches for kind “Application” in version “app.k8s.io/v1beta1”

What did you expect to happen: Expecte kubeflow deployment successful on my cluster

Environment:

Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard):
kfctl version: (use kfctl version): kfctl v0.6.1-rc.2-1-g3a37cbc6
Kubernetes platform: (e.g. minikube): minikube
Kubernetes version: (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“15”, GitVersion:“v1.15.2”, GitCommit:“f6278300bebbb750328ac16ee6dd3aa7d3549568”, GitTreeState:“clean”, BuildDate:“2019-08-05T16:57:42Z”, GoVersion:“go1.12.7”, Compiler:“gc”, Platform:“darwin/amd64”} Server Version: version.Info{Major:“1”, Minor:“15”, GitVersion:“v1.15.2”, GitCommit:“f6278300bebbb750328ac16ee6dd3aa7d3549568”, GitTreeState:“clean”, BuildDate:“2019-08-05T09:15:22Z”, GoVersion:“go1.12.5”, Compiler:“gc”, Platform:“linux/amd64”}
OS (e.g. from /etc/os-release):uname -a Darwin XXX-MacBook-Pro.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 29 (18 by maintainers)

Most upvoted comments

Hi @jlwei yes, the issue is fully resolved.

Thank you.

praveen049 on Oct 14, 2019

@hemantha-kumara

yes, the problem was with the proxy. Now the image download are ok. Thanks

praveen049 on Oct 10, 2019

@praveen049 From the log, all seems fine(docker is getting pulled as there is no image pull off error), may be proxy is slow, hence it is taking time. You can try pulling docker manually and see how fast image is getting pulled. docker pull gcr.io/kubeflow-images-public/kubernetes-sigs/application:1.0-beta

hemantha-kumara on Oct 8, 2019

@praveen049 you are hitting a race condition as mentioned above. If you want to deploy with kfctl v0.6.2 you will need to do the following

Follow the KF instructions to run kfctl apply which will fail with the above errors
Create an apply.sh with the following contents

#!/usr/bin/env bash

for i in $(cat app.yaml | grep '^        path: ' | awk -F: '{print $2}');do
  childdir=${i#*/}
  if [[ -d kustomize/$childdir && -f kustomize/$childdir/kustomization.yaml ]]; then
    kustomize build kustomize/$childdir | kubectl apply -f -
    sleep 3
  fi
done

chmod +x apply.sh
move the script to where you called kfctl generate.
Run the script from that location

Ref https://github.com/kubeflow/kubeflow/issues/3810#issuecomment-527687504

Long term fix will be to use kfctl 0.7 which will fix the issues with rerunning kfctl apply.

jlewi on Oct 5, 2019