kubeflow: Looping "Could not find namespace kubeflow-anonymous, wait and retry" on initial apply

You can create kubeflow-anonymous namespace in advance.

kubectl create namesapce kubeflow-anonymous

Here’s how I was able to resolve this one on my end.

If ever you have already applied do this:

1. I deleted all my Kubeflow clusters

$ kfctl delete all --delete_storage

1. Deleted the istio-system namespace from Kubernetes — so that the error on Port already being used won’t happen

$ kubectl delete namespaces istio-system

1. Deleted the path I initialized

$ rm -rf <kapp_path> 

1. Created a namespace kubeflow-anonymous

$ kubectl create namespace kubeflow-anonymous

1. Reinitialized, re-generated and re-applied

$ kfctl init ${KFAPP} --config=${CONFIG} -V
$ cd ${KFAPP}
$ kfctl generate all -V
$ kfctl apply all -V

1. Verified if working

$ kubectl get pods -n kubeflow

If ever you’re just going to start initializing, download the config files first and then add the kubeflow-anonymous namespace.

Hope this helps someone. Props to: @wzxJayce

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.99. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

same issue:

• kfctl: v0.6.2-0-g47a0e4c7

E0916 12:32:59.188756   13066 memcache.go:135] couldn't get resource list for machinelearning.seldon.io/v1alpha2: the server could not find the requested resource
INFO[0012] creating Profile/kubeflow-anonymous           filename="kustomize/kustomize.go:447"
WARN[0012] Could not find namespace kubeflow-anonymous, wait and retry: namespaces "kubeflow-anonymous" not found  filename="kustomize/kustomize.go:353"
WARN[0016] Could not find namespace kubeflow-anonymous, wait and retry: namespaces "kubeflow-anonymous" not found  filename="kustomize/kustomize.go:353"
WARN[0022] Could not find namespace kubeflow-anonymous, wait and retry: namespaces "kubeflow-anonymous" not found  filename="kustomize/kustomize.go:353"
WARN[0030] Could not find namespace kubeflow-anonymous, wait and retry: namespaces "kubeflow-anonymous" not found  filename="kustomize/kustomize.go:353"
WARN[0039] Could not find namespace kubeflow-anonymous, wait and retry: namespaces "kubeflow-anonymous" not found  filename="kustomize/kustomize.go:353"
WARN[0053] Could not find namespace kubeflow-anonymous, wait and retry: namespaces "kubeflow-anonymous" not found  filename="kustomize/kustomize.go:353"

kubectl create namesapce kubeflow-anonymous

@wzxJayce , you saved my day

Is there any new info on this? I am also seeing the same error running on an EKS cluster on AWS. I am following the steps here: https://www.kubeflow.org/docs/aws/deploy/install-kubeflow/ . I am seeing the error occur on the “apply” step: kfctl apply all -V. And it does not appear to be a warning. The process does eventually terminate with the following error after retrying for about 15 mins.

couldn’t apply KfApp: (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize: (kubeflow.error): Code 400 with message: Could not find namespace kubeflow-anonymous, wait and retry: Get https://2D737A4E4A13EBC879857AB7571B0E38.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/kubeflow-anonymous: write tcp 10.0.0.250:50495->3.229.9.250:443: write: can’t assign requested address

@gar1t I believe that you saw this message during apply, not generate. Can you confirm?

This is more of a UX issue than a bug. The kfctl code is waiting for the default namespace to be created before exiting.

However the code should be of LogLevel info and not warn, since it is not an abnormal condition.

In addition, I feel that the wording should be different to convey that this is a normal part of the installation. The current wording makes it seem like there is something wrong. How about: Waiting for default Profile namespace (kubeflow-anonymous) to be created...