kubeflow: Running kfctl apply k8s fails on 2nd application

/kind bug

What steps did you take and what happened: I ran kfctl apply k8s a 2nd time and it failed saying istio had already bound the port.

output:

time=“2019-08-01T14:24:56-07:00” level=info msg=“creating Service/istio-galley\n” filename=“kustomize/kustomize.go:440” time=“2019-08-01T14:24:56-07:00” level=info msg=“creating Service/istio-ingressgateway\n” filename=“kustomize/kustomize.go:440” Error: couldn’t apply KfApp: (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize: (kubeflow.error): Code 500 with message: couldn’t create resources from istio-install Error: Service “istio-ingressgateway” is invalid: spec.ports[1].nodePort: Invalid value: 31380: provided port is already allocated Usage: kfctl apply [all(=default)|k8s|platform] [flags]

Flags: -h, --help help for apply -V, --verbose verbose output default is false

couldn’t apply KfApp: (kubeflow.error): Code 500 with message: kfApp Apply failed for kustomize: (kubeflow.error): Code 500 with message: couldn’t create resources from istio-install Error: Service “istio-ingressgateway” is invalid: spec.ports[1].nodePort: Invalid value: 31380: provided port is already allocated

What did you expect to happen:

Since the docs https://www.kubeflow.org/docs/gke/customizing-gke/ suggest that I can reapply, I expected this to silently skip istio or otherwise not fail.

Anything else you would like to add: Short term solution could be a doc change, longer term I think we do want to support kfctl apply k8s multiple times.

Environment:

  • Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): master (df2b68d92cf15374b1706d3d91b8123972b8920c)
  • kfctl version: (use kfctl version): v20181207-4e7f4ed-198-gaeea303e-dirty-03e65e
  • Kubernetes platform: (e.g. minikube) gke
  • Kubernetes version: (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“15”, GitVersion:“v1.15.0”, GitCommit:“e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529”, GitTreeState:“clean”, BuildDate:“2019-06-19T16:40:16Z”, GoVersion:“go1.12.5”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“12+”, GitVersion:“v1.12.9-gke.7”, GitCommit:“b6001a5d99c235723fc19342d347eee4394f2005”, GitTreeState:“clean”, BuildDate:“2019-06-18T12:12:30Z”, GoVersion:“go1.10.8b4”, Compiler:“gc”, Platform:“linux/amd64”}
  • OS (e.g. from /etc/os-release): NAME=“Ubuntu” VERSION=“18.04.2 LTS (Bionic Beaver)” ID=ubuntu ID_LIKE=debian PRETTY_NAME=“Ubuntu 18.04.2 LTS” VERSION_ID=“18.04” HOME_URL=“https://www.ubuntu.com/” SUPPORT_URL=“https://help.ubuntu.com/” BUG_REPORT_URL=“https://bugs.launchpad.net/ubuntu/” PRIVACY_POLICY_URL=“https://www.ubuntu.com/legal/terms-and-policies/privacy-policy” VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 5
  • Comments: 40 (34 by maintainers)

Commits related to this issue

Most upvoted comments

@kkasravi suggested using the following work around until the fix is released in kfctl 0.7.

Create an apply.sh with the following contents

#!/usr/bin/env bash

for i in $(cat app.yaml | grep '^        path: ' | awk -F: '{print $2}');do
  childdir=${i#*/}
  if [[ -d kustomize/$childdir && -f kustomize/$childdir/kustomization.yaml ]]; then
    kustomize build kustomize/$childdir | kubectl apply -f -
    sleep 3
  fi
done

Then

  1. chmod +x apply.sh
  2. move the script to where you called kfctl generate.
  3. Run the script from that location
+5
jlewi on Sep 4, 2019

Has it ever been considered to shell out to kubectl apply ... rather than have client-go code in kfctl that talks to the k8s api-server directly?

+3
mattnworb on Aug 9, 2019

We should consider running kfctl apply twice in our E2E tests as a way to verify that it works as expected.

+3
jlewi on Aug 5, 2019

@kkasravi Yes I think we should change the behavior. as @holdenk run kfctl apply should be idempotent if everything is already deployed correctly.

+3
jlewi on Aug 5, 2019

there has been some prototyping related to kustomize plugins where most of kfctl can be embedded within kustomize. Given the fact that kustomize is part of kubectl (https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/) there is a pathway where kfctl could mostly if not entirely disappear - with parts being integrated into kubectl via kustomize plugins. At this point we wouldn’t need to shell out since we’re already ‘in’ 😃

+2
kkasravi on Aug 9, 2019

@ashahba we should give it a try, it would be great if so.

+1
kkasravi on Aug 10, 2019

@jlewi @kkasravi once server-side apply becomes available we should use that. Until then, any ideas about what is the best practice available to have a behaviour close to kubectl apply?

+1
yanniszark on Aug 7, 2019

@mattnworb yes that’s right.

@jlewi @kunming @yanniszark should we change this behavior - effectively an apply instead of create?

+1
kkasravi on Aug 2, 2019

Should be dup of https://github.com/kubeflow/kubeflow/issues/3686

@abhi-g also ran into this recently I think. It’s a istio issue: https://github.com/istio/istio/issues/10795. The workaround as pointed out in that issue is to edit the manifest (istio-install) and comment out the specified port number.

We should find a way in kfctl to get around this.

+1
lluunn on Aug 2, 2019
Read more comments on GitHub
← kubeflow: [profile-controller] profiles CRD does not preserve unknown fields in plugins.spec
kubeflow: Several pods not starting due to various errors related to using NFS as dynamic provisioner →