kubeflow: kubeflow user pipelines interface not working and shows error

/kind bug

What steps did you take and what happened: Deploy kubeflow version 1.2 on kubernetes using the kfctl operator

KFDEF:

apiVersion: kfdef.apps.kubeflow.org/v1 kind: KfDef metadata: name: kubeflow namespace: kubeflow spec: applications: - kustomizeConfig: repoRef: name: manifests path: namespaces/base name: namespaces - kustomizeConfig: repoRef: name: manifests path: application/v3 name: application - kustomizeConfig: repoRef: name: manifests path: stacks/kubernetes/application/cert-manager-crds name: cert-manager-crds - kustomizeConfig: repoRef: name: manifests path: stacks/kubernetes/application/cert-manager-kube-system-resources name: cert-manager-kube-system-resources - kustomizeConfig: repoRef: name: manifests path: stacks/kubernetes/application/cert-manager name: cert-manager - kustomizeConfig: repoRef: name: manifests path: metacontroller/base name: metacontroller - kustomizeConfig: repoRef: name: manifests path: admission-webhook/bootstrap/overlays/application name: bootstrap - kustomizeConfig: repoRef: name: manifests path: stacks/kubernetes name: kubeflow-apps - kustomizeConfig: repoRef: name: manifests path: kfserving/installs/generic name: kfserving repos: - name: manifests uri: https://github.com/kubeflow/manifests/archive/v1.2.0.tar.gz

What did you expect to happen: I did not expect the error to be there

Anything else you would like to add: Is it related to the separation of pipelines UIs for users?

Environment:

Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard): 1.2 (build version v1beta1)
kfctl version: (use kfctl version): 1.2
Kubernetes platform: (e.g. minikube) from a cloud provider
Kubernetes version: (use kubectl version): 1.17.1
OS (e.g. from /etc/os-release): Linux

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (5 by maintainers)

Most upvoted comments

@solarist I had accidentally deleted this resource:

apiVersion: authentication.istio.io/v1alpha1
kind: MeshPolicy
metadata:
  name: default
  namespace: kubeflow
spec:
  peers:
  - mtls:
      mode: PERMISSIVE

recreating fixed my issue. I guess make sure it’s there just in case something weird happened.

jonasdebeukelaer on Feb 5, 2021

I am using Istio 1.6 and the connection between metadata-envoy-deployment and metadata-grpc-deployment was the problem, was throwing upstream connect error or disconnect/reset before headers. reset reason: connection termination error, I did this to fix the connection issue.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: kubeflow
spec:
  mtls:
    mode: PERMISSIVE

deepk2u on Feb 28, 2021

@JohanWork not sure what makefile you’re talking about but you can just run the follow to make sure it’s there

cat <<EOF | kubectl apply -f -
apiVersion: authentication.istio.io/v1alpha1
kind: MeshPolicy
metadata:
  name: default
  namespace: kubeflow
spec:
  peers:
  - mtls:
      mode: PERMISSIVE
EOF

jonasdebeukelaer on Feb 17, 2021

@jonasdebeukelaer wow that fixed it, thank you! I am relatively sure that it hasn’t been deleted in our deployment, and I can’t seem to find such a mesh policy created at all per the k8s-dex manifests. Also why a service would want plain text in the first place is actually beyond me…

solarist on Feb 5, 2021