pipelines: client.list_experiments() not working as well as creating runs, authentication failing

What steps did you take

We deployed kubeflow on premise using kustomize file https://github.com/kubeflow/manifests/tree/v1.3.1, v 1.3.1 Created a notebook, tried to create a run from it using kfp

I can’t connect from the notebook to pipelines (to create runs and see experiments ) import kfp kfp_endpoint=None client.list_experiments()

but i can enumerate pipelines
import kfp kfp_endpoint=None client = kfp.Client(host=kfp_endpoint) print(client.list_pipelines())

What happened:

i’am getting Reason: Internal Server Error HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'trailer': 'Grpc-Trailer-Content-Type', 'date': 'Wed, 13 Oct 2021 11:48:57 GMT', 'x-envoy-upstream-service-time': '2', 'server': 'envoy', 'transfer-encoding': 'chunked'}) HTTP response body: {"error":"Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.\nFailed to authorize with API resource references\ngithub.com/kubeflow/pipelines/backend/src/common/util.Wrap\n\t/go/src/github.com/kubeflow/pipelines .....

I tried multiple things, some poeple faced the same issue, adding the namespace, creting a manifest as described in https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api/#connect-to-kubeflow-pipelines-from-the-same-cluster Nothing worked for me . I would really appreciate any help

What did you expect to happen:

See the expierments, create runs from my pipelines

Environment:

How do you deploy Kubeflow Pipelines (KFP)?

Deployed using Kustomize https://github.com/kubeflow/manifests/tree/v1.3.1

KFP version:

KFP SDK version:

Anything else you would like to add:

if i pass the YRL of the pipeline UI in kfp client, there is no error but nothing is returned {'experiments': None, 'next_page_token': None, 'total_size': None}

Labels

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 5
Comments: 22 (1 by maintainers)

Most upvoted comments

@rexad In my case, the issue was an outdated version of kfp. After running pip install --upgrade kfp and upgrading to kfp==1.8.6 my issue was resolved via the method @zijianjoy mentioned in the e2e notebook example.

nschenone on Oct 23, 2021

I would also like to highlight this point: After trying many things for hours I also just did a (I imagined somehow that I already did this; apparently I did not): !pip install --upgrade kfp before running (dont forget to restart your kernel…)

import kfp
client = kfp.Client()
print(client.list_experiments())

So @zijianjoy 's suggestion actually works with the newest kfp version…

Would you like to try the multi-user mode approach here? https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api/#multi-user-mode

MarkusThill on Nov 12, 2021

Glad that upgrading kfp SDK works out for you both!

On-prem server deployment can be used by advanced users. The other approach is to deploy via packaged distribution: https://www.kubeflow.org/docs/started/installing-kubeflow/#install-a-packaged-kubeflow-distribution.

Feel free to close this issue as the original problem has been resolved.

zijianjoy on Oct 25, 2021

Glad to hear it. I’m using a local on-prem server and installing via the manifests. I’ve actually deployed my k8s cluster using k3d. Not exactly for prod but great for dev and testing

nschenone on Oct 25, 2021

@rexad Lots of trial and error…and error…and error 😃 It is not explicitly stated anywhere, but I noticed in the notebook example that @zijianjoy linked that they are installing a newer version of kfp than I had.

When I installed kfp==1.8.6 and instantiated a client, there was a message printed about the token located at /var/run/secrets/kubeflow/pipelines/token which is where this guide comes into play.

nschenone on Oct 24, 2021