pipelines: Strange "Error 401 (Unauthorized)!!1" when submitting runs, but not with other APIs.

Submitting pipelines using kfp.Client(host=kfp_endpoint).create_run_from_pipeline_func(pipeline, arguments={}) suddenly stopped working. It worked great just this morning. I have not updated the cluster or SDK or anything else. I’ve tried re-running a recent notebook and it also started failing.

A weird thing is that listing runs does not fail: kfp.Client(host=kfp_endpoint).runs.list_runs(). Another weird thing is that the error seems to come from Google, not my cluster.

The error I’m getting:

...

~/kfp/sdk/python/kfp/_client.py in run_pipeline(self, experiment_id, job_name, pipeline_package_path, params, pipeline_id, version_id)
    403         pipeline_spec=job_config.spec, resource_references=job_config.resource_references, name=job_name)
    404 
--> 405     response = self._run_api.create_run(body=run_body)
    406 
    407     if self._is_ipython():

...

~/.local/lib/python3.5/site-packages/kfp_server_api/rest.py in request(self, method, url, query_params, headers, body, post_params, _preload_content, _request_timeout)
    226 
    227         if not 200 <= r.status <= 299:
--> 228             raise ApiException(http_resp=r)
    229 
    230         return r

ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'X-Xss-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'X-Content-Type-Options': 'nosniff', 'Date': 'Wed, 06 May 2020 01:24:09 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Content-Length': '1452', 'Set-Cookie': 'S=cloud_datalab_tunnel=wDLDslTO0CzR204H7_evCuJ-YgbpzSfFtNEI0TolqX0; Path=/; Max-Age=3600'})
HTTP response body: 
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 401 (Unauthorized)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/logos/errorpage/error_logo-150x54-2x.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>401.</b> <ins>That’s an error.</ins>
  <p>  <ins>That’s all we know.</ins>

What did you expect to happen:

Environment:

How did you deploy Kubeflow Pipelines (KFP)? Marketplace KFP version: 0.2.4

KFP SDK version: 0.5.0

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

/kind bug

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 20 (13 by maintainers)

Most upvoted comments

Rollback done.

It should can work in all region now.

Verified myside in us-central1-a

rmgogogo on May 6, 2020