pipelines: KFP sdk client authentication error
/kind bug
What steps did you take and what happened:
Enabled authentication with Azure AD on AKS and installing Kubeflow with kfctl_istio_dex.v1.1.0.yaml but skipping the dex from the manifest as Azure AD is an OIDC provider. The load balancer is exposed over https with TLS 1.3 self-signed cert.
OIDC Auth Service Configuration:
- client_id=XXXX
- oidc_provider=https://login.microsoftonline.com/XXXX/v2.0
- oidc_redirect_uri=https://XXXX/login/oidc
- oidc_auth_url=https://login.microsoftonline.com/XXXX/oauth2/v2.0/authorize
- application_secret=XXXX
- skip_auth_uri=
- namespace=istio-system
- userid-header=kubeflow-userid
- userid-prefix=
Issue When using KFP client to upload the pipeline (client.pipeline_uploads.upload_pipeline()) with below client config throws an error.
client = kfp.Client(host='https://<LoadBalancer IP Address>/pipeline', existing_token=<token>)
Error HTTPSConnectionPool(host=‘<Host_IP>’, port=443): Max retries exceeded with url: /pipeline/apis/v1beta1/pipelines/upload?name=local_exp-6714175b-6d59-40d0-9019-5b4ee58dc483 (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1076)’)))
Is there a way to override cert verification?
or
When using KFP client to upload the pipeline (client.pipeline_uploads.upload_pipeline()) with below client config redirects to google auth error.
client = kfp.Client(host='https://<LoadBalancer IP Address>/pipeline ,client_id=<client_id>, other_client_id=<client_id>,other_client_secret=<application_secret>,namespace='kfauth')
Environment:
- Kubeflow version:
v1.1.O - kfctl version:
kfctl_v1.1.0-0-g9a3621e_linux.tar.gz - kfp version:
1.0.1 - python version:
3.6.8 - kfp-server-api version:
1.0.1 - Kubernetes platform:
Azure Kubernetes Service - Kubernetes version:
1.17.11
CC: @Bobgy
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 43 (12 by maintainers)
Commits related to this issue
- fix(sdk): add verify_ssl to kfp client, fixes #4569 (#7174) * adding verify_ssl * adding code formatter changes * retrigger checks * updating release.md * retrigger checks * retrigger ... — committed to kubeflow/pipelines by vmallya-123 2 years ago
- fix(sdk): add verify_ssl to kfp client, fixes #4569 (#7174) * adding verify_ssl * adding code formatter changes * retrigger checks * updating release.md * retrigger checks * retrigger checks * ... — committed to abaland/pipelines by vmallya-123 2 years ago
I agree with the others here. This isn’t an AKS or cloud issue. The Kubeflow docs instruct you to use certmanager to create a self signed certificate. But obviously browsers and curl can’t verify the identity, so you just ignore/suppress that.
The issue is that the
kfp.Clientclass doesn’t allow you to passverify=falsethrough to the underlying requests library, so you can’t ignore the non-verifiable certificate. And therefore you can’t usekfp.Clienton clusters that have been setup following the standard KF docs.When I try to consume REST API directly using a Bearer access token generated with grant-type client credentials it still redirects to the authorization endpoint forcing iterative login.
url=r'https://host/pipeline/apis/v1beta1/pipelines' header = {'Authorization': 'Bearer ' + token} response = requests.get(url,headers=header,verify=False)The above code redirects to the authorization endpoint
https://login.microsoftonline.com/<tenant_id>/v2.0/authorize?client_id=<client_id> &redirect_uri=https%3A%2F%2F<host>%2Flogin%2Foidc &response_type=code &scope=profile+email+openid &state=<xxxxx>@yanniszark what type of token can I use for REST API to skip interactive login? If that works we can have optional SSL verification on the client.