pipelines: Cannot submit pipelines job through SDK in jupyterbook server: microk8s, default namespace

What steps did you take:

[A clear and concise description of what the bug is.] I’m trying to submit a kfp run from the SDK inside a jupyter notebook server on the default admin namespace.

What happened:

I get the following error when running the function:

kfp.Client().create_run_from_pipeline_func(plane_pipeline, arguments=arguments)

error

2020-06-23 15:00:05,231 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b5c0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,231 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b5c0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,231 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b5c0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,231 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b5c0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b5c0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,238 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308bcc0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,238 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308bcc0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,238 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308bcc0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,238 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308bcc0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308bcc0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,244 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b710>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,244 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b710>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,244 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b710>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,244 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b710>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b710>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments?page_token=&page_size=100&sort_by=&resource_reference_key.type=NAMESPACE&resource_reference_key.id=
2020-06-23 15:00:05,252 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308be48>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
2020-06-23 15:00:05,252 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308be48>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
2020-06-23 15:00:05,252 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308be48>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
2020-06-23 15:00:05,252 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308be48>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308be48>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
2020-06-23 15:00:05,257 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b4a8>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments

2020-06-23 15:00:05,257 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b4a8>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
2020-06-23 15:00:05,257 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b4a8>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
2020-06-23 15:00:05,257 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b4a8>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b4a8>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
2020-06-23 15:00:05,263 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b518>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
2020-06-23 15:00:05,263 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b518>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
2020-06-23 15:00:05,263 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b518>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
2020-06-23 15:00:05,263 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b518>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b518>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/v1beta1/experiments

---------------------------------------------------------------------------
gaierror                                  Traceback (most recent call last)
~/.local/lib/python3.6/site-packages/urllib3/connection.py in _new_conn(self)
    159             conn = connection.create_connection(
--> 160                 (self._dns_host, self.port), self.timeout, **extra_kw
    161             )

~/.local/lib/python3.6/site-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     60 
---> 61     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
     62         af, socktype, proto, canonname, sa = res

/usr/lib/python3.6/socket.py in getaddrinfo(host, port, family, type, proto, flags)
    744     addrlist = []
--> 745     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    746         af, socktype, proto, canonname, sa = res

gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

NewConnectionError                        Traceback (most recent call last)
~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    676                 headers=headers,
--> 677                 chunked=chunked,
    678             )

~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    391         else:
--> 392             conn.request(method, url, **httplib_request_kw)
    393 

/usr/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1263         """Send a complete request to the server."""
-> 1264         self._send_request(method, url, body, headers, encode_chunked)
   1265 

/usr/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1309             body = _encode(body, 'body')
-> 1310         self.endheaders(body, encode_chunked=encode_chunked)
   1311 

/usr/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1258             raise CannotSendHeader()
-> 1259         self._send_output(message_body, encode_chunked=encode_chunked)
   1260 

/usr/lib/python3.6/http/client.py in _send_output(self, message_body, encode_chunked)
   1037         del self._buffer[:]
-> 1038         self.send(msg)
   1039 

/usr/lib/python3.6/http/client.py in send(self, data)
    975             if self.auto_open:
--> 976                 self.connect()
    977             else:

~/.local/lib/python3.6/site-packages/urllib3/connection.py in connect(self)
    186     def connect(self):
--> 187         conn = self._new_conn()
    188         self._prepare_conn(conn)

~/.local/lib/python3.6/site-packages/urllib3/connection.py in _new_conn(self)
    171             raise NewConnectionError(
--> 172                 self, "Failed to establish a new connection: %s" % e
    173             )

NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fc4d308b080>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

MaxRetryError                             Traceback (most recent call last)
<ipython-input-15-a8070a4999bc> in <module>
----> 1 kfp.Client().create_run_from_pipeline_func(plane_pipeline, arguments=arguments)

~/.local/lib/python3.6/site-packages/kfp/_client.py in create_run_from_pipeline_func(self, pipeline_func, arguments, run_name, experiment_name, pipeline_conf, namespace)
    536       (_, pipeline_package_path) = tempfile.mkstemp(suffix='.zip')
    537       compiler.Compiler().compile(pipeline_func, pipeline_package_path, pipeline_conf=pipeline_conf)
--> 538       return self.create_run_from_pipeline_package(pipeline_package_path, arguments, run_name, experiment_name, namespace)
    539     finally:
    540       os.remove(pipeline_package_path)

~/.local/lib/python3.6/site-packages/kfp/_client.py in create_run_from_pipeline_package(self, pipeline_file, arguments, run_name, experiment_name, namespace)
    576     experiment_name = overridden_experiment_name or 'Default'
    577     run_name = run_name or pipeline_name + ' ' + datetime.now().strftime('%Y-%m-%d %H-%M-%S')
--> 578     experiment = self.create_experiment(name=experiment_name, namespace=namespace)
    579     run_info = self.run_pipeline(experiment.id, run_name, pipeline_file, arguments)
    580     return RunPipelineResult(self, run_info)

~/.local/lib/python3.6/site-packages/kfp/_client.py in create_experiment(self, name, description, namespace)
    277         description=description,
    278         resource_references=resource_references)
--> 279       experiment = self._experiment_api.create_experiment(body=experiment)
    280 
    281     if self._is_ipython():

~/.local/lib/python3.6/site-packages/kfp_server_api/api/experiment_service_api.py in create_experiment(self, body, **kwargs)
    149             return self.create_experiment_with_http_info(body, **kwargs)  # noqa: E501
    150         else:
--> 151             (data) = self.create_experiment_with_http_info(body, **kwargs)  # noqa: E501
    152             return data
    153 

~/.local/lib/python3.6/site-packages/kfp_server_api/api/experiment_service_api.py in create_experiment_with_http_info(self, body, **kwargs)
    226             _preload_content=params.get('_preload_content', True),
    227             _request_timeout=params.get('_request_timeout'),
--> 228             collection_formats=collection_formats)
    229 
    230     def delete_experiment(self, id, **kwargs):  # noqa: E501

~/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py in call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, async_req, _return_http_data_only, collection_formats, _preload_content, _request_timeout)
    328                                    response_type, auth_settings,
    329                                    _return_http_data_only, collection_formats,
--> 330                                    _preload_content, _request_timeout)
    331         else:
    332             thread = self.pool.apply_async(self.__call_api, (resource_path,

~/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py in __call_api(self, resource_path, method, path_params, query_params, header_params, body, post_params, files, response_type, auth_settings, _return_http_data_only, collection_formats, _preload_content, _request_timeout)
    159             post_params=post_params, body=body,
    160             _preload_content=_preload_content,
--> 161             _request_timeout=_request_timeout)
    162 
    163         self.last_response = response_data

~/.local/lib/python3.6/site-packages/kfp_server_api/api_client.py in request(self, method, url, query_params, headers, post_params, body, _preload_content, _request_timeout)
    371                                          _preload_content=_preload_content,
    372                                          _request_timeout=_request_timeout,
--> 373                                          body=body)
    374         elif method == "PUT":
    375             return self.rest_client.PUT(url,

~/.local/lib/python3.6/site-packages/kfp_server_api/rest.py in POST(self, url, headers, query_params, post_params, body, _preload_content, _request_timeout)
    273                             _preload_content=_preload_content,
    274                             _request_timeout=_request_timeout,
--> 275                             body=body)
    276 
    277     def PUT(self, url, headers=None, query_params=None, post_params=None,

~/.local/lib/python3.6/site-packages/kfp_server_api/rest.py in request(self, method, url, query_params, headers, body, post_params, _preload_content, _request_timeout)
    165                         preload_content=_preload_content,
    166                         timeout=timeout,
--> 167                         headers=headers)
    168                 elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
    169                     r = self.pool_manager.request(

~/.local/lib/python3.6/site-packages/urllib3/request.py in request(self, method, url, fields, headers, **urlopen_kw)
     78         else:
     79             return self.request_encode_body(
---> 80                 method, url, fields=fields, headers=headers, **urlopen_kw
     81             )
     82 

~/.local/lib/python3.6/site-packages/urllib3/request.py in request_encode_body(self, method, url, fields, headers, encode_multipart, multipart_boundary, **urlopen_kw)
    169         extra_kw.update(urlopen_kw)
    170 
--> 171         return self.urlopen(method, url, **extra_kw)

~/.local/lib/python3.6/site-packages/urllib3/poolmanager.py in urlopen(self, method, url, redirect, **kw)
    334             response = conn.urlopen(method, url, **kw)
    335         else:
--> 336             response = conn.urlopen(method, u.request_uri, **kw)
    337 
    338         redirect_location = redirect and response.get_redirect_location()

~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    763                 chunked=chunked,
    764                 body_pos=body_pos,
--> 765                 **response_kw
    766             )
    767 

~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    763                 chunked=chunked,
    764                 body_pos=body_pos,
--> 765                 **response_kw
    766             )
    767 

~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    763                 chunked=chunked,
    764                 body_pos=body_pos,
--> 765                 **response_kw
    766             )
    767 

~/.local/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    723 
    724             retries = retries.increment(
--> 725                 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    726             )
    727             retries.sleep()

~/.local/lib/python3.6/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
    437 
    438         if new_retry.is_exhausted():
--> 439             raise MaxRetryError(_pool, url, error or ResponseError(cause))
    440 
    441         log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)

MaxRetryError: HTTPConnectionPool(host='ml-pipeline.kubeflow.svc.cluster.local', port=8888): Max retries exceeded with url: /apis/v1beta1/experiments (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc4d308b080>: Failed to establish a new connection: [Errno -2] Name or service not known',))

What did you expect to happen:

Successful submission of pipeline run.

Environment:

How did you deploy Kubeflow Pipelines (KFP)?

microk8s

KFP version:

dev_local

KFP SDK version:

kfp                      0.5.1
kfp-server-api           0.5.0

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

/kind bug