microk8s: Failed to enable kubeflow

have the same problem

I have the same problem

@xelios: Ah, it looks like there was an incomplete teardown of a previous Kubeflow deployment, as the admission-webhook shouldn’t be deployed before the network check. The easiest way to fix this (assuming you don’t have anything else in MicroK8s you need to keep) is to just reinstall the MicroK8s snap:

sudo snap remove microk8s --purge
sudo snap install microk8s --classic

Otherwise, you can list the existing mutating and validating webhooks with:

microk8s kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations

And then running microk8s delete on any webhooks with admission-webhook in the name.

sudo iptables -P FORWARD ACCEPT - helped me with same issue, maybe it can help someone

I got the same error message: Couldn’t contact api.jujucharms.com from within the Kubernetes cluster.

os: Ubuntu 18.04.5 LTS microk8s: v1.19.4 rev1827

$ snap list

$ microk8s kubectl get all --all-namespaces

$ microk8s enable kubeflow

+ microk8s-status.wrapper --wait-ready
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dashboard            # The Kubernetes dashboard
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    ingress              # Ingress controller for external access
    metallb              # Loadbalancer for your Kubernetes cluster
    metrics-server       # K8s Metrics Server for API access to service metrics
    storage              # Storage class; allocates storage from host directory
  disabled:
    ambassador           # Ambassador API Gateway and Ingress
    cilium               # SDN, fast with full network policy
    fluentd              # Elasticsearch-Fluentd-Kibana logging and monitoring
    gpu                  # Automatic enablement of Nvidia CUDA
    helm                 # Helm 2 - the package manager for Kubernetes
    helm3                # Helm 3 - Kubernetes package manager
    host-access          # Allow Pods connecting to Host services smoothly
    istio                # Core Istio service mesh services
    jaeger               # Kubernetes Jaeger operator with its simple config
    knative              # The Knative framework on Kubernetes.
    kubeflow             # Kubeflow for easy ML deployments
    linkerd              # Linkerd is a service mesh for Kubernetes and other frameworks
    multus               # Multus CNI enables attaching multiple network interfaces to pods
    portainer            # Portainer UI for your Kubernetes cluster
    prometheus           # Prometheus operator for monitoring and logging
    rbac                 # Role-Based Access Control for authorisation
    registry             # Private image registry exposed on localhost:32000
    traefik              # traefik Ingress controller for external access

+ microk8s-kubectl.wrapper -nkube-system rollout status deployment.apps/calico-kube-controllers
deployment "calico-kube-controllers" successfully rolled out

Enabling dns...
+ microk8s-enable.wrapper dns
Addon dns is already enabled.

Enabling storage...
+ microk8s-enable.wrapper storage
Addon storage is already enabled.

Enabling dashboard...
+ microk8s-enable.wrapper dashboard
Addon dashboard is already enabled.

Enabling ingress...
+ microk8s-enable.wrapper ingress
Addon ingress is already enabled.

Enabling metallb:10.64.140.43-10.64.140.49...
+ microk8s-enable.wrapper metallb:10.64.140.43-10.64.140.49
Addon metallb is already enabled.

+ microk8s-status.wrapper --wait-ready
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dashboard            # The Kubernetes dashboard
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    ingress              # Ingress controller for external access
    metallb              # Loadbalancer for your Kubernetes cluster
    metrics-server       # K8s Metrics Server for API access to service metrics
    storage              # Storage class; allocates storage from host directory
  disabled:
    ambassador           # Ambassador API Gateway and Ingress
    cilium               # SDN, fast with full network policy
    fluentd              # Elasticsearch-Fluentd-Kibana logging and monitoring
    gpu                  # Automatic enablement of Nvidia CUDA
    helm                 # Helm 2 - the package manager for Kubernetes
    helm3                # Helm 3 - Kubernetes package manager
    host-access          # Allow Pods connecting to Host services smoothly
    istio                # Core Istio service mesh services
    jaeger               # Kubernetes Jaeger operator with its simple config
    knative              # The Knative framework on Kubernetes.
    kubeflow             # Kubeflow for easy ML deployments
    linkerd              # Linkerd is a service mesh for Kubernetes and other frameworks
    multus               # Multus CNI enables attaching multiple network interfaces to pods
    portainer            # Portainer UI for your Kubernetes cluster
    prometheus           # Prometheus operator for monitoring and logging
    rbac                 # Role-Based Access Control for authorisation
    registry             # Private image registry exposed on localhost:32000
    traefik              # traefik Ingress controller for external access

+ microk8s-kubectl.wrapper -nkube-system rollout status ds/calico-node
daemon set "calico-node" successfully rolled out

Waiting for DNS and storage plugins to finish setting up
+ microk8s-kubectl.wrapper wait --for=condition=available -nkube-system deployment/coredns deployment/hostpath-provisioner --timeout=10m
deployment.apps/coredns condition met
deployment.apps/hostpath-provisioner condition met

Couldn't contact api.jujucharms.com from within the Kubernetes cluster
Please check your network connectivity before enabling Kubeflow.
Failed to enable kubeflow

$ microk8s inspect

inspection-report-20201124_171509.tar.gz

I also got the same error message: Couldn’t contact api.jujucharms.com from within the Kubernetes cluster.

Solution:

1. do again below command

microk8s enable kubeflow

1. if the above method doesn’t work, reinstall the microk8s and kubeflow

sudo snap remove microk8s --purge

# install the latest stable version of kubernetes 1.20.1
sudo snap install microk8s --classic --channel=latest/edge && sudo snap refresh

microk8s enable dns dashboard storage gpu

microk8s enable kubeflow

os: Ubuntu 20.04.1 LTS microk8s: v1.20.1 rev1894

After some tries with different versions, I could make it work (on v1.19 stable) with:

sudo iptables -P FORWARD ACCEPT

microk8s inspect told me that, but the “permanent” solution did not work alone (sudo apt install iptables-persistent). So the iptables rule was needed. I am on Debian 10.

@hpcaicom: It looks like there’s a bug in that version of microk8s, can you run this and then try to enable Kubeflow again?

sudo snap switch microk8s --channel=latest/candidate
sudo snap refresh

same issue on v1.22.3 sudo iptables -P FORWARD ACCEPT didn’t work for me

I am too facing same error with 1.18,1.19,1.20 versions of microk8s.

Error on enabling kubeflow addon:

$ microk8s.enable kubeflow
Enabling dns...
Enabling storage...
Enabling ingress...
Enabling metallb:10.64.140.43-10.64.140.49...
Waiting for other addons to finish initializing...
Addon setup complete. Checking connectivity...
Couldn't contact api.jujucharms.com
Please check your network connectivity before enabling Kubeflow.
Failed to enable kubeflow

microk8s inspect:

$  microk8s inspect
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-flanneld is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Service snap.microk8s.daemon-etcd is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster

Building the report tarball
  Report tarball is at /var/snap/microk8s/2647/inspection-report-20220217_212519.tar.gz

ping api.jujucharms.com is failing with packets not being received.

$ ping api.jujucharms.com

--- api.jujucharms.com ping statistics ---
17 packets transmitted, 0 received, 100% packet loss, time 16389ms

curl -v https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-88/icon.svg seems to be not found error as the url is no longer valid it seems.

$ curl -v https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-88/icon.svg
*   Trying 162.213.33.79...
* TCP_NODELAY set
* Connected to api.jujucharms.com (162.213.33.79) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=api.jujucharms.com
*  start date: Jan 13 06:20:36 2022 GMT
*  expire date: Apr 13 06:20:35 2022 GMT
*  subjectAltName: host "api.jujucharms.com" matched cert's "api.jujucharms.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /charmstore/v5/~kubeflow-charmers/ambassador-88/icon.svg HTTP/1.1
> Host: api.jujucharms.com
> User-Agent: curl/7.58.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/1.1 404 NOT FOUND
< Date: Thu, 17 Feb 2022 21:28:25 GMT
< Server: gunicorn
< content-type: text/html
< content-length: 232
< snap-store-version: 46
< x-vcs-revision: ceb1c973
< x-request-id: A2D5214FBEEC0A325D7701BB620EBDF919FC12F2
< 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>404 Not Found</title>
<h1>Not Found</h1>
<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>
* Connection #0 to host api.jujucharms.com left intact

Any idea on how to overcome this?

per discussion on slack with @RFMVasconcelos (thanks again!), i ran the following network test:

microk8s.kubectl run --rm -it --restart=Never --image=ubuntu connectivity-check -- bash -c "apt update && apt install -y curl && curl https://api.jujucharms.com/charmstore/v5/istio-pilot-5/icon.svg"

with the following error:

dial tcp: lookup athena: Temporary failure in name resolution

In my case, the issue turned out to be that the local dns hostname didn’t match the machine name (eg my machine name is athena, but i had the name athena-40g being assigned by dns), and then all sort of wierdness happened elsewhere/kubeflow was dying. Simply making the interface name match the host name resolved the issue and now all is working properly, thanks Rui!

I’ve tried in 1.20/stable, but there’s still same error.

DNS and storage setup complete. Checking connectivity… Couldn’t contact api.jujucharms.com from within the Kubernetes cluster Please check your network connectivity before enabling Kubeflow.

inspection-report-20201216_094714.tar.gz kubeflow_installation_log.txt

microk8s.kubectl run --rm -it --restart=Never --image=ubuntu connectivity-check – bash -c “apt update && apt install -y curl && curl https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-88/icon.svg”

@knkski I think the issue is the proxy is not being set inside the pod container. I created an issue for this #1719

Removing microk8s and installing the old 1.18/stable version seems to fix the error.