microk8s: Cannot connect to microk8s from another machine

I just ran a fresh install of microk8s with this command:

sudo snap install microk8s --classic --channel=1.14/stable

I can access everything while I’m SSH’ed into the box this is running on, with commands much like this:

microk8s.kubectl get pods

However when I set my ~/.kube/config file - on another machine on the same network - to this:

apiVersion: v1
clusters:
- cluster:
    server: http://192.168.1.123:8080
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    username: admin

And then try to run this command:

kubectl get pods

I get this error message:

The connection to the server 192.168.1.123:8080 was refused - did you specify the right host or port?

In fact, when I run curl localhost:8080 while on the box, I see valid K8S output, but if I try running curl 192.168.1.123:8080 from outside the box, even that returns:

curl: (7) Failed to connect to 192.168.1.123 port 8080: Connection refused

I’ve tried a couple of things to get this working, first of which was allowing access to 8080 on the firewall with this command:

sudo ufw allow 8080
sudo ufw enable

Unfortunately that doesn’t solve the problem. When I run netstat -an | grep "LISTEN " I can see an entry for port 8080 as follows:

tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN

But because it says 127.0.0.1 instead of 0.0.0.0, this would explain why it’s not working off the box. Is there some microk8s.* command I don’t know about, to open up access?

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 7
Comments: 31 (9 by maintainers)

Commits related to this issue

Feature: modifiable csr (#441) * allow modifiable csr conf template - https://github.com/ubuntu/microk8s/issues/421 — committed to canonical/microk8s by balchua 5 years ago

Most upvoted comments

In v1.14 we restricted the insecure access to MicroK8s from port 8080 to local users only.

You have two options:

Open insecure port 8080 to everyone. To do this you need to edit /var/snap/microk8s/current/args/kube-apiserver and set the --insecure-bind-address=127.0.0.1 to 0.0.0.0. Then restart MicroK8s with microk8s.stop and microk8s.start.
Use the secure port 16443. Due to a recent bug fix you need to get MicroK8s from edge sudo snap install microk8s --channel=1.14/edge --classic. This bug fix will soon reach stable. Then use the config you get with microk8s.config to reach the cluster.

+24

ktsakalozos on Apr 9, 2019

@shoover Thank you so much for that suggestion, that was indeed the issue! The /var/snap/microk8s/current/certs/csr.conf file had up to IP.4. So I added my public IP as IP.5 and was able to connect without issue. Thanks again!

kawsark on Apr 5, 2020

Hello @soapergem Thank you for your help. Unfortunately I am still unable to get this working. For now I am using a workaround of SSH Tunneling. I provided the steps to reproduce the problem below. Any help is greatly appreciated!

Installed mircrok8s on a ubuntu 18.04.4 LTS server using sudo snap install microk8s --classic
Wait until all services are up, then edit /var/snap/microk8s/current/certs/csr.conf.template to add IP.3 as below

[ alt_names ]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster
DNS.5 = kubernetes.default.svc.cluster.local
IP.1 = 127.0.0.1
IP.2 = 10.152.183.1
IP.3 = 34.xx.xx.xx
#MOREIPS

Run microk8s.stop and then microk8s.start
Save the output from sudo microk8s.kubectl config view --raw locally as and modify server: https://127.0.0.1:16443 to server: https://34.xx.xx.xx:16443
Run export KUBECONFIG=./microk8s.yaml and kubectl get nodes. I get the error:

➜  kubectl get nodes
Unable to connect to the server: x509: certificate is valid for 127.0.0.1, 10.152.183.1, 10.142.0.19, 10.1.14.0, not 34.xx.xx.xx

kawsark on Mar 17, 2020

I think I may have fat-fingered something when I copied my ~/.kube/config file. This time I copied it verbatim from the output of microk8s.config – and then updated it to use the external IP address, as before – and now I’m getting a different error:

Unable to connect to the server: x509: certificate is valid for 127.0.0.1, 10.152.183.1, 192.168.1.123, not <my-external-ip>

And here I’m using <my-external-ip> as a placeholder; it does in fact show my external IP on that line. Is there any way for me to re-generate the cert such that it allows me to enter the server’s external IP?

P.S. I’m not sure that adding -v=9 to that file did anything. Would that add more information to a log file, or something like that?

soapergem on Apr 10, 2019

@kawsark Thanks a lot, you saved my day.

For anybody interested in this, I’ve used the above hint to create an Ansible playbook (https://github.com/pfisterer/edsc-microk8s-playbook) that installs microk8s in OpenStack and modifies the CSR accordingly.

pfisterer on Jun 19, 2020

Hi @mjordan79, can you check whether running the command below solves your issue:

sudo snap set microk8s test="$(date)"

This is going to ensure that certificates are refreshed and get the latest version of the csr.conf.template. We are aware of a bug with the refresh-certs command, but this should be a valid workaround for now.

neoaggelos on Feb 8, 2023

The feature seems to be broken. I have modified the csr.conf.template adding an IP.4, IP.5 and IP.6. After restarting microk8s, the csr.conf file still has just 3 IPs, none of them corresponds to mine. Using 1.26 stable.

What am I doing wrong?

mjordan79 on Jan 29, 2023

@kawsark check the generated crt conf. When I did this, the MOREIPS tag expanded to overwrite IP.3 with another IP. After changing the template to IP.9 (out of range of those added by MOREIPS), it worked.

shoover on Apr 5, 2020

Hello I have installed microk8s in wsl and in my .wslconfig I am using networkingMode=mirrored and I am also unable to connect the kubectl by my windows private ip address

sadiqueWiseboxs on Dec 14, 2023

Hi @mjordan79.

Ah, yes, my bad, _test is not a valid option name. The option name itself is irrelevant, so try:
sudo snap set microk8s test="$(date)"
Apologies! Editing my previous comment as well to prevent confusion.

@neoaggelos Sorry for the late reply. It worked.

mjordan79 on Feb 13, 2023

@kawsark after editing the file you need to run first microk8s.stop and then microk8s.start. FYI I posted about this, along with another way to regenerate certs, on this post.

soapergem on Mar 11, 2020

As a data point for someone who is not new to k8s but is new to microk8s, this was my process:

look at mk8s docs site for how to allow external access to API provider
look at README in repo
repo Issues search
back to mk8s docs site
back to this Issue to ask

nrvale0 on Jan 7, 2020

At the moment, @soapergem, the best way to check for new releases is with:

snap info microk8s

The edge channel gets immediately updated with the code we have on this repo. Upon an upstream patch release (eg from 1.12.7 to 1.12.8) the edge channel is pushed to beta and candidate. About a week after a candidate release the new version is pushed to stable.

In the future we would like to automate the release notes authoring and make use of the github release page.

ktsakalozos on May 2, 2019