microk8s: microk8s is not running. microk8s.inspect showing no error

Solved it for me @gavinB-orange - had to remove my previously installed kubectl and then it microk8s started working!

rm -rf /usr/local/bin/kubectl

Try refreshing the certificate

sudo microk8s refresh-certs

then check for microk8s status this must work

I have solved the issue. Do not run in root mode. Try running in user mode.

k8s-master@k8s-master:-$ sudo microk8s.status

Either removing and re-installing fixed the issue or the version: installed: v1.17.0 (1109) 179MB classic

$ snap remove microk8s 
microk8s removed

$ microk8s.status
bash: /snap/bin/microk8s.status: No such file or directory


$ sudo snap install microk8s --classic --channel=1.17/stable

microk8s (1.17/stable) v1.17.0 from Canonical✓ installed


$ microk8s.start
Started.
Enabling pod scheduling
node/blushy already uncordoned


$ microk8s.status
microk8s is running
addons:
cilium: disabled
dashboard: disabled
dns: disabled
fluentd: disabled
gpu: disabled
helm: disabled
ingress: disabled
istio: disabled
jaeger: disabled
juju: disabled
knative: disabled
kubeflow: disabled
linkerd: disabled
metallb: disabled
metrics-server: disabled
prometheus: disabled
rbac: disabled
registry: disabled
storage: disabled

As Chinese user, it may caused by GFW.

As /var/snap/microk8s/3xxx/inspection-report/snap.microk8s.daemon-kubelite/journal.log shows:

Sep 23 12:41:15 an microk8s.daemon-kubelite[1773]: E0923 12:41:15.844909    1773 remote_runtime.go:222] "RunPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = failed to get sandbox image \"k8s.gcr.io/pause:3.7\": failed to pull image \"k8s.gcr.io/pause:3.7\": failed to pull and unpack image \"k8s.gcr.io/pause:3.7\": failed to resolve reference \"k8s.gcr.io/pause:3.7\": failed to do request: Head \"https://k8s.gcr.io/v2/pause/manifests/3.7\": dial tcp 142.251.8.82:443: i/o timeout"

How about changing the message:

Currently:

microk8s is not running. Use microk8s.inspect for a deeper inspection.

New:

microk8s is not running. Verify your config is valid and use microk8s.inspect for a deeper inspection.

Or:

microk8s is not running in cluster $CLUSTERNAME. Use microk8s.inspect for a deeper inspection.

Something that would make the user think about having a misconfigured client and not necessarily microk8s dying.

@TribalNightOwl microk8s is not actually dying. The status command uses the kubeconfig settings to verify the cluster’s health. So if the kubeconfig is misconfigured it will not be able to gather kubernetes resources, hence it will say not running

The message can be misleading though.

I also had the same issue. It happens with v1.17 only (other versions like v1.16, v1.15, v1.14) are ok. Also, it happens with v1.17 after I try to install kubeflow using microk8s.enable kubeflow which basically throws an error about Juju. To resolve that I did install Juju and lxd and did juju bootstrap after this microk8s.status gives me microk8s not running.

I reproduced this behaviour in another machine as well.

How about changing the message

We could also detect such problems and suggest a fix in microk8s.inspect https://github.com/ubuntu/microk8s/blob/master/scripts/inspect.sh#L106

@balchua same question!

enviroment

root@ajinlong:/var/snap/microk8s/3597# uname -a
Linux ajinlong 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@ajinlong:/var/snap/microk8s/3597# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy

After snap install microk8s --classic i check microk8s status. then microk8s is not running. Use microk8s inspect for a deeper inspection

Inspecting system
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-kubelite is running
  Service snap.microk8s.daemon-k8s-dqlite is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster
Inspecting dqlite
  Inspect dqlite

Building the report tarball
  Report tarball is at /var/snap/microk8s/3597/inspection-report-20220912_154441.tar.gz

But i found processes referring microk8s

root@ajinlong:/home/ajinlong# ps -ef | grep microk8s
root       13376       1 17 15:40 ?        00:01:16 /snap/microk8s/3597/kubelite --scheduler-args-file=/var/snap/microk8s/3597/args/kube-scheduler --controller-manager-args-file=/var/snap/microk8s/3597/args/kube-controller-manager --proxy-args-file=/var/snap/microk8s/3597/args/kube-proxy --kubelet-args-file=/var/snap/microk8s/3597/args/kubelet --apiserver-args-file=/var/snap/microk8s/3597/args/kube-apiserver --kubeconfig-file=/var/snap/microk8s/3597/credentials/client.config --start-control-plane=true
root       13397       1  0 15:40 ?        00:00:00 /bin/bash /snap/microk8s/3597/apiservice-kicker
root       13438       1  5 15:40 ?        00:00:23 /snap/microk8s/3597/bin/k8s-dqlite --storage-dir=/var/snap/microk8s/3597/var/kubernetes/backend/ --listen=unix:///var/snap/microk8s/3597/var/kubernetes/backend/kine.sock:12379
root       13544       1  0 15:40 ?        00:00:00 /bin/bash /snap/microk8s/3597/run-cluster-agent-with-args
root       13556       1  0 15:40 ?        00:00:04 /snap/microk8s/3597/bin/containerd --config /var/snap/microk8s/3597/args/containerd.toml --root /var/snap/microk8s/common/var/lib/containerd --state /var/snap/microk8s/common/run/containerd --address /var/snap/microk8s/common/run/containerd.sock
root       13745   13544  0 15:40 ?        00:00:01 /snap/microk8s/3597/bin/cluster-agent --bind 0.0.0.0:25000 --keyfile /var/snap/microk8s/3597/certs/server.key --certfile /var/snap/microk8s/3597/certs/server.crt --timeout 240
root       17820   15171  0 15:48 pts/2    00:00:00 grep --color=auto microk8s

inspection-report-20220912_154441.tar.gz

I run into the same issue today. microk8s start => started microk8s status => microk8s is not running. Use microk8s inspect for a deeper inspection. microk8s inspect => shows no errors I have to say that i also had the same problem yesterday, and reinstalled both microk8s and kubectl which worked for a while but not this morning.

apparently for me it came down to the x509certificate, eg: microk8s kubectl get ns Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2022-01-13T13:02:19Z is before 2022-01-13T13:32:23Z

If I move the clock on the computer 1 hour forward everything works fine then.

I run it on an Ubuntu 20.04 laptop, set to the Automatic Date/Time and GMT timezone. Not sure what caused this, but hope this is useful.

UPD: it seems to be caused by the laptop’s RTC being misconfigured for some reason. The following command fixed it: timedatectl set-local-rtc 0

Yes definitely will run into port conflicts whether u have an already k8s running or an application using the ports. The list of ports and services are here. https://microk8s.io/docs/services-and-ports

On my system I found that the problem went away after I updated the rather too old kubectl installed in /usr/local/bin on my system. I had assumed that microk8s would exclusively use it’s own kubectl, but apparently not.

@balchua not seem working

wtf@k8s-master:~$ microk8s.stop
[sudo] password for wtf:
Stopped.
wtf@k8s-master:~$ microk8s.start
Started.
Enabling pod scheduling
wtf@k8s-master:~$ microk8s.status
microk8s is not running. Use microk8s.inspect for a deeper inspection.
wtf@k8s-master:~$