microk8s: microk8s.reset does not remove resources

Summary

What Should Happen Instead?

All k8s resources should be remove, no exception!

System info

archlinux up-to-date
snap version
snap    2.56-1
snapd   2.56-1
series  16
arch    -
kernel  5.18.3-arch1-1

Reproduction Steps

1. sudo snap install microk8s --classic
2. microk8s enable dns

2022-06-14T14:25:12+07:00 INFO Waiting for automatic snapd restart...
microk8s (1.24/stable) v1.24.0 from Canonical✓ installed

1. microk8s.kubectl get all -A

NAMESPACE     NAME                                           READY   STATUS              RESTARTS   AGE
kube-system   pod/calico-kube-controllers-5c54889544-9h5fk   0/1     ContainerCreating   0          58s
kube-system   pod/calico-node-5wtlb                          1/1     Running             0          59s

NAMESPACE   NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.152.183.1   <none>        443/TCP   66s

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   64s

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   0/1     1            0           64s

NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-5c54889544   1         1         0       58s

1. sudo microk8s.reset

Disabling all addons
Disabling addon : core/dashboard
Disabling addon : core/dns
Disabling addon : core/gpu
Disabling addon : core/helm
Disabling addon : core/helm3
Disabling addon : core/host-access
Disabling addon : core/hostpath-storage
Disabling addon : core/ingress
Disabling addon : core/mayastor
Disabling addon : core/metallb
Disabling addon : core/metrics-server
Disabling addon : core/prometheus
Disabling addon : core/rbac
Disabling addon : core/registry
Disabling addon : core/storage
All addons are disabled.
Deleting the CNI
Cleaning resources in namespace kube-system
Cleaning resources in namespace kube-public
Cleaning resources in namespace kube-node-lease
Cleaning resources in namespace default
Removing CRDs
Removing PriorityClasses
Removing StorageClasses
Restarting cluster
Stopped.
Started.
sudo: symbol lookup error: /snap/core18/current/lib/x86_64-linux-gnu/libpthread.so.0: undefined symbol: __libc_vfork, version GLIBC_PRIVATE
Setting up the CNI

1. microk8s.kubectl get all -A

NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   pod/calico-node-m9jxn                         1/1     Running   0          2m26s
kube-system   pod/calico-kube-controllers-9969d55bb-b7t76   1/1     Running   0          2m25s

NAMESPACE   NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.152.183.1   <none>        443/TCP   5m21s

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   3m28s

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           3m28s

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-9969d55bb   1         1         1       2m25s

So as you can see above steps, resources weren’t removed after microk8s.reset command.

I tried to do a clean refresh and re-installed snapd like below:

sudo systemctl disable snapd.socket
sudo rm -rf /var/lib/snapd
sudo pacman -Rns snapd
sudo shutdown -r now #reboot the pc
paru -S snapd #re-install snapd
sudo systemctl enable --now snapd.socket

Surprisingly is that even I’ve not enabled dns yet, after above command and then I’ve just issued the command microk8s.kubectl get all -A, all calico pods and all of the resources still there (even I removed /var/lib/snapd and reboot the pc)

So why resources weren’t removed and how to have a clean reset microk8s?

Full microk8s inspect logs here