microk8s: cilium on microk8s: core-dns, hostpath-provisioner and hubble fails.

Enabling cilium on microk8s results into cluster not working.

coredns stops working
hospath pod stops working looking into pod describe state shows warning (not sure if related): failed to find plugin “cilium-cni” in path [/var/snap/microk8s/2551/opt/cni/bin]

ENV:

Distributor ID:	Ubuntu
Description:	Ubuntu 21.10
Release:	21.10
Codename:	impish
microk8s 1.22/stable (Single node physical server)

inspection-report-20211018_140644.tar.gz

Plugins

microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    cilium               # SDN, fast with full network policy
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    helm3                # Helm 3 - Kubernetes package manager
    metallb              # Loadbalancer for your Kubernetes cluster
    storage              # Storage class; allocates storage from host directory

Enabling cilium plugin causes

dns pod fails (No policy applied)
hostPath provision fails (storage)

.:53
[INFO] plugin/reload: Running configuration MD5 = be0f52d3c13480652e0c73672f2fa263
CoreDNS-1.8.0
linux/amd64, go1.15.3, 054c9ae
[INFO] 127.0.0.1:38711 - 2364 "HINFO IN 8559940237512818014.7305508818995797580. udp 57 false 512" NOERROR - 0 6.001203496s
[ERROR] plugin/errors: 2 8559940237512818014.7305508818995797580. HINFO: read udp 10.0.0.146:50304->8.8.8.8:53: i/o timeout
[INFO] 127.0.0.1:40286 - 35177 "HINFO IN 8559940237512818014.7305508818995797580. udp 57 false 512" NOERROR - 0 6.000765939s
[ERROR] plugin/errors: 2 8559940237512818014.7305508818995797580. HINFO: read udp 10.0.0.146:33405->8.8.4.4:53: i/o timeout
[INFO] 127.0.0.1:58037 - 43993 "HINFO IN 8559940237512818014.7305508818995797580. udp 57 false 512" NOERROR - 0 4.000622412s
[ERROR] plugin/errors: 2 8559940237512818014.7305508818995797580. HINFO: read udp 10.0.0.146:60612->8.8.4.4:53: i/o timeout
I1018 12:06:34.490262       1 trace.go:205] Trace[774965466]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156 (18-Oct-2021 12:06:04.489) (total time: 30000ms):
Trace[774965466]: [30.000588379s] [30.000588379s] END
E1018 12:06:34.490290       1 reflector.go:127] pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://10.152.183.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.152.183.1:443: i/o timeout
I1018 12:06:34.490319       1 trace.go:205] Trace[436340495]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.19.2/tools/cache/reflector.go:156 (18-Oct-2021 12:06:04.489) (total time: 30000ms):

Looking at pod describe:

Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Warning  FailedScheduling        66m                    default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Warning  FailedScheduling        65m                    default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
  Normal   Scheduled               58m                    default-scheduler  Successfully assigned kube-system/coredns-7f7f9dc75b-fq7wx to antaresinc-cluster
  Warning  FailedCreatePodSandBox  58m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cf739b3ee917271f23fd6187b9e9f5f6417ecca0a942aba36acbf2bc25ee0b80": failed to find plugin "cilium-cni" in path [/var/snap/microk8s/2551/opt/cni/bin]

Cilium Hubble

Warning  FailedCreatePodSandBox  34m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d54e5a6079466384ab662149f9dc6f3c6865a82669b06a8b953d0bcc84817f8f": failed to find plugin "cilium-cni" in path [/var/snap/microk8s/2551/opt/cni/bin]

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 26 (11 by maintainers)

Most upvoted comments

I have downgraded Ubuntu 20.04 and cilium seems to be working now.

usma0118 on Oct 22, 2021

I think the next steps would probably need to involve trying something like using an older Ubuntu version to see whether this is a regression in the combination of Ubuntu 21.10 and Cilium 1.10.x. I can tell you that I was able to successfully install microk8s+cilium on Ubuntu 20.04 LTS here with a Linux 5.10.x kernel. If you’re able to try that out, it could help to confirm whether there is a problem with the integration to Ubuntu 21.10 or whether there is another problem in your environment.

In the mean time, I reached out to the cilium.io Slack community #development channel to see whether any others have experience running Cilium on this new version of Ubuntu yet, so hopefully we can find out whether that’s the problem and whether there are any mitigations or fixes there.

joestringer on Oct 18, 2021

From the above, doesn’t seem like an IPAM issue then. 🤔

joestringer on Oct 18, 2021

default via 192.100.1.1 dev eno2 proto dhcp src 192.100.1.120 metric 100 
10.0.0.0/24 via 10.0.0.196 dev cilium_host src 10.0.0.196 
10.0.0.196 dev cilium_host scope link 
192.100.1.0/24 dev eno2 proto kernel scope link src 192.100.1.120 
192.100.1.1 dev eno2 proto dhcp scope link src 192.100.1.120 metric 100

usma0118 on Oct 18, 2021

Well… it shows coredns entry.

NAMESPACE     NAME                       ENDPOINT ID   IDENTITY ID   INGRESS ENFORCEMENT   EGRESS ENFORCEMENT   VISIBILITY POLICY   ENDPOINT STATE   IPV4        IPV6
kube-system   coredns-7f9c69c78c-hzhpz   253           22566                                                                        ready            10.0.0.49

and i can see info via hubble observe as well

kubectl -n kube-system exec cilium-tsvmf -it -- hubble observe
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), clean-cilium-state (init)
Oct 18 19:51:06.531: kube-system/coredns-7f9c69c78c-hzhpz:42644 -> 8.8.4.4:53 to-stack FORWARDED (UDP)
Oct 18 19:51:06.714: 10.0.0.196:39900 <- kube-system/coredns-7f9c69c78c-hzhpz:8080 to-stack FORWARDED (TCP Flags: SYN, ACK)
Oct 18 19:51:06.884: kube-system/coredns-7f9c69c78c-hzhpz:34118 -> 8.8.8.8:53 to-stack FORWARDED (UDP)
Oct 18 19:51:06.970: 10.0.0.196:39374 <- kube-system/coredns-7f9c69c78c-hzhpz:8181 to-stack FORWARDED (TCP Flags: SYN, ACK)
Oct 18 19:51:07.994: kube-system/coredns-7f9c69c78c-hzhpz:56842 -> 8.8.4.4:53 to-stack FORWARDED (UDP)
Oct 18 19:51:08.467: kube-system/coredns-7f9c69c78c-hzhpz:42122 -> 8.8.8.8:53 to-stack FORWARDED (UDP)
Oct 18 19:51:09.018: kube-system/coredns-7f9c69c78c-hzhpz:56094 -> 192.100.1.120:16443 to-stack FORWARDED (TCP Flags: SYN)
Oct 18 19:51:09.274: kube-system/coredns-7f9c69c78c-hzhpz:56096 -> 192.100.1.120:16443 to-stack FORWARDED (TCP Flags: SYN)
Oct 18 19:51:09.457: kube-system/coredns-7f9c69c78c-hzhpz:51990 -> 8.8.4.4:53 to-stack FORWARDED (UDP)

usma0118 on Oct 18, 2021

nop. only info messages.

evel=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=668 identity=4 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Removed endpoint" containerID= datapathPolicyRevision=1 desiredPolicyRevision=1 endpointID=668 identity=4 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=1979 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Resolving identity labels (blocking)" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=1979 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Identity of endpoint changed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=1979 identity=4 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ oldIdentity="no identity" subsys=endpoint

usma0118 on Oct 18, 2021