calico: [solved] Calico doesn't start correctly on minikube

Expected Behavior

Installing Calico on minikube should start both containers in the calico-node pod.

Current Behavior

I recently upgraded minikube to 0.21, which means deleting the cluster and reinstalling it. When I tried to reinstall Calico, the calico-node container in the calico node pod never comes up, and networking is broken.

I’m running minikube with minikube start --network-plugin=cni --host-only-cidr=192.168.0.0/16, using virtualbox as the driver.

The logs for calico-node contain entries like:

2017-08-10 00:28:51.825 [PANIC][2269] ipsets.go 283: Failed to update IP sets after mutliple retries. family="inet"
panic: (*logrus.Entry) (0x16df120,0xc4204744b0)
github.com/projectcalico/felix/vendor/github.com/sirupsen/logrus.(*Entry).Panic(0xc42041a870, 0xc4208d5ee0, 0x1, 0x1)

2017-08-10 00:29:15.131 [WARNING][2972] ipsets.go 271: Failed to update IP sets. Marking dataplane for resync. error=exit status 1 family="inet"
2017-08-10 00:29:15.270 [WARNING][2972] ipsets.go 621: Failed to complete ipset restore, IP sets may be out-of-sync. closeErr=<nil> commitErr=<nil> family="inet" flushErr=<nil> input="create cali4tall-hosts hash:ip family inet maxelem 1048576\nadd cali4tall-hosts 10.1.0.1\nswap cali4-all-hosts cali4tall-hosts\ndestroy cali4tall-hosts\ncreate cali4-all-ipam-pools hash:net family inet maxelem 1048576\ncreate cali4tall-ipam-pools hash:net family inet maxelem 1048576\nadd cali4tall-ipam-pools 192.168.0.0/16\nswap cali4-all-ipam-pools cali4tall-ipam-pools\ndestroy cali4tall-ipam-pools\ncreate cali4-masq-ipam-pools hash:net family inet maxelem 1048576\ncreate cali4tmasq-ipam-pools hash:net family inet maxelem 1048576\nadd cali4tmasq-ipam-pools 192.168.0.0/16\nswap cali4-masq-ipam-pools cali4tmasq-ipam-pools\ndestroy cali4tmasq-ipam-pools\nCOMMIT\n" processErr=exit status 1 stderr="ipset v6.32: Error in line 5: Kernel error received: set type not supported\n" stdout="" writeErr=<nil>
2017-08-10 00:29:15.270 [WARNING][2972] ipsets.go 271: Failed to update IP sets. Marking dataplane for resync. error=exit status 1 family="inet"

Steps to Reproduce (for bugs)

Install minikube with homebrew (https://github.com/kubernetes/minikube).
Install Virtualbox (https://www.virtualbox.org/wiki/Downloads).

Start minikube with minikube start --network-plugin=cni --host-only-cidr=192.168.0.0/16
Install calico with kubectl apply -f https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.6/calico.yaml

Context

I’m trying to create a working development environment for testing network policy on my local machine so I don’t have to depend on a remote cluster for doing development work. I know this is beta software, but it was working so I suspect that a change in Calico or minikube is responsible.

Your Environment

Calico: v2.4 minikube: v0.21.0, Kubernetes version 1.7.0 OS: Mac OS X 10.12.6

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 31 (11 by maintainers)

Most upvoted comments

@beeradb Thank you for your report on what worked or didn’t for you. I’ve been spending some time trying to understand this and have figured out some options to make Calico work with minikube. There are 2 ways to run Calico policy-only mode and networking (which includes policy) mode. I’m going go through using both here, if you (or anyone) are able to try them out please report back your results.

Note: In both set ups below the pods that are deployed as minikube addons will be brought up with the default networking that minikube sets up. This does not seem to hinder much though because of that it is not possible to create policy that will apply to the addon pods. I attempted to restart the addon pods so they would be networked with the newly configured networking but for some reason after that I could not reach the dns.

Policy-only mode (Kubernetes networking)

In this mode Calico is not setting up the networking but uses the ‘host-local’ plugin for IPAM so depends on K8s for IPAM. The minikube start command I found to work is:

Minikube v0.21

minikube start --network-plugin=cni --host-only-cidr 172.17.17.1/24 \
    --extra-config=kubelet.ClusterCIDR=192.168.0.0/16 \
    --extra-config=proxy.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.CIDRAllocatorType=RangeAllocator \
    --extra-config=controller-manager.AllocateNodeCIDRs=true \
    --iso-url=https://storage.googleapis.com/minikube-builds/1817/minikube.iso

Minikube v0.22

For minikube v0.22.1 it seems like the options have changed, I found that the following worked with it: (also notice the --iso-url is not needed with v0.22.1)

minikube start --network-plugin=cni --host-only-cidr 172.17.17.1/24 \
    --extra-config=kubelet.PodCIDR=192.168.0.0/16 \
    --extra-config=proxy.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.CIDRAllocatorType=RangeAllocator \
    --extra-config=controller-manager.AllocateNodeCIDRs=true

I found that without CIDRAllocatorType or ClusterCIDR set on the controller-manager you would not have a working api-server. (@beeradb, I think this is what you saw when you didn’t specify kubernetes 1.6.)

After K8s is started with the above then it was possible to install calico with kubectl apply -f https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.6/calico.yaml

Note: I’m not sure why but I had to specify the ClusterCIDR, the default would not work. I also tried specifying the default CIDR to the controller-manager but then I would end up with multiple pods with the same IP addresses.

Calico networking mode

In this mode Calico handles configuring the networking and provides policy.

Minikube v0.21

minikube start --network-plugin=cni --host-only-cidr 172.17.17.1/24 \
    --extra-config=kubelet.ClusterCIDR=192.168.0.0/16 \
    --extra-config=proxy.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16 \
    --iso-url=https://storage.googleapis.com/minikube-builds/1817/minikube.iso

Minikube v0.22

minikube start --network-plugin=cni --host-only-cidr 172.17.17.1/24 \
    --extra-config=kubelet.PodCIDR=192.168.0.0/16 \
    --extra-config=proxy.ClusterCIDR=192.168.0.0/16 \
    --extra-config=controller-manager.ClusterCIDR=192.168.0.0/16

After K8s is started with the above then it is possible to install Calico though some slight modifications will need to be made to the manifests. The following commands can be used to pull down a manifest, update it, and then apply the manifest.

curl -O -L https://docs.projectcalico.org/v2.4/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml
sed -i -e '/nodeSelector/d' calico.yaml
sed -i -e '/node-role.kubernetes.io\/master: ""/d' calico.yaml
sed -i -e 's/10\.96\.232/10.0.0/' calico.yaml
kubectl apply -f calico.yaml

tmjd on Sep 15, 2017

This works fine with Calico v3.1 as well. https://docs.projectcalico.org/v3.1/getting-started/kubernetes/

install Calico v3.1

curl -O -L https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml
sed -i -e '/nodeSelector/d' calico.yaml
sed -i -e '/node-role.kubernetes.io\/master: ""/d' calico.yaml
kubectl apply -f calico.yaml

Camerons-MacBook-Pro:calico cameron$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY     STATUS    RESTARTS   AGE
kube-system   calico-etcd-krb9d                          1/1       Running   0          18m
kube-system   calico-kube-controllers-685755779f-smwrs   1/1       Running   0          18m
kube-system   calico-node-4cjzb                          2/2       Running   0          18m
kube-system   kube-addon-manager-minikube                1/1       Running   0          22m
kube-system   kube-dns-6dcb57bcc8-pwswp                  3/3       Running   0          22m
kube-system   kubernetes-dashboard-5498ccf677-tvj4t      1/1       Running   0          22m
kube-system   storage-provisioner                        1/1       Running   0          22m

ctaggart on May 30, 2018

Quick testing indicates that only --vm-driver=none will work with Calico. I tried out hyperkit and observed that Calico (specifically Felix) was not correctly populating iptables / ipsets rules. I believe this limits Calico+minikube to Linux only.

bcreane on Jun 13, 2018

@taylorKonigsmark I have not tried 0.24.1. It looks like maybe minikube has changed the service CIDR they use. I think you should try again but not do sed -i -e 's/10\.96\.232/10.0.0/' calico.yaml.

What it was attempting to do was put the service IP created for the calico-etcd service into the used service CIDR.

tmjd on Dec 22, 2017

I spent some time trying these out today and both versions worked as expected for me. I did not do a thorough test, but in both cases, I introduced a default-deny policy into the cluster and had networking restricted.

Thanks again for your work on resolving this.

beeradb on Aug 30, 2017

I verified it today with the new iso, it works.

sbueringer on Aug 14, 2017