kubernetes: Flannel (NetworkPlugin cni) error: /run/flannel/subnet.env: no such file or directory

/kind bug

@kubernetes/sig-contributor-experience-bugs

What happened: Installed a single-node kubernetes cluster on centos 7 (VM running on virtual box); my application pod (created via k8s deployment) won’t go into Ready state

Pod Event: Warning FailedCreatePodSandBox . . . Kubelet . . . Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox . . . network for pod “companyemployees-deployment-766c7c7767-t7mc5”: NetworkPlugin cni failed to set up pod “companyemployees-deployment-766c7c7767-t7mc5_default” network: open /run/flannel/subnet.env: no such file or directory

In addition, it looks like the kubernetes coredns docker container keeps exiting – e.g. docker ps -a | grep -i coredns: 6341ce0be652 k8s.gcr.io/pause:3.1 “/pause” . . . Exited (0) 1 second ago k8s_POD_coredns-576cbf47c7-9bxxg_kube-system_e84afb7a-d7b7-11e8-bafa-08002745c4bc_581

What you expected to happen: Flannel not to have the error & Pod to go into ready state

How to reproduce it (as minimally and precisely as possible): Create a simple deployment after creating docker image and pushing the image to a private docker registry kubectl create -f companyemployees-deployment.yaml deployment yaml:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: companyemployees-deployment
  labels:
    app: companyemployees
spec:
  replicas: 1
  selector:
    matchLabels:
      app: companyemployees
  template:
    metadata:
      labels:
        app: companyemployees
    spec:
      containers:
      - name: companyemployees
        image: localhost:5000/companyemployees:1.0
        ports:
        - containerPort: 9092

Anything else we need to know?: ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 08:00:27:45:c4:bc brd ff:ff:ff:ff:ff:ff 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 08:00:27:21:0f:92 brd ff:ff:ff:ff:ff:ff 4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT link/ether 02:42:1b:04:1f:7c brd ff:ff:ff:ff:ff:ff 6: veth3f5bcb4@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT link/ether b2:1f:d4:fb:84:2e brd ff:ff:ff:ff:ff:ff link-netnsid 0 7: flannel.1: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN mode DEFAULT link/ether e6:44:ed:15:dd:97 brd ff:ff:ff:ff:ff:ff

Environment:

• Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“12”, GitVersion:“v1.12.1”, GitCommit:“4ed3216f3ec431b140b1d899130a69fc671678f4”, GitTreeState:“clean”, BuildDate:“2018-10-05T16:46:06Z”, GoVersion:“go1.10.4”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“12”, GitVersion:“v1.12.1”, GitCommit:“4ed3216f3ec431b140b1d899130a69fc671678f4”, GitTreeState:“clean”, BuildDate:“2018-10-05T16:36:14Z”, GoVersion:“go1.10.4”, Compiler:“gc”, Platform:“linux/amd64”}

• Cloud provider or hardware configuration: Single-node kubernetes cluster on CentOS 7 VM running on virtual box (virtual box is running on windows 7 pro)

• OS (e.g. from /etc/os-release): cat /etc/os-release: NAME=“CentOS Linux” VERSION=“7 (Core)” ID=“centos” ID_LIKE=“rhel fedora” VERSION_ID=“7” PRETTY_NAME=“CentOS Linux 7 (Core)” ANSI_COLOR=“0;31” CPE_NAME=“cpe:/o:centos:centos:7” HOME_URL=“https://www.centos.org/” BUG_REPORT_URL=“https://bugs.centos.org/”

CENTOS_MANTISBT_PROJECT=“CentOS-7” CENTOS_MANTISBT_PROJECT_VERSION=“7” REDHAT_SUPPORT_PRODUCT=“centos” REDHAT_SUPPORT_PRODUCT_VERSION=“7”

rpm -q centos-release centos-release-7-4.1708.el7.centos.x86_64

• Kernel (e.g. uname -a): uname -a Linux ibm-ms 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

My team’s centos image had docker, kubernetes, flannel and docker private registry already on the image; it was working and then recently I had issues w/ it that resulted in my uninstalling kubernetes, docker and flannel and reinstalling.

Install steps:

Switch to root: su - root

install docker

1. yum install -y yum-utils device-mapper-persistent-data lvm2
2. yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
3. yum install docker-ce
4. systemctl daemon-reload
5. systemctl enable docker
6. systemctl start docker
7. docker run hello-world

install private docker registry

1. docker pull registry
2. docker run -d -p 5000:5000 --restart=always --name registry registry
3. Note: firewalld is not running

install k8s:

1. setenforce 0
2. sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
3. swapoff -a
4. Edit /etc/fstab and comment-out /dev/mapper/centos-swap swap
5. Add kubernetes repo for yum - edit /etc/yum.repos.d/kubernetes.repo and add

[kubernetes]
	name=Kubernetes
	baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
	enabled=1
	gpgcheck=1
	repo_gpgcheck=1
	gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
		https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

1. yum install -y kubelet kubeadm kubectl
2. systemctl enable kubelet
3. systemctl start kubelet
4. kubeadm init --pod-network-cidr= 10.244.0.0/16
5. k8s config for user – running as root: export KUBECONFIG=/etc/kubernetes/admin.conf

install flannel:

1. sysctl net.bridge.bridge-nf-call-iptables=1
2. kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

Remove master node taint (to allow scheduling pods on master): kubectl taint nodes --all node-role.kubernetes.io/master-

• Others: Prior to installing, uninstalled using following steps:

Switch to root: su - root

Uninstall k8s (Although on master node, I did this a few times and included draining the node the last time)

1. kubectl drain mynodename --delete-local-data --force --ignore-daemonsets
2. kubectl delete node mynodename
3. kubeadm reset
4. systemctl stop kubelet
5. yum remove kubeadm kubectl kubelet kubernetes-cni kube*
6. yum autoremove
7. rm -rf ~/.kube
8. rm -rf /var/lib/kubelet/*

Uninstall docker:

1. docker rm docker ps -a -q``
2. docker stop (as needed)
3. docker rmi -f docker images -q``
4. Check that all containers and images were deleted: docker ps -a; docker images
5. systemctl stop docker
6. yum remove yum-utils device-mapper-persistent-data lvm2
7. yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-selinux docker-engine-selinux docker-engine
8. yum remove docker-ce
9. rm -rf /var/lib/docker 12. rm -rf /etc/docker

Uninstall flannel

1. rm -rf /var/lib/cni/
2. rm -rf /run/flannel
3. rm -rf /etc/cni/
4. Remove interfaces related to docker and flannel: ip link For each interface for docker or flannel, do the following ifconfig <name of interface from ip link> down ip link delete <name of interface from ip link>