kubernetes: Fail to get rootfs information unable to find data for container /

After debugging I found it’s caused by cgroup in my case. Similar to #55386, on my server cpuacct.stat has 6 fields which expected to be 4 and cadvisor cannot collect container metrics.

Modify some codes at vendor/github.com/opencontainers/runc/libcontainer/cgroups/fs/cpuacct.go#L88 and recompile kubelet works for me.

...
    fields := strings.Fields(string(data))
    if len(fields) != 4 && len(fields) != 6{
        return 0, 0, fmt.Errorf("failure - %s is expected to have 4 or 6 fields", filepath.Join(path, cgroupCpuacctStat))
    }

error log from kubelet v1.9.1:

Jan 15 12:36:53 l23-27-101 kubelet[7335]: E0115 12:36:53.885011    7335 container_manager_linux.go:583] [ContainerManager]: Fail to get rootfs information unable to find data for container /
Jan 15 12:36:54 l23-27-101 kubelet[7335]: E0115 12:36:54.885214    7335 container_manager_linux.go:583] [ContainerManager]: Fail to get rootfs information unable to find data for container /

Major 0 is used by the linux kernel for mountpoints which do not actually have a direct backing device. This applies to tmpfs, nsfs, overlayfs and also btrfs subvolumes for example.

the same error from kubelet 1.9.1:

root@ubuntu-64:/opt/bin# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2018-01-28 10:01:55 CST; 17min ago
     Docs: https://github.com/kubernetes
 Main PID: 20940 (kubelet)
    Tasks: 21
   Memory: 71.0M
      CPU: 58.328s
   CGroup: /system.slice/kubelet.service
           └─20940 /opt/bin/kubelet --logtostderr=false --v=2 --log-dir=/var/log/kubernetes --address=0.0.0.0 --hostname-override=ubuntu-64 --allow-privileged=true --cgroup-driver=systemd --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cl

Jan 28 10:01:55 ubuntu-64 systemd[1]: Started Kubernetes Kubelet Server.
Jan 28 10:02:00 ubuntu-64 kubelet[20940]: E0128 10:02:00.996798   20940 kubelet.go:1275] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data for container /
Jan 28 10:02:01 ubuntu-64 kubelet[20940]: E0128 10:02:01.060356   20940 container_manager_linux.go:583] [ContainerManager]: Fail to get rootfs information unable to find data for container /
Jan 28 10:02:02 ubuntu-64 kubelet[20940]: E0128 10:02:02.060852   20940 container_manager_linux.go:583] [ContainerManager]: Fail to get rootfs information unable to find data for container /
Jan 28 10:02:03 ubuntu-64 kubelet[20940]: E0128 10:02:03.061452   20940 container_manager_linux.go:583] [ContainerManager]: Fail to get rootfs information unable to find data for container /

Ughhh: https://github.com/kubernetes/kubernetes/pull/40050

With kubelet.kubeconfig:

apiVersion: v1
kind: Config
clusters:
  - cluster:
      server: http://127.0.0.1:8080/
    name: local
contexts:
  - context:
      cluster: local
    name: local
current-context: local

and setting --fail-swap-on=false --kubeconfig=/var/lib/kubelet/kubelet.kubeconfig I was able to create local cluster with f27 build of kubernetes-1.9.1

+1 Having error: Fail to get rootfs information unable to find data for container / kubeadm, 1.9.1-0 kubectl, 1.9.1-0 kubelet, 1.9.1-0 kubernetes-cni, 0.6.0-0

docker: 18.03.1-ce overlay2 on xfs

+1

kubernets: v1.9.2 docker: 17.03.2-ce

on host, du command hang there when this error happening

@voor can you verify?