kubernetes: ./hack/local-up-cluster.sh fails on Fedora rawhide

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

I attempted to start all-on-one testing environment with

ENABLE_DAEMON=true PATH='$(pwd)/kubernetes/third_party/etcd:${PATH}' ./hack/local-up-cluster.sh

on Fedora rawhide, the way it works just fine on Fedora 27. The node never becomes Ready.

What you expected to happen:

The ./hack/local-up-cluster.sh finishing and the node 127.0.0.1 becoming Ready shortly after.

How to reproduce it (as minimally and precisely as possible):

git clone https://github.com/kubernetes/kubernetes.git
( cd kubernetes && ./hack/install-etcd.sh )
( cd kubernetes && ENABLE_DAEMON=true PATH='$(pwd)/kubernetes/third_party/etcd:${PATH}' ./hack/local-up-cluster.sh )
kubectl get nodes

Anything else we need to know?:

On Fedora 27, these steps give me working setup.

The kubectl describe node 127.0.0.1 says (among others)

Conditions:
  Type             Status    LastHeartbeatTime                 LastTransitionTime                Reason                    Message
  ----             ------    -----------------                 ------------------                ------                    -------
  OutOfDisk        Unknown   Wed, 21 Mar 2018 09:23:21 -0400   Wed, 21 Mar 2018 09:24:04 -0400   NodeStatusUnknown         Kubelet stopped posting node status.
  MemoryPressure   Unknown   Wed, 21 Mar 2018 09:23:21 -0400   Wed, 21 Mar 2018 09:24:04 -0400   NodeStatusUnknown         Kubelet stopped posting node status.
  DiskPressure     Unknown   Wed, 21 Mar 2018 09:23:21 -0400   Wed, 21 Mar 2018 09:24:04 -0400   NodeStatusUnknown         Kubelet stopped posting node status.
  PIDPressure      False     Wed, 21 Mar 2018 09:23:21 -0400   Wed, 21 Mar 2018 09:23:21 -0400   KubeletHasSufficientPID   kubelet has sufficient PID available
  Ready            Unknown   Wed, 21 Mar 2018 09:23:21 -0400   Wed, 21 Mar 2018 09:24:04 -0400   NodeStatusUnknown         Kubelet stopped posting node status.

The /tmp/kubelet.log ends with

I0321 09:23:23.492042   27717 factory.go:110] Factory "systemd" can handle container "/system.slice/dev-hugepages.mount", but ignoring.
I0321 09:23:23.492059   27717 factory.go:110] Factory "systemd" can handle container "/system.slice/tmp.mount", but ignoring.
I0321 09:23:23.496752   27717 cpu_manager.go:155] [cpumanager] starting with none policy
I0321 09:23:23.496810   27717 cpu_manager.go:156] [cpumanager] reconciling every 10s
I0321 09:23:23.496830   27717 policy_none.go:42] [cpumanager] none policy: Start
I0321 09:23:23.517797   27717 factory.go:113] Using factory "raw" for container "/libcontainer_27717_systemd_test_default.slice"
I0321 09:23:23.518259   27717 manager.go:997] Added container: "/libcontainer_27717_systemd_test_default.slice" (aliases: [], namespace: "")
I0321 09:23:23.518614   27717 container.go:448] Start housekeeping for container "/libcontainer_27717_systemd_test_default.slice"
I0321 09:23:23.521188   27717 manager.go:1054] Destroyed container: "/libcontainer_27717_systemd_test_default.slice" (aliases: [], namespace: "")
E0321 09:23:23.527391   27717 node_container_manager.go:52] Failed to create "/kubepods" cgroup
F0321 09:23:23.527441   27717 kubelet.go:1363] Failed to start ContainerManager Delegation not available for unit type

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.1341+c5d4a032d73f90", GitCommit:"c5d4a032d73f90f286fde1a7ca3e782538c47f58", GitTreeState:"clean", BuildDate:"2018-03-21T13:21:57Z", GoVersion:"go1.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.1341+c5d4a032d73f90", GitCommit:"c5d4a032d73f90f286fde1a7ca3e782538c47f58", GitTreeState:"clean", BuildDate:"2018-03-21T13:21:57Z", GoVersion:"go1.10", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
I0321 09:23:21.140790   27717 manager.go:227] Machine: {NumCores:8 CpuFrequency:2412325 MemoryCapacity:16817651712 HugePages:[{PageSize:2048 NumPages:0}] MachineID:f72669acdc2840be9a3ca8a2019e3fdb SystemUUID:00020003-0004-0005-0006-000700080009 BootID:39dff605-2d5a-4abc-9cc8-c26c44c0c601 Filesystems:[{Device:tmpfs DeviceMajor:0 DeviceMinor:20 Capacity:8408825856 Type:vfs Inodes:2052936 HasInodes:true} {Device:/dev/mapper/fedora-root DeviceMajor:253 DeviceMinor:0 Capacity:16095641600 Type:vfs Inodes:7864320 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity:1023303680 Type:vfs Inodes:65536 HasInodes:true}] DiskMap:map[253:1:{Name:dm-1 Major:253 Minor:1 Size:8476688384 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:897581056 Scheduler:none} 253:3:{Name:dm-3 Major:253 Minor:3 Size:346738917376 Scheduler:none} 253:4:{Name:dm-4 Major:253 Minor:4 Size:346738917376 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:896998047744 Scheduler:cfq} 253:0:{Name:dm-0 Major:253 Minor:0 Size:16106127360 Scheduler:none}] NetworkDevices:[{Name:enp2s0 MacAddress:00:d0:68:10:03:66 Speed:1000 Mtu:1500} {Name:enp3s0 MacAddress:00:d0:68:10:03:67 Speed:-1 Mtu:1500}] Topology:[{Id:0 Memory:4159594496 Cores:[{Id:0 Threads:[0] Caches:[{Size:65536 Type:Data Level:1} {Size:65536 Type:Instruction Level:1} {Size:1048576 Type:Unified Level:2}]} {Id:1 Threads:[1] Caches:[{Size:65536 Type:Data Level:1} {Size:65536 Type:Instruction Level:1} {Size:1048576 Type:Unified Level:2}]}] Caches:[]} {Id:1 Memory:4227317760 Cores:[{Id:0 Threads:[2] Caches:[{Size:65536 Type:Data Level:1} {Size:65536 Type:Instruction Level:1} {Size:1048576 Type:Unified Level:2}]} {Id:1 Threads:[3] Caches:[{Size:65536 Type:Data Level:1} {Size:65536 Type:Instruction Level:1} {Size:1048576 Type:Unified Level:2}]}] Caches:[]} {Id:2 Memory:4203909120 Cores:[{Id:0 Threads:[4] Caches:[{Size:65536 Type:Data Level:1} {Size:65536 Type:Instruction Level:1} {Size:1048576 Type:Unified Level:2}]} {Id:1 Threads:[5] Caches:[{Size:65536 Type:Data Level:1} {Size:65536 Type:Instruction Level:1} {Size:1048576 Type:Unified Level:2}]}] Caches:[]} {Id:3 Memory:4226830336 Cores:[{Id:0 Threads:[6] Caches:[{Size:65536 Type:Data Level:1} {Size:65536 Type:Instruction Level:1} {Size:1048576 Type:Unified Level:2}]} {Id:1 Threads:[7] Caches:[{Size:65536 Type:Data Level:1} {Size:65536 Type:Instruction Level:1} {Size:1048576 Type:Unified Level:2}]}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
  • OS (e.g. from /etc/os-release):
NAME=Fedora
VERSION="29 (Rawhide)"
ID=fedora
VERSION_ID=29
PRETTY_NAME="Fedora 29 (Rawhide)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:29"
HOME_URL="https://fedoraproject.org/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=rawhide
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=rawhide
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
  • Kernel (e.g. uname -a):
Linux redacted.example.com 4.16.0-0.rc5.git3.1.fc29.x86_64 #1 SMP Fri Mar 16 15:28:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
  • Others:
systemd-238-4.fc29.x86_64

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 46 (30 by maintainers)

Commits related to this issue

Most upvoted comments

Now the kubelet.log ends with

I0321 10:22:51.143239   22943 container.go:448] Start housekeeping for container "/libcontainer_22943_systemd_test_default.slice"
W0321 10:22:51.144407   22943 container.go:406] Failed to get RecentStats("/libcontainer_22943_systemd_test_default.slice") while determining the next housekeeping: unable to find data for container /libcontainer_22943_systemd_test_default.slice
I0321 10:22:51.144413   22943 manager.go:1054] Destroyed container: "/libcontainer_22943_systemd_test_default.slice" (aliases: [], namespace: "")
I0321 10:22:51.144473   22943 handler.go:325] Added event &{/libcontainer_22943_systemd_test_default.slice 2018-03-21 10:22:51.144455959 -0400 EDT m=+3.030844677 containerDeletion {<nil>}}
E0321 10:22:51.152560   22943 node_container_manager.go:52] Failed to create "/kubepods" cgroup
F0321 10:22:51.152602   22943 kubelet.go:1363] Failed to start ContainerManager Delegation not available for unit type