kubernetes: Internal PreStartContainer hook failed: not enough cpus available to satisfy request

/kind bug What happened: Sometimes when an deployment creates new pods they get scheduled on a node successfully but the containers fail to start with:

Warning  Failed                 14m (x8 over 15m)    kubelet, ip-172-20-162-142.ec2.internal  Internal PreStartContainer hook failed: not enough cpus available to satisfy request

What you expected to happen: The scheduler shouldn’t put pods on nodes without enough resources.

How to reproduce it (as minimally and precisely as possible): I don’t know how it got into this situation. Deleting the pods and have the RS recreate them works around this issue.

Anything else we need to know?: We’re using the CPUManager with static policy which might contribute to this problem.

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-beta.2", GitCommit:"63dad40a0391b7af32c34fdbf41fa199c3b247ad", GitTreeState:"clean", BuildDate:"2018-03-07T20:42:18Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): coreos

/sig scheduling