kubernetes: kubelet failed to record some events.

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug

Kubernetes version (use kubectl version): master

What happened:

E0306 07:26:10.013767    3385 event.go:259] Could not construct reference to: '&v1.Node{...}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'NodeAllocatableEnforced' '       Updated Node Allocatable limit across pods'

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 3
Comments: 20 (12 by maintainers)

Commits related to this issue

Merge pull request #42939 from k82cn/k8s_42701 Automatic merge from submit-queue Used ObjectReference for events. fixes #42701 ```release-note None ``` — committed to kubernetes/kubernetes by deleted user 7 years ago

Most upvoted comments

Hi All,

We’re also experiencing this with 1.8.2 with docker 1.13 :

Failed to update Node Allocatable Limits "": failed to set supported cgroup subsystems for cgroup : Failed to set config for supported subsystems : failed to write 8201408512 to memory.limit_in_bytes: write /var/lib/docker/devicemapper/mnt/69641d63999364ce6bed9ff9a37e18922d1738df76d84db691a854f0e931e435/rootfs/sys/fs/cgroup/memory/memory.limit_in_bytes: invalid argument

any updates on why this is happening on 1.8 ?

sokoow on Nov 16, 2017

ok, let me creat a backmerge pr for 1.6

k82cn on Jul 6, 2017

To bump this, we’re also experiencing this exact issue above ^ on a 3-node AKS cluster running k8s 1.8.7. Here’s the output from kubectl describe node <hostname>:

  Type     Reason                            Age               From                               Message
  ----     ------                            ----              ----                               -------
  Warning  FailedNodeAllocatableEnforcement  1m (x96 over 1h)  kubelet, aks-agentpool-10045148-3  Failed to update Node Allocatable Limits "": failed to set supported cgroup subsystems for cgroup : Failed to set config for supported subsystems : failed to write 4114153472 to memory.limit_in_bytes: write /var/lib/docker/overlay2/df563466adada9e2d5109d2d6084fa59fdf3b106cc6424b276834d7f80fa2f1e/merged/sys/fs/cgroup/memory/memory.limit_in_bytes: invalid argument

Is there any fixes for this?

joshuatalb on Mar 20, 2018

@anilreddyv , @ZhangYou0122 the fix was not included in 1.6; if necessary, we can backmerge it.

k82cn on Jun 26, 2017

@tristanls , thanks very much for your verification. That maybe my miss on the usage of that event to make it non-release-blocker. Anyway, I’ll ping approver to get that PR merged; and dig why it impacts cluster configuration.

k82cn on Apr 18, 2017

Append node infos, it seems UID is also empty:

E0306 07:26:10.013767 3385 event.go:259] Could not construct reference to: ‘&v1.Node{TypeMeta:v1.TypeMeta{Kind:“”, APIVersion:“”}, ObjectMeta:v1.ObjectMeta{Name:“bootstrap-e2e-minion-group-sh51”, GenerateName:“”, Namespace:“”, SelfLink:“”, UID:“”, ResourceVersion :“”, Generation:0, CreationTimestamp:v1.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)( nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{“beta.kubernetes.io/instance-type”:“n1-standard-2”, “failure -domain.beta.kubernetes.io/zone”:“us-central1-f”, “failure-domain.beta.kubernetes.io/region”:“us-central1”, “beta.kubernetes.io/os”:" linux", “beta.kubernetes.io/arch”:“amd64”, “alpha.kubernetes.io/fluentd-ds-ready”:“true”, “kubernetes.io/hostname”:“bootstrap-e2e-min ion-group-sh51”, “agent-light-10”:“”}, Annotations:map[string]string{“volumes.kubernetes.io/controller-managed-attach-detach”:“true”} , OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:“”}, Spec:v1.NodeSpec{PodCIDR:“”, ExternalID:“51070 54328687931504”, ProviderID:“gce://kubernetes-jkns-e2e-gce-serial/us-central1-f/bootstrap-e2e-minion-group-sh51”, Unschedulable:false , Taints:[]v1.Taint(nil)}, Status:v1.NodeStatus{Capacity:v1.ResourceList{“cpu”:resource.Quantity{i:resource.int64Amount{value:2000, s cale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:“”, Format:“DecimalSI”}, “memory”:resource.Quantity{i:resource.int64Amount{ value:7864107008, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:“”, Format:“BinarySI”}, “pods”:resource.Quantity{i:resour ce.int64Amount{value:110, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:“”, Format:“DecimalSI”}}, Allocatable:v1.Resource List{“cpu”:resource.Quantity{i:resource.int64Amount{value:2000, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:“”, Format :“DecimalSI”}, “memory”:resource.Quantity{i:resource.int64Amount{value:7601963008, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(n il)}, s:“”, Format:“BinarySI”}, “pods”:resource.Quantity{i:resource.int64Amount{value:110, scale:0}, d:resource.infDecAmount{Dec:(*in f.Dec)(nil)}, s:“”, Format:“DecimalSI”}}, Phase:“”, Conditions:[]v1.NodeCondition{v1.NodeCondition{Type:“NetworkUnavailable”, Status: “True”, LastHeartbeatTime:v1.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Tim e{sec:63624381969, nsec:996530228, loc:(*time.Location)(0x4e91740)}}, Reason:“NoRouteCreated”, Message:“Node created without a route” }, v1.NodeCondition{Type:“OutOfDisk”, Status:“False”, LastHeartbeatTime:v1.Time{Time:time.Time{sec:63624381970, nsec:5781404, loc:(*t ime.Location)(0x4e91740)}}, LastTransitionTime:v1.Time{Time:time.Time{sec:63624381970, nsec:5781404, loc:(*time.Location)(0x4e91740)} }, Reason:“KubeletHasSufficientDisk”, Message:“kubelet has sufficient disk space available”}, v1.NodeCondition{Type:“MemoryPressure”, Status:“False”, LastHeartbeatTime:v1.Time{Time:time.Time{sec:63624381970, nsec:5843328, loc:(*time.Location)(0x4e91740)}}, LastTrans itionTime:v1.Time{Time:time.Time{sec:63624381970, nsec:5843328, loc:(*time.Location)(0x4e91740)}}, Reason:“KubeletHasSufficientMemory “, Message:“kubelet has sufficient memory available”}, v1.NodeCondition{Type:“DiskPressure”, Status:“False”, LastHeartbeatTime:v1.Tim e{Time:time.Time{sec:63624381970, nsec:5858974, loc:(*time.Location)(0x4e91740)}}, LastTransitionTime:v1.Time{Time:time.Time{sec:6362 4381970, nsec:5858974, loc:(*time.Location)(0x4e91740)}}, Reason:“KubeletHasNoDiskPressure”, Message:“kubelet has no disk pressure”}, v1.NodeCondition{Type:“Ready”, Status:“False”, LastHeartbeatTime:v1.Time{Time:time.Time{sec:63624381970, nsec:5874905, loc:(*time.Lo cation)(0x4e91740)}}, LastTransitionTime:v1.Time{Time:time.Time{sec:63624381970, nsec:5874905, loc:(*time.Location)(0x4e91740)}}, Rea son:“KubeletNotReady”, Message:“container runtime is down,PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago ; threshold is 3m0s,network state unknown”}}, Addresses:[]v1.NodeAddress{v1.NodeAddress{Type:“InternalIP”, Address:“10.240.0.5”}, v1. NodeAddress{Type:“ExternalIP”, Address:“146.148.66.23”}, v1.NodeAddress{Type:“Hostname”, Address:“bootstrap-e2e-minion-group-sh51”}}, DaemonEndpoints:v1.NodeDaemonEndpoints{KubeletEndpoint:v1.DaemonEndpoint{Port:10250}}, NodeInfo:v1.NodeSystemInfo{MachineID:””, Syst emUUID:“0C3E9175-B5A0-F5AB-D1C8-D26CFE2E8639”, BootID:“39fc8721-1f50-4b42-a044-c693e3f0ecea”, KernelVersion:“3.16.0-4-amd64”, OSImage :“Debian GNU/Linux 7 (wheezy)”, ContainerRuntimeVersion:“docker://1.11.2”, KubeletVersion:“v1.7.0-alpha.0.849+df70b30e59b5da”, KubePr oxyVersion:“v1.7.0-alpha.0.849+df70b30e59b5da”, OperatingSystem:“linux”, Architecture:“amd64”}, Images:[]v1.ContainerImage{v1.Contain erImage{Names:[]string{“gcr.io/google_containers/kube-proxy:ccb1e93b18593cec918935ec091a0cd7”}, SizeBytes:109638053}, v1.ContainerIma ge{Names:[]string{“gcr.io/google_containers/pause-amd64:3.0”}, SizeBytes:746888}}, VolumesInUse:[]v1.UniqueVolumeName(nil), VolumesAt tached:[]v1.AttachedVolume(nil)}}’ due to: ‘selfLink was empty, can’t make reference’. Will not report event: ‘Normal’ ‘NodeAllocatab leEnforced’ ‘Updated Node Allocatable limit across pods’

k82cn on Mar 11, 2017