kubernetes: resourcelock *EndpointsLock.Update may trigger panic

When using the resourcelock of the endpoints type, panic may be triggered if the update of the election record(e.g. tryAcquireOrRenew) fails.

func (el *EndpointsLock) Update(ler LeaderElectionRecord) error {
	if el.e == nil {
		return errors.New("endpoint not initialized, call get or create first")
	}
	recordBytes, err := json.Marshal(ler)
	if err != nil {
		return err
	}
	el.e.Annotations[LeaderElectionRecordAnnotationKey] = string(recordBytes)
	el.e, err = el.Client.Endpoints(el.EndpointsMeta.Namespace).Update(el.e)
	return err
}

If the el.Client.Endpoints(el.EndpointsMeta.Namespace).Update(el.e) call fails, it will return an empty &v1.Endpoints{}. The next time you call this method (e.g. release) will panic：

E1025 14:23:11.797180     528 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'LeaderElection' '11.168.116.130:8000 stopped leading'
time="2019-10-25T14:23:11+08:00" level=info msg="no longer leader, config: &{EndpointsMeta:{Name:cni-service GenerateName: Namespace:kube-system SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Initializers:nil Finalizers:[] ClusterName:} Client:0xc420413930 LockConfig:{Identity:11.168.116.130:8000 EventRecorder:0xc4201c1540} e:0xc420bf5320}"
panic: assignment to entry in nil map

goroutine 83 [running]:
xx.com/xx/xx/vendor/k8s.io/client-go/tools/leaderelection/resourcelock.(*EndpointsLock).Update(0xc42010eea0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
  /home/admin/107_20190819200832752_99814907_code/gopath/src/xx.com/xx/xx/vendor/k8s.io/client-go/tools/leaderelection/resourcelock/endpointslock.go:84 +0x137
xx.com/xx/xx/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).release(0xc42010f0e0,0xb2d05e00)
  /home/admin/107_20190819200832752_99814907_code/gopath/src/xx.com/xx/xx/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:281 +0xb1
xx.com/xx/xx/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew(0xc42010f0e0, 0x153c800, 0xc4200154c0)
  /home/admin/107_20190819200832752_99814907_code/gopath/src/xx.com/xx/xx/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:269 +0x115
xx.com/xx/xx/vendor/k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc42010f0e0, 0x153c800, 0xc420015440)
  /home/admin/107_20190819200832752_99814907_code/gopath/src/xx.com/xx/xx/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:183 +0x110
main.main.func1(0xc42010f0e0)
  /home/admin/107_20190819200832752_99814907_code/gopath/src/xx.com/xx/xx/main.go:77 +0x3b
created by main.main
  /home/admin/107_20190819200832752_99814907_code/gopath/src/xx.com/xx/xx/main.go:73 +0x791

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 16 (12 by maintainers)

Commits related to this issue

also fix nil panic in lease and add tests for #84729 — committed to mikedanese/kubernetes by mikedanese 5 years ago

Most upvoted comments

How about the following change ?

diff --git a/staging/src/k8s.io/client-go/tools/leaderelection/resourcelock/configmaplock.go b/staging/src/k8s.io/client-go/tools/leaderelection/resourcelock/configmaplock.go
index 608f7524991..d8b96d20908 100644
--- a/staging/src/k8s.io/client-go/tools/leaderelection/resourcelock/configmaplock.go
+++ b/staging/src/k8s.io/client-go/tools/leaderelection/resourcelock/configmaplock.go
@@ -90,6 +90,9 @@ func (cml *ConfigMapLock) Update(ctx context.Context, ler LeaderElectionRecord)
        }
        cml.cm.Annotations[LeaderElectionRecordAnnotationKey] = string(recordBytes)
        cml.cm, err = cml.Client.ConfigMaps(cml.ConfigMapMeta.Namespace).Update(ctx, cml.cm, metav1.UpdateOptions{})
+       if cml.cm.Annotations == nil {
+               cml.cm.Annotations = make(map[string]string)
+       }
        return err
 }

tedyu on Mar 20, 2020

https://github.com/kubernetes/kubernetes/blob/d1e8702d36947ccc23ad820eaae7f4afbaf0b058/staging/src/k8s.io/client-go/tools/leaderelection/resourcelock/configmaplock.go#L51-L53

@mikedanese, that doesn’t cover the case in (*ConfigMapLock).Update when the call to ConfigMapsGetter.Update fails, returning a non-nil error and an empty *v1.ConfigMap. At that point, cml.cm is non-nil, but cml.cm.Annotations is nil.

seh on Mar 20, 2020