ingress-nginx: leaderelection: Failed to update lock

Hello, I’m testing the 0.9.0-beta.12 release and I see this error in log:

[gaudy-flee-nginx-ingress-controller-vrqxl] E0904 09:53:44.017545       5 leaderelection.go:258] Failed to update lock: Operation cannot be fulfilled on configmaps "ingress-controller-leader-nginx": the object has been modified; please apply your changes to the latest version and try again

The nginx-ingress controller is deployed as a DaemonSet and it’s the only ingress controller on the cluster.

Here is content of ingress-controller-leader-nginx ConfigMap:

$ kubectl --namespace internal describe configmap ingress-controller-leader-nginx                                                                                                                           
Name:           ingress-controller-leader-nginx
Namespace:      internal
Labels:         <none>
Annotations:    control-plane.alpha.kubernetes.io/leader={"holderIdentity":"ingress-controller-leader-nginx","leaseDurationSeconds":30,"acquireTime":"2017-09-04T09:10:13Z","renewTime":"2017-09-04T09:56:28Z","leaderTr...

Data
====
Events: <none>

Another suspicious observation is that each pod is elected itself as the leader:

I0904 09:40:22.489304       5 leaderelection.go:174] attempting to acquire leader lease...                                                                                                                          I0904 09:40:22.489800       5 status.go:152] skipping Ingress status update (I am not the current leader)                                                                                                           
I0904 09:40:22.496448       5 template.go:92] adjusting template buffer size from 65535 to 262484                                                                                                                   
I0904 09:40:22.506258       5 leaderelection.go:184] successfully acquired lease internal/ingress-controller-leader-nginx                                                                                           I0904 09:40:22.506374       5 status.go:171] new leader elected (gaudy-flee-nginx-ingress-controller-vrqxl)                                                                                                         
I0904 09:40:22.506397       5 status.go:173] I am the new status update leader

This is logged by each of the pods.

Is that because the leader election doesn’t work correctly?

Another issue: too many update events, similar to #1289:

Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath   Type            Reason  Message
  ---------     --------        -----   ----                    -------------   --------        ------  -------
  1h            49m             80      ingress-controller                      Normal          UPDATE  Ingress hosting/hosting-yg5ohbhu-custom
  1h            49m             80      ingress-controller                      Normal          UPDATE  Ingress hosting/hosting-yg5ohbhu-custom
  1h            49m             80      ingress-controller                      Normal          UPDATE  Ingress hosting/hosting-yg5ohbhu-custom
  1h            49m             53      ingress-controller                      Normal          UPDATE  Ingress hosting/hosting-yg5ohbhu-custom
  1h            49m             81      ingress-controller                      Normal          UPDATE  Ingress hosting/hosting-yg5ohbhu-custom
  48m           48m             1       ingress-controller                      Normal          CREATE  Ingress hosting/hosting-yg5ohbhu-custom
  48m           48m             1       ingress-controller                      Normal          CREATE  Ingress hosting/hosting-yg5ohbhu-custom
  48m           48m             1       ingress-controller                      Normal          CREATE  Ingress hosting/hosting-yg5ohbhu-custom
  48m           48m             1       ingress-controller                      Normal          CREATE  Ingress hosting/hosting-yg5ohbhu-custom
  48m           48m             1       ingress-controller                      Normal          CREATE  Ingress hosting/hosting-yg5ohbhu-custom
  47m           47m             1       ingress-controller                      Normal          UPDATE  Ingress hosting/hosting-yg5ohbhu-custom
  47m           47m             1       ingress-controller                      Normal          UPDATE  Ingress hosting/hosting-yg5ohbhu-custom
  47m           47m             1       ingress-controller                      Normal          UPDATE  Ingress hosting/hosting-yg5ohbhu-custom
  47m           47m             1       ingress-controller                      Normal          UPDATE  Ingress hosting/hosting-yg5ohbhu-custom
  47m           47m             1       ingress-controller                      Normal          UPDATE  Ingress hosting/hosting-yg5ohbhu-custom
  18m           18m             1       ingress-controller                      Normal          CREATE  Ingress hosting/hosting-yg5ohbhu-custom
  18m           18m             1       ingress-controller                      Normal          CREATE  Ingress hosting/hosting-yg5ohbhu-custom
  18m           18m             1       ingress-controller                      Normal          CREATE  Ingress hosting/hosting-yg5ohbhu-custom
  17m           17m             1       ingress-controller                      Normal          CREATE  Ingress hosting/hosting-yg5ohbhu-custom
  17m           17m             1       ingress-controller                      Normal          CREATE  Ingress hosting/hosting-yg5ohbhu-custom

Also because the leaderelection?

Btw, can I ask what is the leader election for? It’s about which nginx-ingress-controller pod updates an ingress?

Thanks for help!

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 5
  • Comments: 24 (15 by maintainers)

Most upvoted comments

As a follow up, 0.9.0-beta.12 and 0.9.0-beta.13 do not have leader election properly working. The last release that does work is 0.9.0-beta.11

I just noticed that 0.9.0-beta.14 has been released: https://gcr.io/google_containers/nginx-ingress-controller

It does solve this problem for me.

@fcgravalos @metral It is included into the 0.9.0-beta.14 (yet to be released). Based on the release cycle that they seem to follow (more or less 1 and half months per release) and given that the latest one (0.9.0-beta.13) was at 8-Sept we might have to wait a little (or not, who knows).

@aledbf thanks, it’s works!

@stibi @mikesplain @AssafShaikevich @brenix please test quay.io/aledbf/nginx-ingress-controller:0.217

Any news about the 0.9.0-beta.xx that fixes leader election? I don’t want to go back to 0.9.0-beta.11 since it has other problems too

Thanks @aledbf, worked for us too.

+1 - seeing this as well (same version / daemonset)

Btw, can I ask what is the leader election for? It’s about which nginx-ingress-controller pod updates an ingress?

The election is to allow that only one pod running the ingress controller keep the status field in the ingress updated (the IP address or FQDN of the LB)