ingress-nginx: leaderelection: Failed to update lock
Hello, I’m testing the 0.9.0-beta.12 release and I see this error in log:
[gaudy-flee-nginx-ingress-controller-vrqxl] E0904 09:53:44.017545 5 leaderelection.go:258] Failed to update lock: Operation cannot be fulfilled on configmaps "ingress-controller-leader-nginx": the object has been modified; please apply your changes to the latest version and try again
The nginx-ingress controller is deployed as a DaemonSet and it’s the only ingress controller on the cluster.
Here is content of ingress-controller-leader-nginx
ConfigMap:
$ kubectl --namespace internal describe configmap ingress-controller-leader-nginx
Name: ingress-controller-leader-nginx
Namespace: internal
Labels: <none>
Annotations: control-plane.alpha.kubernetes.io/leader={"holderIdentity":"ingress-controller-leader-nginx","leaseDurationSeconds":30,"acquireTime":"2017-09-04T09:10:13Z","renewTime":"2017-09-04T09:56:28Z","leaderTr...
Data
====
Events: <none>
Another suspicious observation is that each pod is elected itself as the leader:
I0904 09:40:22.489304 5 leaderelection.go:174] attempting to acquire leader lease... I0904 09:40:22.489800 5 status.go:152] skipping Ingress status update (I am not the current leader)
I0904 09:40:22.496448 5 template.go:92] adjusting template buffer size from 65535 to 262484
I0904 09:40:22.506258 5 leaderelection.go:184] successfully acquired lease internal/ingress-controller-leader-nginx I0904 09:40:22.506374 5 status.go:171] new leader elected (gaudy-flee-nginx-ingress-controller-vrqxl)
I0904 09:40:22.506397 5 status.go:173] I am the new status update leader
This is logged by each of the pods.
Is that because the leader election doesn’t work correctly?
Another issue: too many update events, similar to #1289:
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h 49m 80 ingress-controller Normal UPDATE Ingress hosting/hosting-yg5ohbhu-custom
1h 49m 80 ingress-controller Normal UPDATE Ingress hosting/hosting-yg5ohbhu-custom
1h 49m 80 ingress-controller Normal UPDATE Ingress hosting/hosting-yg5ohbhu-custom
1h 49m 53 ingress-controller Normal UPDATE Ingress hosting/hosting-yg5ohbhu-custom
1h 49m 81 ingress-controller Normal UPDATE Ingress hosting/hosting-yg5ohbhu-custom
48m 48m 1 ingress-controller Normal CREATE Ingress hosting/hosting-yg5ohbhu-custom
48m 48m 1 ingress-controller Normal CREATE Ingress hosting/hosting-yg5ohbhu-custom
48m 48m 1 ingress-controller Normal CREATE Ingress hosting/hosting-yg5ohbhu-custom
48m 48m 1 ingress-controller Normal CREATE Ingress hosting/hosting-yg5ohbhu-custom
48m 48m 1 ingress-controller Normal CREATE Ingress hosting/hosting-yg5ohbhu-custom
47m 47m 1 ingress-controller Normal UPDATE Ingress hosting/hosting-yg5ohbhu-custom
47m 47m 1 ingress-controller Normal UPDATE Ingress hosting/hosting-yg5ohbhu-custom
47m 47m 1 ingress-controller Normal UPDATE Ingress hosting/hosting-yg5ohbhu-custom
47m 47m 1 ingress-controller Normal UPDATE Ingress hosting/hosting-yg5ohbhu-custom
47m 47m 1 ingress-controller Normal UPDATE Ingress hosting/hosting-yg5ohbhu-custom
18m 18m 1 ingress-controller Normal CREATE Ingress hosting/hosting-yg5ohbhu-custom
18m 18m 1 ingress-controller Normal CREATE Ingress hosting/hosting-yg5ohbhu-custom
18m 18m 1 ingress-controller Normal CREATE Ingress hosting/hosting-yg5ohbhu-custom
17m 17m 1 ingress-controller Normal CREATE Ingress hosting/hosting-yg5ohbhu-custom
17m 17m 1 ingress-controller Normal CREATE Ingress hosting/hosting-yg5ohbhu-custom
Also because the leaderelection?
Btw, can I ask what is the leader election for? It’s about which nginx-ingress-controller pod updates an ingress?
Thanks for help!
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 5
- Comments: 24 (15 by maintainers)
As a follow up,
0.9.0-beta.12
and0.9.0-beta.13
do not have leader election properly working. The last release that does work is0.9.0-beta.11
I just noticed that 0.9.0-beta.14 has been released: https://gcr.io/google_containers/nginx-ingress-controller
It does solve this problem for me.
@fcgravalos @metral It is included into the 0.9.0-beta.14 (yet to be released). Based on the release cycle that they seem to follow (more or less 1 and half months per release) and given that the latest one (0.9.0-beta.13) was at 8-Sept we might have to wait a little (or not, who knows).
@aledbf thanks, it’s works!
@stibi @mikesplain @AssafShaikevich @brenix please test
quay.io/aledbf/nginx-ingress-controller:0.217
Any news about the 0.9.0-beta.xx that fixes leader election? I don’t want to go back to 0.9.0-beta.11 since it has other problems too
Thanks @aledbf, worked for us too.
+1 - seeing this as well (same version / daemonset)
The election is to allow that only one pod running the ingress controller keep the status field in the ingress updated (the IP address or FQDN of the LB)