ingress-nginx: Frequent backend reloads causing disruption

We have a kubernetes cluster with 3 pods of NGINX ingress controller receiving incoming HTTP traffic into services. Our applications creates new ingress rules as new services get deployed and deletes rules when services are deployed. This happens very frequently throughout the day.

The frequent change in ingress rules list causes the ingress controller to reload configuration. The issue is that all the 3 pods pretty much reload the configuration at about the same time and this disturbs the HTTP traffic coming into the services.

Example logs from the 3 pods:

ingress-controller-64648f4c78-psvbj
I0107 05:53:01.661475       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:08.073274       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:08.173880       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:14.287824       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:21.320332       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:27.280998       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:27.374023       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:33.498625       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:38.579716       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:44.803349       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:44.880559       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:51.072775       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:57.979804       6 controller.go:141] Configuration changes detected, backend reload required.
 
ingress-controller-64648f4c78-svscm
I0107 05:53:01.661250       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:07.896207       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:07.987639       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:14.191442       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:21.320347       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:27.383899       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:27.476242       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:33.565391       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:38.579606       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:44.606052       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:44.683937       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:50.699345       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:57.979718       6 controller.go:141] Configuration changes detected, backend reload required.
 
ingress-controller-64648f4c78-qlw9s
I0107 05:53:01.661575       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:07.900405       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:08.091107       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:14.188193       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:21.320520       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:27.216136       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:27.374890       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:33.714696       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:38.579095       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:44.504924       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:44.690168       6 controller.go:141] Configuration changes detected, backend reload required.
I0107 05:53:50.990380       6 controller.go:157] Backend successfully reloaded.
I0107 05:53:57.980288       6 controller.go:141] Configuration changes detected, backend reload required.
 

The typical reload lasts about 6-8 seconds but the three pods are almost synchronized when they reload. This results in some down time (usually small but significant to our SLA).

Would the number of ‘worker processes’ or ‘worker-shutdown-timeout’ affect downtime or unavailability of ingress? Most of the web traffic we receive is fairly short transaction time (less than 60s).

Are there any configurations we can do to minimize this downtime or stagger the pods from reloading? It does appear that NGINX appears to queue the requests for a bit during the reload, is there an option to increase this time to account for potential reload interval?

Any pointers will help.

Many thanks!

/triage support