apisix-ingress-controller: bug: ingress-controller doesn't recover from failed sync
Current Behavior
occasionally, our controller will stop doing anything after failing to sync, but it also doesn’t exit
we have logs for this occurring immediately following an upgrade, but we’ve seen similar behavior elsewhere too
Expected Behavior
The ingress controller should either retry running its logic or exit.
Error Logs
2023-09-22T17:40:34+08:00 info ingress/ingress.go:145 start ingress controller
2023-09-22T17:40:34+08:00 info providers/controller.go:155 start api server
2023-09-22T17:40:34+08:00 info providers/controller.go:170 start leader election
I0922 17:40:34.168741 1 leaderelection.go:250] attempting to acquire leader lease apisix/ingress-apisix-internal-leader...
2023-09-22T17:40:34+08:00 warn providers/controller.go:220 found a new leader apisix-internal-ingress-controller-54d9c8d775-9qwkf
2023-09-22T17:40:34+08:00 info providers/controller.go:222 controller now is running as a candidate {"namespace": "apisix", "pod": "apisix-internal-ingress-controller-5948cfdc95-hq8cm"}
2023-09-22T17:40:34+08:00 info providers/controller.go:387 controller tries to leading ... {"namespace": "apisix", "pod": "apisix-internal-ingress-controller-5948cfdc95-hq8cm"}
2023-09-22T17:40:34+08:00 warn apisix/cluster.go:423 waiting cluster default to ready, it may takes a while
2023-09-22T17:40:34+08:00 info apisix/cluster.go:248 syncing cache {"cluster": "default"}
2023-09-22T17:40:34+08:00 info apisix/cluster.go:463 syncing schema {"cluster": "default"}
2023-09-22T17:40:34+08:00 error apisix/route.go:90 failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}
2023-09-22T17:40:34+08:00 error apisix/cluster.go:298 failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}
2023-09-22T17:40:36+08:00 error apisix/route.go:90 failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}
2023-09-22T17:40:36+08:00 error apisix/cluster.go:298 failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}
2023-09-22T17:40:38+08:00 error apisix/cluster.go:298 failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}
2023-09-22T17:40:40+08:00 error apisix/route.go:90 failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}
2023-09-22T17:40:40+08:00 error apisix/cluster.go:298 failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}
2023-09-22T17:40:42+08:00 error apisix/route.go:90 failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}
2023-09-22T17:40:42+08:00 error apisix/cluster.go:298 failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}
2023-09-22T17:40:42+08:00 error apisix/cluster.go:258 failed to sync cache {"cost_time": "8.011791225s", "cluster": "default"}
2023-09-22T17:40:42+08:00 error providers/controller.go:419 failed to wait the default cluster to be ready: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}
2023-09-22T17:40:42+08:00 info apisix/cluster.go:248 syncing cache {"cluster": "default"}
2023-09-22T17:40:42+08:00 info apisix/cluster.go:463 syncing schema {"cluster": "default"}
2023-09-22T17:40:42+08:00 error apisix/route.go:90 failed to list routes: Get "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/routes": context canceled
2023-09-22T17:40:42+08:00 error apisix/plugin.go:46 failed to list plugins' names: Get "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": context canceled
2023-09-22T17:40:42+08:00 error apisix/cluster.go:298 failed to list routes in APISIX: Get "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/routes": context canceled
2023-09-22T17:40:42+08:00 error apisix/cluster.go:483 failed to list plugin names in APISIX: Get "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": context canceled
2023-09-22T17:40:42+08:00 error apisix/cluster.go:446 failed to sync schema: Get "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": context canceled
2023-09-22T17:40:42+08:00 error apisix/cluster.go:258 failed to sync cache {"cost_time": "99.599µs", "cluster": "default"}
Do note that i pulled this log on 2023-09-26 - the ingress has just not logged anything in days.
Steps to Reproduce
Not fully known, but etcd/apisix being unavailable at critical moments seems to trigger it.
Environment
- APISIX Ingress controller version (run
apisix-ingress-controller version --long)apache/apisix-ingress-controller:1.7.0 - Kubernetes cluster version (run
kubectl version)v1.26.6 - OS version if running APISIX Ingress controller in a bare-metal environment (run
uname -a)N/A
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 8
- Comments: 27 (15 by maintainers)
This is a criticial issue and checking if the TCP port is available is absolutely not sufficient, @tao12345666333. We had yet another ingress controller that simply got stuck. Frankly, the entire leader election code seems broken (e.g reports as “running as leader” in the callback, where nothing else happens, starts controller loop once any controller is elected), and I’ll look into getting a PR in for this as soon as possible (after #2139), this is unfortunately making us look pretty bad as an infra team for adopting the ingress.
Also running into this issue here.
That is to be expected, since this happened during a helm upgrade and APISIX pods were being recreated at the same time.
Either way, the controller getting stuck doing nothing is the unexpected behavior. The controller simply can not assume that the APISIX API will be reachable at startup. I’d suggest a failed sync should cause the ingress controller to exit (or retry, if refactored to, it seems to cancel an in-flight retry attempt via the defer cancel), it is not a good signal for giving up on leader election.