apisix-ingress-controller: bug: ingress-controller doesn't recover from failed sync

Current Behavior

occasionally, our controller will stop doing anything after failing to sync, but it also doesn’t exit

we have logs for this occurring immediately following an upgrade, but we’ve seen similar behavior elsewhere too

Expected Behavior

The ingress controller should either retry running its logic or exit.

Error Logs

2023-09-22T17:40:34+08:00	info	ingress/ingress.go:145	start ingress controller
2023-09-22T17:40:34+08:00	info	providers/controller.go:155	start api server
2023-09-22T17:40:34+08:00	info	providers/controller.go:170	start leader election
I0922 17:40:34.168741       1 leaderelection.go:250] attempting to acquire leader lease apisix/ingress-apisix-internal-leader...
2023-09-22T17:40:34+08:00	warn	providers/controller.go:220	found a new leader apisix-internal-ingress-controller-54d9c8d775-9qwkf
2023-09-22T17:40:34+08:00	info	providers/controller.go:222	controller now is running as a candidate	{"namespace": "apisix", "pod": "apisix-internal-ingress-controller-5948cfdc95-hq8cm"}
2023-09-22T17:40:34+08:00	info	providers/controller.go:387	controller tries to leading ...	{"namespace": "apisix", "pod": "apisix-internal-ingress-controller-5948cfdc95-hq8cm"}
2023-09-22T17:40:34+08:00	warn	apisix/cluster.go:423	waiting cluster default to ready, it may takes a while
2023-09-22T17:40:34+08:00	info	apisix/cluster.go:248	syncing cache	{"cluster": "default"}
2023-09-22T17:40:34+08:00	info	apisix/cluster.go:463	syncing schema	{"cluster": "default"}
2023-09-22T17:40:34+08:00	error	apisix/route.go:90	failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2023-09-22T17:40:34+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2023-09-22T17:40:36+08:00	error	apisix/route.go:90	failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2023-09-22T17:40:36+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2023-09-22T17:40:38+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2023-09-22T17:40:40+08:00	error	apisix/route.go:90	failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2023-09-22T17:40:40+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2023-09-22T17:40:42+08:00	error	apisix/route.go:90	failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2023-09-22T17:40:42+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2023-09-22T17:40:42+08:00	error	apisix/cluster.go:258	failed to sync cache	{"cost_time": "8.011791225s", "cluster": "default"}
2023-09-22T17:40:42+08:00	error	providers/controller.go:419	failed to wait the default cluster to be ready: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2023-09-22T17:40:42+08:00	info	apisix/cluster.go:248	syncing cache	{"cluster": "default"}
2023-09-22T17:40:42+08:00	info	apisix/cluster.go:463	syncing schema	{"cluster": "default"}
2023-09-22T17:40:42+08:00	error	apisix/route.go:90	failed to list routes: Get "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/routes": context canceled
2023-09-22T17:40:42+08:00	error	apisix/plugin.go:46	failed to list plugins' names: Get "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": context canceled
2023-09-22T17:40:42+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: Get "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/routes": context canceled
2023-09-22T17:40:42+08:00	error	apisix/cluster.go:483	failed to list plugin names in APISIX: Get "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": context canceled
2023-09-22T17:40:42+08:00	error	apisix/cluster.go:446	failed to sync schema: Get "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": context canceled
2023-09-22T17:40:42+08:00	error	apisix/cluster.go:258	failed to sync cache	{"cost_time": "99.599µs", "cluster": "default"}

Do note that i pulled this log on 2023-09-26 - the ingress has just not logged anything in days.

Steps to Reproduce

Not fully known, but etcd/apisix being unavailable at critical moments seems to trigger it.

Environment

  • APISIX Ingress controller version (run apisix-ingress-controller version --long) apache/apisix-ingress-controller:1.7.0
  • Kubernetes cluster version (run kubectl version) v1.26.6
  • OS version if running APISIX Ingress controller in a bare-metal environment (run uname -a) N/A

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 8
  • Comments: 27 (15 by maintainers)

Most upvoted comments

This is a criticial issue and checking if the TCP port is available is absolutely not sufficient, @tao12345666333. We had yet another ingress controller that simply got stuck. Frankly, the entire leader election code seems broken (e.g reports as “running as leader” in the callback, where nothing else happens, starts controller loop once any controller is elected), and I’ll look into getting a PR in for this as soon as possible (after #2139), this is unfortunately making us look pretty bad as an infra team for adopting the ingress.

2024-01-19T21:29:00+08:00	info	ingress/ingress.go:121	init apisix ingress controller
2024-01-19T21:29:00+08:00	info	ingress/ingress.go:123	version:
Version: 1.7.0
Git SHA: no-git-module
Go Version: go1.20.8
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64

2024-01-19T21:29:00+08:00	info	ingress/ingress.go:133	use configuration
{
  "cert_file": "/etc/webhook/certs/cert.pem",
  "key_file": "/etc/webhook/certs/key.pem",
  "log_level": "info",
  "log_output": "stderr",
  "log_rotate_output_path": "",
  "log_rotation_max_size": 100,
  "log_rotation_max_age": 0,
  "log_rotation_max_backups": 0,
  "http_listen": ":8080",
  "https_listen": ":8443",
  "ingress_publish_service": "apisix/apisix-internal-gateway",
  "ingress_status_address": [],
  "enable_profiling": true,
  "kubernetes": {
    "kubeconfig": "",
    "resync_interval": "6h0m0s",
    "namespace_selector": [],
    "election_id": "ingress-apisix-internal-leader",
    "ingress_class": "apisix-internal",
    "ingress_version": "networking/v1",
    "watch_endpoint_slices": false,
    "api_version": "apisix.apache.org/v2",
    "enable_gateway_api": false,
    "disable_status_updates": false,
    "enable_admission": false
  },
  "apisix": {
    "admin_api_version": "v3",
    "default_cluster_name": "default",
    "default_cluster_base_url": "http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin",
    "default_cluster_admin_key": "******"
  },
  "apisix_resource_sync_interval": "1h0m0s",
  "apisix_resource_sync_comparison": true,
  "plugin_metadata_cm": "",
  "etcdserver": {
    "enabled": false,
    "prefix": "/apisix",
    "listen_address": ":12379",
    "ssl_key_encrypt_salt": "edd1c9f0985e76a2"
  }
}
2024-01-19T21:29:00+08:00	info	ingress/ingress.go:145	start ingress controller
2024-01-19T21:29:00+08:00	info	providers/controller.go:170	start leader election
2024-01-19T21:29:00+08:00	info	providers/controller.go:155	start api server
I0119 21:29:00.286860       1 leaderelection.go:250] attempting to acquire leader lease apisix/ingress-apisix-internal-leader...
2024-01-19T21:29:00+08:00	warn	providers/controller.go:220	found a new leader apisix-internal-ingress-controller-578bf88584-jvs59
2024-01-19T21:29:00+08:00	info	providers/controller.go:222	controller now is running as a candidate	{"namespace": "apisix", "pod": "apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
2024-01-19T21:29:00+08:00	info	providers/controller.go:387	controller tries to leading ...	{"namespace": "apisix", "pod": "apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
2024-01-19T21:29:00+08:00	info	apisix/stream_route.go:38	resource stream_routes is disabled
2024-01-19T21:29:00+08:00	info	apisix/cluster.go:248	syncing cache	{"cluster": "default"}
2024-01-19T21:29:00+08:00	warn	apisix/cluster.go:423	waiting cluster default to ready, it may takes a while
2024-01-19T21:29:00+08:00	info	apisix/cluster.go:463	syncing schema	{"cluster": "default"}
2024-01-19T21:29:00+08:00	error	apisix/route.go:90	failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2024-01-19T21:29:00+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2024-01-19T21:29:02+08:00	error	apisix/route.go:90	failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2024-01-19T21:29:02+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2024-01-19T21:29:03+08:00	info	providers/controller.go:141	LeaderElection	{"message": "apisix-internal-ingress-controller-9d96f45cb-4kg8l became leader", "event_type": "Normal"}
I0119 21:29:03.113259       1 leaderelection.go:260] successfully acquired lease apisix/ingress-apisix-internal-leader
2024-01-19T21:29:03+08:00	warn	providers/controller.go:220	found a new leader apisix-internal-ingress-controller-9d96f45cb-4kg8l
2024-01-19T21:29:03+08:00	info	providers/controller.go:212	controller now is running as leader	{"namespace": "apisix", "pod": "apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
2024-01-19T21:29:04+08:00	error	apisix/route.go:90	failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2024-01-19T21:29:04+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2024-01-19T21:29:06+08:00	error	apisix/route.go:90	failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2024-01-19T21:29:06+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2024-01-19T21:29:08+08:00	error	apisix/route.go:90	failed to list routes: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2024-01-19T21:29:08+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2024-01-19T21:29:08+08:00	error	apisix/cluster.go:258	failed to sync cache	{"cost_time": "8.016969049s", "cluster": "default"}
2024-01-19T21:29:08+08:00	error	providers/controller.go:419	failed to wait the default cluster to be ready: unexpected status code 503; error message: {"error_msg":"invalid response code: 502"}

2024-01-19T21:29:08+08:00	info	apisix/stream_route.go:38	resource stream_routes is disabled
2024-01-19T21:29:08+08:00	info	apisix/cluster.go:248	syncing cache	{"cluster": "default"}
2024-01-19T21:29:08+08:00	info	apisix/cluster.go:463	syncing schema	{"cluster": "default"}
2024-01-19T21:29:08+08:00	error	apisix/route.go:90	failed to list routes: Get "http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin/routes": context canceled
2024-01-19T21:29:08+08:00	error	apisix/cluster.go:298	failed to list routes in APISIX: Get "http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin/routes": context canceled
2024-01-19T21:29:08+08:00	error	apisix/cluster.go:258	failed to sync cache	{"cost_time": "60.702µs", "cluster": "default"}
2024-01-19T21:29:08+08:00	error	apisix/plugin.go:46	failed to list plugins' names: Get "http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": context canceled
2024-01-19T21:29:08+08:00	error	apisix/cluster.go:483	failed to list plugin names in APISIX: Get "http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": context canceled
2024-01-19T21:29:08+08:00	error	apisix/cluster.go:446	failed to sync schema: Get "http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true": context canceled
E0122 23:07:05.784720       1 leaderelection.go:369] Failed to update lock: Put "https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/apisix/leases/ingress-apisix-internal-leader": context deadline exceeded
I0122 23:07:05.784783       1 leaderelection.go:285] failed to renew lease apisix/ingress-apisix-internal-leader: timed out waiting for the condition
2024-01-22T23:07:09+08:00	info	providers/controller.go:141	LeaderElection	{"message": "apisix-internal-ingress-controller-9d96f45cb-4kg8l stopped leading", "event_type": "Normal"}
2024-01-22T23:07:09+08:00	info	providers/controller.go:231	controller now is running as a candidate	{"namespace": "apisix", "pod": "apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
I0122 23:07:09.152585       1 leaderelection.go:250] attempting to acquire leader lease apisix/ingress-apisix-internal-leader...
2024-01-22T23:07:09+08:00	info	providers/controller.go:141	LeaderElection	{"message": "apisix-internal-ingress-controller-9d96f45cb-4kg8l became leader", "event_type": "Normal"}
I0122 23:07:09.984780       1 leaderelection.go:260] successfully acquired lease apisix/ingress-apisix-internal-leader
2024-01-22T23:07:09+08:00	info	providers/controller.go:212	controller now is running as leader	{"namespace": "apisix", "pod": "apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
E0124 13:07:40.830633       1 leaderelection.go:369] Failed to update lock: Put "https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/apisix/leases/ingress-apisix-internal-leader": context deadline exceeded
I0124 13:07:40.830667       1 leaderelection.go:285] failed to renew lease apisix/ingress-apisix-internal-leader: timed out waiting for the condition
2024-01-24T13:07:40+08:00	info	providers/controller.go:141	LeaderElection	{"message": "apisix-internal-ingress-controller-9d96f45cb-4kg8l stopped leading", "event_type": "Normal"}
2024-01-24T13:07:40+08:00	info	providers/controller.go:231	controller now is running as a candidate	{"namespace": "apisix", "pod": "apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
I0124 13:07:40.843705       1 leaderelection.go:250] attempting to acquire leader lease apisix/ingress-apisix-internal-leader...
2024-01-24T13:07:40+08:00	info	providers/controller.go:141	LeaderElection	{"message": "apisix-internal-ingress-controller-9d96f45cb-4kg8l became leader", "event_type": "Normal"}
I0124 13:07:40.864098       1 leaderelection.go:260] successfully acquired lease apisix/ingress-apisix-internal-leader
2024-01-24T13:07:40+08:00	info	providers/controller.go:212	controller now is running as leader	{"namespace": "apisix", "pod": "apisix-internal-ingress-controller-9d96f45cb-4kg8l"}

Also running into this issue here.

That is to be expected, since this happened during a helm upgrade and APISIX pods were being recreated at the same time.

Either way, the controller getting stuck doing nothing is the unexpected behavior. The controller simply can not assume that the APISIX API will be reachable at startup. I’d suggest a failed sync should cause the ingress controller to exit (or retry, if refactored to, it seems to cancel an in-flight retry attempt via the defer cancel), it is not a good signal for giving up on leader election.