rancher: Rancher logs spammed with errors relating to tokens of deleted clusters

Rancher Server Setup

Rancher version:2.6.3

Describe the bug

The Rancher logs are spammed with hundreds of error events referencing clusters that have long been deleted. Apparently this is related to some Rancher resources (e.g. clusterRegistrationToken) not being cleaned up as part of cluster deletion routine.

error syncing 'harvey': handler machine-worker-label: [machines.cluster.x-k8s.io](http://machines.cluster.x-k8s.io/) "custom-453b4de06b71" not found, requeuing                                                                                     │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-2zqdg2fz/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-2zqdg2fz" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-srn5dqg2/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-srn5dqg2" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-rnpgwx8w/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-rnpgwx8w" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-s5f444bf/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-s5f444bf" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-gznflpgk/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-gznflpgk" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-nspssk27/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-nspssk27" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-2z72n8dw/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-2z72n8dw" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-krspd9ff/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-krspd9ff" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-pnk5gs7w/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-pnk5gs7w" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-k9v8kr47/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-k9v8kr47" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-zj6czbpx/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-zj6czbpx" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-w6dccvm5/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-w6dccvm5" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-sqq8xsbk/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-sqq8xsbk" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-9jrgc6m9/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-9jrgc6m9" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-j75f64xr/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-j75f64xr" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-wl6wggdx/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-wl6wggdx" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-vmq5ggw7/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-vmq5ggw7" not found, requeuing                                                              │
│ 2022/02/16 09:48:31 [ERROR] error syncing 'c-m-cf7n8w45/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-cf7n8w45" not found, requeuing                                                              │
│ 2022/02/16 09:48:33 [ERROR] error syncing 'c-m-fw2gg458/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-fw2gg458" not found, requeuing                                                              │
│ 2022/02/16 09:48:33 [ERROR] error syncing 'c-m-bw4zcz24/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-bw4zcz24" not found, requeuing                                                              │
│ 2022/02/16 09:48:33 [ERROR] error syncing 'c-m-b6cbptch/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-b6cbptch" not found, requeuing                                                              │
│ 2022/02/16 09:48:33 [ERROR] error syncing 'c-m-8s9qzmzz/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-8s9qzmzz" not found, requeuing                                                              │
│ 2022/02/16 09:48:33 [ERROR] error syncing 'c-m-dxbvmt2w/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-dxbvmt2w" not found, requeuing                                                              │
│ 2022/02/16 09:48:33 [ERROR] Failed to handling tunnel request from 192.168.200.113:39714: response 400: cluster not found                                                                                                                  │
│ 2022/02/16 09:48:34 [ERROR] error syncing 'c-m-6g5h62hr/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-6g5h62hr" not found, requeuing                                                              │
│ 2022/02/16 09:48:34 [ERROR] error syncing 'c-m-zfb7gdhm/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-zfb7gdhm" not found, requeuing                                                              │
│ 2022/02/16 09:48:34 [ERROR] error syncing 'c-m-npqt8pb2/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-npqt8pb2" not found, requeuing                                                              │
│ 2022/02/16 09:48:34 [ERROR] error syncing 'c-m-mlz4x68c/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-mlz4x68c" not found, requeuing                                                              │
│ 2022/02/16 09:48:34 [ERROR] error syncing 'c-m-xk2px568/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-xk2px568" not found, requeuing                                                              │
│ 2022/02/16 09:48:34 [ERROR] error syncing 'c-m-rc2tht7m/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-rc2tht7m" not found, requeuing                                                              │
│ 2022/02/16 09:48:34 [ERROR] error syncing 'c-m-fkdfnwdl/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-fkdfnwdl" not found, requeuing                                                              │
│ 2022/02/16 09:48:34 [ERROR] error syncing 'c-m-jh26sf77/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-jh26sf77" not found, requeuing                                                              │
│ 2022/02/16 09:48:38 [ERROR] error syncing 'c-m-nnxx4dsx/default-token': handler cluster-registration-token: [clusters.management.cattle.io](http://clusters.management.cattle.io/) "c-m-nnxx4dsx" not found, requeuing                                                              │
│ 2022/02/16 09:48:43 [ERROR] Failed to handling tunnel request from 10.42.0.0:43162: response 400: cluster not found                                                                                                                        │
│ 2022/02/16 09:48:53 [ERROR] Failed to handling tunnel request from 10.42.0.0:43320: response 400: cluster not found                                                                                                                        │
│ 2022/02/16 09:49:03 [ERROR] Failed to handling tunnel request from 10.42.0.0:42676: response 400: cluster not found                                                                                                                        │
│ 2022/02/16 09:49:13 [ERROR] error syncing 'donna': handler machine-worker-label: [machines.cluster.x-k8s.io](http://machines.cluster.x-k8s.io/) "custom-cb1f97064c25" not found, requeuing                                                                                      │
│ 2022/02/16 09:49:13 [ERROR] Failed to handling tunnel request from 10.42.0.0:43546: response 400: cluster not found                                                                                                                        │
│ 2022/02/16 09:49:18 [ERROR] error syncing 'mike': handler machine-worker-label: [machines.cluster.x-k8s.io](http://machines.cluster.x-k8s.io/) "custom-922b012e146b" not found, requeuing                                                                                       │
│ 2022/02/16 09:49:18 [ERROR] error syncing 'donna': handler machine-worker-label: [machines.cluster.x-k8s.io](http://machines.cluster.x-k8s.io/) "custom-cb1f97064c25" not found, requeuing

To Reproduce

Create a couple of node driver RKE2 clusters and delete them.

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 7
Comments: 21 (2 by maintainers)

Most upvoted comments

Workaroud for these remaining errors:

2022/09/17 12:55:55 [ERROR] error syncing 'c-m-tgzdjmqc/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-tgzdjmqc" not found, requeuing
2022/09/17 12:55:55 [ERROR] error syncing 'c-m-v5bkkkmm/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-v5bkkkmm" not found, requeuing
2022/09/17 12:55:55 [ERROR] error syncing 'c-m-rl4xbzsk/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-rl4xbzsk" not found, requeuing

For each error line, delete the respective namespace: kubectl delete namespace c-m-tgzdjmqc

mfbrj on Sep 17, 2022

Workaround for these errors:

2022/09/17 12:23:12 [ERROR] error syncing 'c-m-rl4xbzsk/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-rl4xbzsk" not found, requeuing
2022/09/17 12:23:12 [ERROR] error syncing 'c-m-p8s2rmjk/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-p8s2rmjk" not found, requeuing
2022/09/17 12:23:12 [ERROR] error syncing 'c-m-tgzdjmqc/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-tgzdjmqc" not found, requeuing
2022/09/17 12:23:12 [ERROR] error syncing 'c-m-pdpztlk9/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-pdpztlk9" not found, requeuing
2022/09/17 12:23:12 [ERROR] error syncing 'c-m-v5bkkkmm/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-v5bkkkmm" not found, requeuing

List all the clusterregistrationtokens.management.cattle.io resources: kubectl get clusterregistrationtokens.management.cattle.io --all-namespaces

NAMESPACE      NAME            AGE
local          default-token   36d
c-m-pdpztlk9   default-token   14d
c-m-p8s2rmjk   default-token   14d
c-m-rl4xbzsk   default-token   14d
c-m-tgzdjmqc   default-token   14d
c-m-v5bkkkmm   default-token   14d
c-m-srjzg7gc   default-token   41h
c-m-srjzg7gc   system          41h

Delete all older tokens (in my case the tokens with 14d age), except local and newer tokens: kubectl -n c-m-v5bkkkmm delete clusterregistrationtokens.management.cattle.io default-token

These errors remains on each 10 seconds:

2022/09/17 12:55:55 [ERROR] error syncing 'c-m-tgzdjmqc/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-tgzdjmqc" not found, requeuing
2022/09/17 12:55:55 [ERROR] error syncing 'c-m-v5bkkkmm/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-v5bkkkmm" not found, requeuing
2022/09/17 12:55:55 [ERROR] error syncing 'c-m-rl4xbzsk/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-rl4xbzsk" not found, requeuing

mfbrj on Sep 17, 2022

I found a workaroud for the errors like:

2022/09/17 11:37:15 [ERROR] error syncing 'p-qh8sp/creator-project-owner': handler mgmt-auth-prtb-controller: clusters.management.cattle.io "c-m-p8s2rmjk" not found, requeuing

List all projects kubectl get projects.management.cattle.io --all-namespaces

NAMESPACE      NAME      AGE
local          p-q9xp7   36d
local          p-vn299   36d
c-m-rl4xbzsk   p-ncls5   14d
c-m-mht2k4w7   p-kc6nl   14d
c-m-v5bkkkmm   p-8zg4d   14d
c-m-pdpztlk9   p-4vbgh   14d
c-m-pdpztlk9   p-rs52c   14d
c-m-v5bkkkmm   p-gxtkx   14d
c-m-rl4xbzsk   p-h5lhk   14d
c-m-p8s2rmjk   p-qh8sp   14d
c-m-tgzdjmqc   p-9b954   14d
c-m-tgzdjmqc   p-9htch   14d
c-m-p8s2rmjk   p-s5wv9   14d
c-m-srjzg7gc   p-rgbq9   41h
c-m-srjzg7gc   p-x7s5p   41h
c-m-srjzg7gc   p-dwc2d   40h
c-m-srjzg7gc   p-qfvjz   39h
c-m-srjzg7gc   p-nzrmg   24h

For all “dead” projects (in my case the projects with 14d age), delete each project, except local and newer projects: kubectl -n c-m-rl4xbzsk delete projects.management.cattle.io p-ncls5

These errors remains:

2022/09/17 12:23:12 [ERROR] error syncing 'c-m-rl4xbzsk/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-rl4xbzsk" not found, requeuing
2022/09/17 12:23:12 [ERROR] error syncing 'c-m-p8s2rmjk/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-p8s2rmjk" not found, requeuing
2022/09/17 12:23:12 [ERROR] error syncing 'c-m-tgzdjmqc/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-tgzdjmqc" not found, requeuing
2022/09/17 12:23:12 [ERROR] error syncing 'c-m-pdpztlk9/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-pdpztlk9" not found, requeuing
2022/09/17 12:23:12 [ERROR] error syncing 'c-m-v5bkkkmm/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-m-v5bkkkmm" not found, requeuing
2022/09/17 12:23:15 [ERROR] error syncing 'c-m-tgzdjmqc/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-tgzdjmqc" not found, requeuing
2022/09/17 12:23:15 [ERROR] error syncing 'c-m-v5bkkkmm/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-v5bkkkmm" not found, requeuing
2022/09/17 12:23:15 [ERROR] error syncing 'c-m-rl4xbzsk/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-rl4xbzsk" not found, requeuing
2022/09/17 12:23:25 [ERROR] error syncing 'c-m-tgzdjmqc/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-tgzdjmqc" not found, requeuing
2022/09/17 12:23:25 [ERROR] error syncing 'c-m-v5bkkkmm/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-v5bkkkmm" not found, requeuing
2022/09/17 12:23:25 [ERROR] error syncing 'c-m-rl4xbzsk/creator-cluster-owner': handler mgmt-auth-crtb-controller: clusters.management.cattle.io "c-m-rl4xbzsk" not found, requeuing

mfbrj on Sep 17, 2022

is anyone able to address these issues?

iuli72an on Sep 17, 2022

We have the same problem - even hundreds of errors per second on this.

Could this seeming mismatch have anything to do with it? There are a lot more clusterregistrations than actual clusters.

k get clusterregistrations -n fleet-default; k get clusters -n fleet-default

Are the clusterregistrations not cleared correctly when a cluster is detached from Rancher?

I also noticed records pointing to old clusters in:

clusterrolebindings
clusterroletemplatebindings
clusteralertgroups
clusteralertrules

Update: Actually, the entire namespace of the cluster (c-XYZ12) is not being deleted while the cluster resources in fleet-default are gone. This makes Rancher think there are more clusters than there are. Rancher tries to contact these clusters every second but fails as the cluster-kubeconfig secrets in namespace fleet-default has also been erased. That results in hundreds of error loglines per second.

eelkonio on Aug 26, 2022

We are seeing this but iiuc, it relates to clusters rather than namespaces:

2022/06/17 10:59:15 [ERROR] error syncing 'c-pq85n/default-token': handler cluster-registration-token: clusters.management.cattle.io "c-pq85n" not found, requeuing

The cluster with this id has been deleted; clearly not all state associated with this cluster has been deleted, but it’s not clear to me where this extraneous state lives…

seanrmurphy on Jun 17, 2022