rancher: standalone rancher container (single node install) crashes and restarts

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible):

  • run rancher:v2.2.8 single install
  • add a cluster, enable cluster monitoring
  • upgrade to v2.3.0-rc3
  • everything works
  • check the rancher container status

Result:

  • the container restarts every ~5 mins
root@jiaqi-228:~# docker inspect zealous_brown | grep RestartCount
        "RestartCount": 4,

Rancher’s log:

2019/09/13 00:07:16 [FATAL] k3s exited with: exit status 1

Detailed log:

E0913 00:07:16.919946      31 server.go:252] lost master
lost lease
2019/09/13 00:07:16 [ERROR] NodeController jiaqi-228-ec1 [node-syncer] failed with : Delete https://localhost:6443/apis/project.cattle.io/v3/namespaces/p-q6qfb/apps/rancher-logging?timeout=30s: EOF
E0913 00:07:16.948969       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Secret: Get https://localhost:6443/api/v1/watch/secrets?resourceVersion=11927&timeout=30m0s&timeoutSeconds=302: read tcp [::1]:35894->[::1]:6443: read: connection reset by peer
E0913 00:07:16.949860       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.PipelineExecution: Get https://localhost:6443/apis/project.cattle.io/v3/watch/pipelineexecutions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=569: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.949988       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterRegistrationToken: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clusterregistrationtokens?resourceVersion=11933&timeout=30m0s&timeoutSeconds=479: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.950106       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.NodeDriver: Get https://localhost:6443/apis/management.cattle.io/v3/watch/nodedrivers?resourceVersion=11933&timeout=30m0s&timeoutSeconds=556: read tcp [::1]:35984->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950164       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Setting: Get https://localhost:6443/apis/management.cattle.io/v3/watch/settings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=386: read tcp [::1]:36004->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950183       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.App: Get https://localhost:6443/apis/project.cattle.io/v3/watch/apps?resourceVersion=11933&timeout=30m0s&timeoutSeconds=448: read tcp [::1]:35982->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950227       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.PipelineSetting: Get https://localhost:6443/apis/project.cattle.io/v3/watch/pipelinesettings?resourceVersion=11940&timeout=30m0s&timeoutSeconds=362: read tcp [::1]:35960->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950286       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.User: Get https://localhost:6443/apis/management.cattle.io/v3/watch/users?resourceVersion=11933&timeout=30m0s&timeoutSeconds=590: read tcp [::1]:35958->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950296       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Token: Get https://localhost:6443/apis/management.cattle.io/v3/watch/tokens?resourceVersion=11933&timeout=30m0s&timeoutSeconds=549: read tcp [::1]:35978->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950337       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.GlobalRoleBinding: Get https://localhost:6443/apis/management.cattle.io/v3/watch/globalrolebindings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=397: read tcp [::1]:35942->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950342       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Node: Get https://localhost:6443/apis/management.cattle.io/v3/watch/nodes?resourceVersion=11933&timeout=30m0s&timeoutSeconds=368: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.950344       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterAlertRule: Get https://localhost:6443/apis/management.cattle.io/v3/watch/namespaces/c-5kk66/clusteralertrules?resourceVersion=11937&timeout=30m0s&timeoutSeconds=354: read tcp [::1]:35976->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950382       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.MultiClusterApp: Get https://localhost:6443/apis/management.cattle.io/v3/watch/multiclusterapps?resourceVersion=11933&timeout=30m0s&timeoutSeconds=588: read tcp [::1]:35954->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950388       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ProjectCatalog: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projectcatalogs?resourceVersion=11933&timeout=30m0s&timeoutSeconds=542: read tcp [::1]:35974->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950428       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterCatalog: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clustercatalogs?resourceVersion=11933&timeout=30m0s&timeoutSeconds=501: read tcp [::1]:35952->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950461       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Setting: Get https://localhost:6443/apis/management.cattle.io/v3/watch/settings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=582: read tcp [::1]:35972->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950471       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.UserAttribute: Get https://localhost:6443/apis/management.cattle.io/v3/watch/userattributes?resourceVersion=11933&timeout=30m0s&timeoutSeconds=391: read tcp [::1]:35950->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950517       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ProjectLogging: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projectloggings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=405: read tcp [::1]:35948->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950519       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Notifier: Get https://localhost:6443/apis/management.cattle.io/v3/watch/namespaces/c-5kk66/notifiers?resourceVersion=11937&timeout=30m0s&timeoutSeconds=585: read tcp [::1]:35970->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950556       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ProjectRoleTemplateBinding: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projectroletemplatebindings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=492: read tcp [::1]:35968->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950560       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Cluster: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clusters?resourceVersion=11969&timeout=30m0s&timeoutSeconds=404: read tcp [::1]:35946->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950594       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ProjectAlertGroup: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projectalertgroups?resourceVersion=11937&timeout=30m0s&timeoutSeconds=368: read tcp [::1]:35944->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950627       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterRoleTemplateBinding: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clusterroletemplatebindings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=584: read tcp [::1]:35966->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950632       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Namespace: Get https://localhost:6443/api/v1/watch/namespaces?resourceVersion=11927&timeout=30m0s&timeoutSeconds=390: read tcp [::1]:35956->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950688       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Pipeline: Get https://localhost:6443/apis/project.cattle.io/v3/watch/pipelines?resourceVersion=11933&timeout=30m0s&timeoutSeconds=316: read tcp [::1]:35940->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950689       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.PodSecurityPolicyTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/podsecuritypolicytemplates?resourceVersion=11935&timeout=30m0s&timeoutSeconds=394: read tcp [::1]:36002->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950706       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ProjectNetworkPolicy: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projectnetworkpolicies?resourceVersion=11937&timeout=30m0s&timeoutSeconds=340: read tcp [::1]:35964->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950737       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.CatalogTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/catalogtemplates?resourceVersion=11933&timeout=30m0s&timeoutSeconds=335: read tcp [::1]:35938->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950750       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Notifier: Get https://localhost:6443/apis/management.cattle.io/v3/watch/notifiers?resourceVersion=11937&timeout=30m0s&timeoutSeconds=542: read tcp [::1]:35962->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950778       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Catalog: Get https://localhost:6443/apis/management.cattle.io/v3/watch/catalogs?resourceVersion=11933&timeout=30m0s&timeoutSeconds=573: read tcp [::1]:35936->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950820       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.GlobalDNS: Get https://localhost:6443/apis/management.cattle.io/v3/watch/globaldnses?resourceVersion=11933&timeout=30m0s&timeoutSeconds=378: read tcp [::1]:35934->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950822       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ProjectAlertRule: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projectalertrules?resourceVersion=11937&timeout=30m0s&timeoutSeconds=550: read tcp [::1]:35926->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950864       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterAlertGroup: Get https://localhost:6443/apis/management.cattle.io/v3/watch/namespaces/c-5kk66/clusteralertgroups?resourceVersion=11937&timeout=30m0s&timeoutSeconds=492: read tcp [::1]:35924->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950866       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.RoleTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/roletemplates?resourceVersion=11933&timeout=30m0s&timeoutSeconds=424: read tcp [::1]:35932->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950887       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.RoleTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/roletemplates?resourceVersion=11933&timeout=30m0s&timeoutSeconds=313: read tcp [::1]:36000->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950904       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Project: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projects?resourceVersion=11933&timeout=30m0s&timeoutSeconds=479: read tcp [::1]:35930->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950951       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Node: Get https://localhost:6443/apis/management.cattle.io/v3/watch/namespaces/c-5kk66/nodes?resourceVersion=11933&timeout=30m0s&timeoutSeconds=473: read tcp [::1]:35928->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950963       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ComposeConfig: Get https://localhost:6443/apis/management.cattle.io/v3/watch/composeconfigs?resourceVersion=11935&timeout=30m0s&timeoutSeconds=435: read tcp [::1]:35998->[::1]:6443: read: connection reset by peer
E0913 00:07:16.950990       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.CatalogTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/namespaces/cattle-global-data/catalogtemplates?resourceVersion=11933&timeout=30m0s&timeoutSeconds=394: read tcp [::1]:35922->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951003       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.PodSecurityPolicyTemplateProjectBinding: Get https://localhost:6443/apis/management.cattle.io/v3/watch/podsecuritypolicytemplateprojectbindings?resourceVersion=11935&timeout=30m0s&timeoutSeconds=378: read tcp [::1]:35916->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951051       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterScan: Get https://localhost:6443/apis/management.cattle.io/v3/watch/namespaces/c-5kk66/clusterscans?resourceVersion=11937&timeout=30m0s&timeoutSeconds=405: read tcp [::1]:35914->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951099       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Secret: Get https://localhost:6443/api/v1/watch/namespaces/cattle-system/secrets?resourceVersion=11927&timeout=30m0s&timeoutSeconds=411: read tcp [::1]:35912->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951122       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.PipelineExecution: Get https://localhost:6443/apis/project.cattle.io/v3/watch/pipelineexecutions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=364: read tcp [::1]:35920->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951157       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterLogging: Get https://localhost:6443/apis/management.cattle.io/v3/watch/namespaces/c-5kk66/clusterloggings?resourceVersion=11937&timeout=30m0s&timeoutSeconds=546: read tcp [::1]:35910->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951179       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Project: Get https://localhost:6443/apis/management.cattle.io/v3/watch/namespaces/c-5kk66/projects?resourceVersion=11933&timeout=30m0s&timeoutSeconds=530: read tcp [::1]:35918->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951205       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.PodSecurityPolicyTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/podsecuritypolicytemplates?resourceVersion=11935&timeout=30m0s&timeoutSeconds=404: read tcp [::1]:35908->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951249       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.CatalogTemplateVersion: Get https://localhost:6443/apis/management.cattle.io/v3/watch/catalogtemplateversions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=537: read tcp [::1]:35902->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951255       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRoleBinding: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterrolebindings?resourceVersion=11927&timeout=30m0s&timeoutSeconds=449: read tcp [::1]:35996->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951285       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.App: Get https://localhost:6443/apis/project.cattle.io/v3/watch/apps?resourceVersion=11933&timeout=30m0s&timeoutSeconds=569: read tcp [::1]:35904->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951383       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.SourceCodeCredential: Get https://localhost:6443/apis/project.cattle.io/v3/watch/sourcecodecredentials?resourceVersion=11933&timeout=30m0s&timeoutSeconds=330: read tcp [::1]:35896->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951389       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterTemplateRevision: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clustertemplaterevisions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=583: read tcp [::1]:35994->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951456       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.KontainerDriver: Get https://localhost:6443/apis/management.cattle.io/v3/watch/kontainerdrivers?resourceVersion=11933&timeout=30m0s&timeoutSeconds=572: read tcp [::1]:35992->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951535       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.UserAttribute: Get https://localhost:6443/apis/management.cattle.io/v3/watch/userattributes?resourceVersion=11933&timeout=30m0s&timeoutSeconds=526: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.951577       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.RoleBinding: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1/watch/rolebindings?resourceVersion=11927&timeout=30m0s&timeoutSeconds=584: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.951612       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.User: Get https://localhost:6443/apis/management.cattle.io/v3/watch/users?resourceVersion=11933&timeout=30m0s&timeoutSeconds=422: read tcp [::1]:35990->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951645       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRole: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterroles?resourceVersion=11927&timeout=30m0s&timeoutSeconds=382: read tcp [::1]:35988->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951684       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/namespaces/cattle-global-data/clustertemplates?resourceVersion=11933&timeout=30m0s&timeoutSeconds=513: read tcp [::1]:35986->[::1]:6443: read: connection reset by peer
E0913 00:07:16.949481       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.DynamicSchema: Get https://localhost:6443/apis/management.cattle.io/v3/watch/dynamicschemas?resourceVersion=11933&timeout=30m0s&timeoutSeconds=377: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.951902       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Node: Get https://localhost:6443/apis/management.cattle.io/v3/watch/nodes?resourceVersion=11933&timeout=30m0s&timeoutSeconds=378: read tcp [::1]:36030->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951950       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.RKEK8sServiceOption: Get https://localhost:6443/apis/management.cattle.io/v3/watch/rkek8sserviceoptions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=447: read tcp [::1]:36028->[::1]:6443: read: connection reset by peer
E0913 00:07:16.951994       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clustertemplates?resourceVersion=11933&timeout=30m0s&timeoutSeconds=301: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952036       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Project: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projects?resourceVersion=11933&timeout=30m0s&timeoutSeconds=516: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952078       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Role: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1/watch/roles?resourceVersion=11927&timeout=30m0s&timeoutSeconds=522: read tcp [::1]:36026->[::1]:6443: read: connection reset by peer
E0913 00:07:16.952131       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ProjectCatalog: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projectcatalogs?resourceVersion=11933&timeout=30m0s&timeoutSeconds=525: read tcp [::1]:36010->[::1]:6443: read: connection reset by peer
E0913 00:07:16.952171       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.PodSecurityPolicyTemplateProjectBinding: Get https://localhost:6443/apis/management.cattle.io/v3/watch/podsecuritypolicytemplateprojectbindings?resourceVersion=11935&timeout=30m0s&timeoutSeconds=444: read tcp [::1]:36022->[::1]:6443: read: connection reset by peer
E0913 00:07:16.952206       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Catalog: Get https://localhost:6443/apis/management.cattle.io/v3/watch/catalogs?resourceVersion=11933&timeout=30m0s&timeoutSeconds=592: read tcp [::1]:36008->[::1]:6443: read: connection reset by peer
E0913 00:07:16.952213       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.CatalogTemplateVersion: Get https://localhost:6443/apis/management.cattle.io/v3/watch/catalogtemplateversions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=473: read tcp [::1]:36020->[::1]:6443: read: connection reset by peer
E0913 00:07:16.952255       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Secret: Get https://localhost:6443/api/v1/watch/namespaces/cattle-system/secrets?resourceVersion=11927&timeout=30m0s&timeoutSeconds=437: read tcp [::1]:36018->[::1]:6443: read: connection reset by peer
E0913 00:07:16.952255       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterTemplateRevision: Get https://localhost:6443/apis/management.cattle.io/v3/watch/namespaces/cattle-global-data/clustertemplaterevisions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=585: read tcp [::1]:36006->[::1]:6443: read: connection reset by peer
E0913 00:07:16.952301       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clustertemplates?resourceVersion=11933&timeout=30m0s&timeoutSeconds=417: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952304       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ProjectLogging: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projectloggings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=441: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952340       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.EtcdBackup: Get https://localhost:6443/apis/management.cattle.io/v3/watch/etcdbackups?resourceVersion=11935&timeout=30m0s&timeoutSeconds=559: read tcp [::1]:36016->[::1]:6443: read: connection reset by peer
E0913 00:07:16.952351       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.RKEK8sSystemImage: Get https://localhost:6443/apis/management.cattle.io/v3/watch/rkek8ssystemimages?resourceVersion=11933&timeout=30m0s&timeoutSeconds=552: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952385       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRoleBinding: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterrolebindings?resourceVersion=11927&timeout=30m0s&timeoutSeconds=435: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952395       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Secret: Get https://localhost:6443/api/v1/watch/namespaces/cattle-system/secrets?resourceVersion=11927&timeout=30m0s&timeoutSeconds=408: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952437       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Namespace: Get https://localhost:6443/api/v1/watch/namespaces?resourceVersion=11927&timeout=30m0s&timeoutSeconds=575: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952452       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.GlobalRole: Get https://localhost:6443/apis/management.cattle.io/v3/watch/globalroles?resourceVersion=11933&timeout=30m0s&timeoutSeconds=500: read tcp [::1]:36014->[::1]:6443: read: connection reset by peer
E0913 00:07:16.952480       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Secret: Get https://localhost:6443/api/v1/watch/secrets?resourceVersion=11927&timeout=30m0s&timeoutSeconds=401: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952501       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.UserAttribute: Get https://localhost:6443/apis/management.cattle.io/v3/watch/userattributes?resourceVersion=11933&timeout=30m0s&timeoutSeconds=460: read tcp [::1]:36012->[::1]:6443: read: connection reset by peer
E0913 00:07:16.952521       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Project: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projects?resourceVersion=11933&timeout=30m0s&timeoutSeconds=526: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952537       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.MultiClusterAppRevision: Get https://localhost:6443/apis/management.cattle.io/v3/watch/multiclusterapprevisions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=465: read tcp [::1]:36024->[::1]:6443: read: connection reset by peer
E0913 00:07:16.952566       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.GlobalRoleBinding: Get https://localhost:6443/apis/management.cattle.io/v3/watch/globalrolebindings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=314: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952585       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.NodeTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/nodetemplates?resourceVersion=11933&timeout=30m0s&timeoutSeconds=312: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952604       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Setting: Get https://localhost:6443/apis/management.cattle.io/v3/watch/settings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=368: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952629       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.App: Get https://localhost:6443/apis/project.cattle.io/v3/watch/apps?resourceVersion=11933&timeout=30m0s&timeoutSeconds=303: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952652       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Group: Get https://localhost:6443/apis/management.cattle.io/v3/watch/groups?resourceVersion=11933&timeout=30m0s&timeoutSeconds=312: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952676       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ListenConfig: Get https://localhost:6443/apis/management.cattle.io/v3/watch/listenconfigs?resourceVersion=11933&timeout=30m0s&timeoutSeconds=577: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952696       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.GlobalDNS: Get https://localhost:6443/apis/management.cattle.io/v3/watch/globaldnses?resourceVersion=11933&timeout=30m0s&timeoutSeconds=588: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952717       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Cluster: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clusters?resourceVersion=11969&timeout=30m0s&timeoutSeconds=497: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952740       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.NodeDriver: Get https://localhost:6443/apis/management.cattle.io/v3/watch/nodedrivers?resourceVersion=11933&timeout=30m0s&timeoutSeconds=513: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952783       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Secret: Get https://localhost:6443/api/v1/watch/namespaces/cattle-global-data/secrets?resourceVersion=11927&timeout=30m0s&timeoutSeconds=596: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952784       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterTemplateRevision: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clustertemplaterevisions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=404: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952878       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.RKEAddon: Get https://localhost:6443/apis/management.cattle.io/v3/watch/rkeaddons?resourceVersion=11933&timeout=30m0s&timeoutSeconds=377: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952928       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ProjectRoleTemplateBinding: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projectroletemplatebindings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=536: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952936       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.CatalogTemplateVersion: Get https://localhost:6443/apis/management.cattle.io/v3/watch/catalogtemplateversions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=549: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.952971       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.AuthConfig: Get https://localhost:6443/apis/management.cattle.io/v3/watch/authconfigs?resourceVersion=11933&timeout=30m0s&timeoutSeconds=442: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953016       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.GlobalRoleBinding: Get https://localhost:6443/apis/management.cattle.io/v3/watch/globalrolebindings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=497: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953016       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.RoleTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/roletemplates?resourceVersion=11933&timeout=30m0s&timeoutSeconds=374: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953059       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Token: Get https://localhost:6443/apis/management.cattle.io/v3/watch/tokens?resourceVersion=11933&timeout=30m0s&timeoutSeconds=366: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953070       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.User: Get https://localhost:6443/apis/management.cattle.io/v3/watch/users?resourceVersion=11933&timeout=30m0s&timeoutSeconds=544: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953130       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.SourceCodeRepository: Get https://localhost:6443/apis/project.cattle.io/v3/watch/sourcecoderepositories?resourceVersion=11933&timeout=30m0s&timeoutSeconds=460: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953164       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterRoleTemplateBinding: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clusterroletemplatebindings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=344: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953187       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Pipeline: Get https://localhost:6443/apis/project.cattle.io/v3/watch/pipelines?resourceVersion=11933&timeout=30m0s&timeoutSeconds=375: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953251       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.RKEK8sServiceOption: Get https://localhost:6443/apis/management.cattle.io/v3/watch/rkek8sserviceoptions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=351: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953275       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.MultiClusterAppRevision: Get https://localhost:6443/apis/management.cattle.io/v3/watch/multiclusterapprevisions?resourceVersion=11933&timeout=30m0s&timeoutSeconds=400: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953290       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.GlobalRole: Get https://localhost:6443/apis/management.cattle.io/v3/watch/globalroles?resourceVersion=11933&timeout=30m0s&timeoutSeconds=391: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953336       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.GlobalDNS: Get https://localhost:6443/apis/management.cattle.io/v3/watch/namespaces/cattle-global-data/globaldnses?resourceVersion=11933&timeout=30m0s&timeoutSeconds=511: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953363       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.KontainerDriver: Get https://localhost:6443/apis/management.cattle.io/v3/watch/kontainerdrivers?resourceVersion=11933&timeout=30m0s&timeoutSeconds=525: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953388       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.SourceCodeCredential: Get https://localhost:6443/apis/project.cattle.io/v3/watch/sourcecodecredentials?resourceVersion=11933&timeout=30m0s&timeoutSeconds=315: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953411       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.NodePool: Get https://localhost:6443/apis/management.cattle.io/v3/watch/nodepools?resourceVersion=11933&timeout=30m0s&timeoutSeconds=556: read tcp [::1]:36056->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953429       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Role: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1/watch/roles?resourceVersion=11927&timeout=30m0s&timeoutSeconds=422: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953467       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.GroupMember: Get https://localhost:6443/apis/management.cattle.io/v3/watch/groupmembers?resourceVersion=11933&timeout=30m0s&timeoutSeconds=378: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953471       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Token: Get https://localhost:6443/apis/management.cattle.io/v3/watch/tokens?resourceVersion=11933&timeout=30m0s&timeoutSeconds=332: read tcp [::1]:36054->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953509       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.DynamicSchema: Get https://localhost:6443/apis/management.cattle.io/v3/watch/dynamicschemas?resourceVersion=11933&timeout=30m0s&timeoutSeconds=382: read tcp [::1]:36052->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953512       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.MultiClusterApp: Get https://localhost:6443/apis/management.cattle.io/v3/watch/multiclusterapps?resourceVersion=11933&timeout=30m0s&timeoutSeconds=352: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953551       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.ClusterRole: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1/watch/clusterroles?resourceVersion=11927&timeout=30m0s&timeoutSeconds=588: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953569       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterCatalog: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clustercatalogs?resourceVersion=11933&timeout=30m0s&timeoutSeconds=570: read tcp [::1]:36050->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953606       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.MultiClusterApp: Get https://localhost:6443/apis/management.cattle.io/v3/watch/multiclusterapps?resourceVersion=11933&timeout=30m0s&timeoutSeconds=422: read tcp [::1]:36044->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953611       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.GlobalDNS: Get https://localhost:6443/apis/management.cattle.io/v3/watch/globaldnses?resourceVersion=11933&timeout=30m0s&timeoutSeconds=301: read tcp [::1]:36048->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953650       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.Secret: Get https://localhost:6443/api/v1/watch/secrets?resourceVersion=11927&timeout=30m0s&timeoutSeconds=329: read tcp [::1]:36046->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953657       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.GlobalDNSProvider: Get https://localhost:6443/apis/management.cattle.io/v3/watch/globaldnsproviders?resourceVersion=11935&timeout=30m0s&timeoutSeconds=424: read tcp [::1]:36042->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953682       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ProjectCatalog: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projectcatalogs?resourceVersion=11933&timeout=30m0s&timeoutSeconds=581: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953689       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v1.RoleBinding: Get https://localhost:6443/apis/rbac.authorization.k8s.io/v1/watch/rolebindings?resourceVersion=11927&timeout=30m0s&timeoutSeconds=319: read tcp [::1]:36032->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953703       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.CatalogTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/catalogtemplates?resourceVersion=11933&timeout=30m0s&timeoutSeconds=411: read tcp [::1]:36040->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953723       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Cluster: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clusters?resourceVersion=11969&timeout=30m0s&timeoutSeconds=452: read tcp [::1]:36038->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953731       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.RKEK8sSystemImage: Get https://localhost:6443/apis/management.cattle.io/v3/watch/rkek8ssystemimages?resourceVersion=11933&timeout=30m0s&timeoutSeconds=585: read tcp [::1]:36034->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953748       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ProjectRoleTemplateBinding: Get https://localhost:6443/apis/management.cattle.io/v3/watch/projectroletemplatebindings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=577: read tcp [::1]:36036->[::1]:6443: read: connection reset by peer
E0913 00:07:16.953777       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.CatalogTemplate: Get https://localhost:6443/apis/management.cattle.io/v3/watch/catalogtemplates?resourceVersion=11933&timeout=30m0s&timeoutSeconds=436: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953789       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.Catalog: Get https://localhost:6443/apis/management.cattle.io/v3/watch/catalogs?resourceVersion=11933&timeout=30m0s&timeoutSeconds=509: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953820       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterCatalog: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clustercatalogs?resourceVersion=11933&timeout=30m0s&timeoutSeconds=374: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953884       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.RKEAddon: Get https://localhost:6443/apis/management.cattle.io/v3/watch/rkeaddons?resourceVersion=11933&timeout=30m0s&timeoutSeconds=456: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.953932       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.ClusterRoleTemplateBinding: Get https://localhost:6443/apis/management.cattle.io/v3/watch/clusterroletemplatebindings?resourceVersion=11933&timeout=30m0s&timeoutSeconds=595: dial tcp [::1]:6443: connect: connection refused
E0913 00:07:16.954969       7 reflector.go:270] github.com/rancher/norman/controller/generic_controller.go:175: Failed to watch *v3.NodePool: Get https://localhost:6443/apis/management.cattle.io/v3/watch/nodepools?resourceVersion=11933&timeout=30m0s&timeoutSeconds=529: dial tcp [::1]:6443: connect: connection refused
2019/09/13 00:07:16 [FATAL] k3s exited with: exit status 1

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 6
  • Comments: 55 (5 by maintainers)

Most upvoted comments

Facing the same issue with 2.5.8

+9
piyush94 on May 6, 2021

I am also getting this issue. I fixed it by reverting to Rancher 2.3.5, which isn’t the best solution, but it’ll have to do until this is fixed.

+5
JamsheedMistri on Dec 26, 2020

v2.4.17 seems to be the latest version that works without issues.

+3
abayomi185 on Oct 15, 2021

Its been a couple months now, is there any update on this issue? Rancher now crashes every 3-5 minutes on my RPI 4. I know its not exactly a target environment for running rancher but perhaps is should be? I run rancher/RKE at home on a bunch of raspberry Pis because at work(a rancher paying customer) we use rancher, and I like to try out things in my homelab to help understand it better before running it in production, besides rancher 2.x used to run just fine on a raspberry pi. Tracking resource utilization right before the crash shows that rancher is 100% utilizing disk IO. I would also like to mention that /var/lib/rancher mounted on an iscsi lun hosted by a synology disk array, not the local sd card.

+3
DavidKittleSEL on Aug 19, 2020

My generally theory here is that slow disk writes cause the etcd heartbeat check to fail which then causes k3s crash.

@cjellick we think that too. Happens to us still with 2.3.5. We have two different single rancher instances running on the same host (both single deploy) and both crash at the very same time.

Still: Disk slowliness is very much bad but still shouldn’t kill the etcd imho.

+3
phlegx on Feb 26, 2020

v2.4.17 seems to be the latest version that works without issues.

This older version worked for me as well. While my hardware should be sufficient (a cheap SSD), even aside from any performance preferences, I think it’s clear that crashing is not the preferred alternative to slow execution.

+2
krisvek on Nov 22, 2021

I was able to fix this by using an instance with more processing power

+2
WJay-tec on Oct 16, 2021

So looking back on this, this is definitely a resource issue that can prop up over time as your cluster grows larger / you add more clusters. The Rancher container runs a K3s cluster backed up by etcd inside it’s container. Etcd is a resource hog when it comes to the performance of the drive it runs on.

But there’s some light on the horizon. With the release of Rancher 2.5.x it’s now possible to migrate your Rancher cluster from a standalone container to a Kubernetes / K3s cluster. How is that different from K3s inside a Docker container? Well you can choose to have your K3s cluster backed by SQlite or Postgres for example. Both of which require a lot less disk resources to run.

Rancher has some information in their docs about migrating Rancher from a Docker container to a cluster, but it’s missing a lot of details and is aimed towards people that have an S3 bucket available. I however managed to figure out how to do do it, without the S3 requirement and I’ve published a blog post on my personal blog that details every step of the way https://vashiru.tech/migrating-rancher-single-node-docker-to-a-ha-kubernetes-cluster/.

It will not directly solve your crashing issues, but you might be able to work around them in the future.

+1
rene-demonsters on Jan 31, 2022

I am sitting on a Ryzen Threadripper 3970X 64GB of 3600MHz Memory and Samsung EVO Plus 970 NVMe, so I would guess hardware requirements of the single node rancher deployment should be met. But currentlty I am facing a similar same issue on Ubuntu 21.10. Rancher crashes and restarts after only 11 secs of starting.

+1
l4b4r4b4b4 on Nov 17, 2021

I’ve the same problem, rancher 2.5.5

+1
gricuk on Mar 31, 2021

tested with 2.4.13 and 2.5.5, see the same problem.

+1
DanielYWoo on Feb 22, 2021

We’re still experiencing this issue, even after upgrading the storage performance. So I’ve done some more research and found the following: https://www.ibm.com/cloud/blog/using-fio-to-tell-whether-your-storage-is-fast-enough-for-etcd https://gist.github.com/patrick0057/16b1e9510095b2675ed74540556b6712

I ran those storage benchmarks using fio version 3.6 (need 3.5 or higher for the fdatasync part) (both 22mb and 100mb) on the volume, these are the results:

22 MB test fio --rw=write --ioengine=sync --fdatasync=1 --directory=/fio-test --size=22m --bs=2300 --name=mytest

root@rancher-prod1:/mnt/sdc/test-data# docker run --mount 'type=bind,source=/mnt/sdc/test-data,target=/fio-test' --rm ljishen/fio --rw=write --ioengine=sync --fdatasync=1 --directory=/fio-test --size=22m --bs=2300 --name=mytest
mytest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.6
Starting 1 process
mytest: Laying out IO file (1 file / 22MiB)

mytest: (groupid=0, jobs=1): err= 0: pid=9: Mon Aug 24 15:55:00 2020
  write: IOPS=217, BW=489KiB/s (501kB/s)(21.0MiB/46087msec)
    clat (usec): min=8, max=521, avg=17.81, stdev=11.76
     lat (usec): min=8, max=523, avg=19.03, stdev=12.07
    clat percentiles (usec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   11], 20.00th=[   12],
     | 30.00th=[   13], 40.00th=[   15], 50.00th=[   16], 60.00th=[   17],
     | 70.00th=[   19], 80.00th=[   22], 90.00th=[   28], 95.00th=[   34],
     | 99.00th=[   49], 99.50th=[   65], 99.90th=[  125], 99.95th=[  141],
     | 99.99th=[  396]
   bw (  KiB/s): min=  139, max=  570, per=100.00%, avg=488.16, stdev=80.39, samples=92
   iops        : min=   62, max=  254, avg=217.49, stdev=35.76, samples=92
  lat (usec)   : 10=3.89%, 20=72.48%, 50=22.69%, 100=0.67%, 250=0.24%
  lat (usec)   : 500=0.02%, 750=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=1703, max=376622, avg=4567.17, stdev=7544.63
    sync percentiles (usec):
     |  1.00th=[  1909],  5.00th=[  2008], 10.00th=[  2089], 20.00th=[  2212],
     | 30.00th=[  2376], 40.00th=[  3884], 50.00th=[  4948], 60.00th=[  5211],
     | 70.00th=[  5407], 80.00th=[  5669], 90.00th=[  6259], 95.00th=[  7046],
     | 99.00th=[ 10814], 99.50th=[ 15270], 99.90th=[ 46400], 99.95th=[ 82314],
     | 99.99th=[375391]
  cpu          : usr=0.30%, sys=1.53%, ctx=22689, majf=0, minf=12
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10029,0,0 short=10029,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=489KiB/s (501kB/s), 489KiB/s-489KiB/s (501kB/s-501kB/s), io=21.0MiB (23.1MB), run=46087-46087msec

Disk stats (read/write):
  sdc: ios=0/25580, merge=0/5612, ticks=0/44828, in_queue=44820, util=97.50%

100 MB test fio --rw=write --ioengine=sync --fdatasync=1 --directory=/fio-test --size=100m --bs=2300 --name=mytest

root@rancher-prod1:/mnt/sdc/test-data# docker run --mount 'type=bind,source=/mnt/sdc/test-data,target=/fio-test' --rm ljishen/fio --rw=write --ioengine=sync --fdatasync=1 --directory=/fio-test --size=100m --bs=2300 --name=mytest
mytest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.6
Starting 1 process
mytest: Laying out IO file (1 file / 100MiB)

mytest: (groupid=0, jobs=1): err= 0: pid=9: Mon Aug 24 15:36:23 2020
  write: IOPS=199, BW=448KiB/s (459kB/s)(99.0MiB/228486msec)
    clat (usec): min=6, max=1262, avg=18.78, stdev=15.12
     lat (usec): min=7, max=1264, avg=19.97, stdev=15.36
    clat percentiles (usec):
     |  1.00th=[   10],  5.00th=[   11], 10.00th=[   12], 20.00th=[   13],
     | 30.00th=[   14], 40.00th=[   15], 50.00th=[   16], 60.00th=[   18],
     | 70.00th=[   20], 80.00th=[   23], 90.00th=[   30], 95.00th=[   36],
     | 99.00th=[   55], 99.50th=[   84], 99.90th=[  139], 99.95th=[  165],
     | 99.99th=[  758]
   bw (  KiB/s): min=    4, max=  592, per=100.00%, avg=449.64, stdev=120.08, samples=454
   iops        : min=    2, max=  264, avg=200.36, stdev=53.43, samples=454
  lat (usec)   : 10=3.00%, 20=69.50%, 50=26.14%, 100=1.04%, 250=0.29%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=1650, max=819073, avg=4982.61, stdev=11553.19
    sync percentiles (usec):
     |  1.00th=[  1876],  5.00th=[  1991], 10.00th=[  2073], 20.00th=[  2212],
     | 30.00th=[  2409], 40.00th=[  3949], 50.00th=[  4948], 60.00th=[  5211],
     | 70.00th=[  5473], 80.00th=[  5800], 90.00th=[  6587], 95.00th=[  7767],
     | 99.00th=[ 13960], 99.50th=[ 25822], 99.90th=[170918], 99.95th=[246416],
     | 99.99th=[517997]
  cpu          : usr=0.26%, sys=1.45%, ctx=111829, majf=0, minf=12
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,45590,0,0 short=45590,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=448KiB/s (459kB/s), 448KiB/s-448KiB/s (459kB/s-459kB/s), io=99.0MiB (105MB), run=228486-228486msec

Disk stats (read/write):
  sdc: ios=0/116744, merge=0/25595, ticks=0/224032, in_queue=223984, util=98.09%

In both of the sources I listed the 99th percentile latency is about 2.2 ms (which they deem adequate), for my volume it appears to be between 10.8 and 13.9 ms. Seeing as this is about 5 to 7 times as slow, this could be the culprit.

For the sake of completion, I’ve also ran the same benchmark on the previously used storage (where versions prior to 2.3.x used to run fine on). The results of that were:

root@rancher-prod1:~# docker run --mount 'type=bind,source=/root/test-data,target=/fio-test' --rm ljishen/fio --rw=write --ioengine=sync --fdatasync=1 --directory=/fio-test --size=22m --bs=2300 --name=mytest
mytest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.6
Starting 1 process
mytest: Laying out IO file (1 file / 22MiB)

mytest: (groupid=0, jobs=1): err= 0: pid=9: Mon Aug 24 16:08:21 2020
  write: IOPS=226, BW=509KiB/s (521kB/s)(21.0MiB/44286msec)
    clat (usec): min=7, max=794, avg=16.70, stdev=11.56
     lat (usec): min=8, max=795, avg=17.84, stdev=12.04
    clat percentiles (usec):
     |  1.00th=[   10],  5.00th=[   10], 10.00th=[   11], 20.00th=[   12],
     | 30.00th=[   13], 40.00th=[   14], 50.00th=[   15], 60.00th=[   16],
     | 70.00th=[   18], 80.00th=[   20], 90.00th=[   27], 95.00th=[   33],
     | 99.00th=[   46], 99.50th=[   57], 99.90th=[  108], 99.95th=[  119],
     | 99.99th=[  169]
   bw (  KiB/s): min=  148, max=  646, per=99.97%, avg=507.83, stdev=91.27, samples=88
   iops        : min=   66, max=  288, avg=226.27, stdev=40.61, samples=88
  lat (usec)   : 10=6.11%, 20=74.69%, 50=18.49%, 100=0.59%, 250=0.11%
  lat (usec)   : 1000=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=1737, max=413395, avg=4389.53, stdev=8052.62
    sync percentiles (usec):
     |  1.00th=[  1909],  5.00th=[  2008], 10.00th=[  2089], 20.00th=[  2212],
     | 30.00th=[  2376], 40.00th=[  2802], 50.00th=[  4424], 60.00th=[  4621],
     | 70.00th=[  4883], 80.00th=[  5080], 90.00th=[  5604], 95.00th=[  6456],
     | 99.00th=[ 13566], 99.50th=[ 27395], 99.90th=[113771], 99.95th=[135267],
     | 99.99th=[367002]
  cpu          : usr=0.26%, sys=1.58%, ctx=23465, majf=0, minf=12
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10029,0,0 short=10029,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=509KiB/s (521kB/s), 509KiB/s-509KiB/s (521kB/s-521kB/s), io=21.0MiB (23.1MB), run=44286-44286msec

Disk stats (read/write):
  sda: ios=0/25647, merge=0/5699, ticks=0/42892, in_queue=42888, util=97.01%

Currently attempting to mitigate the latency issues by using 4 volumes in software raid0. Will update once test results come in.

+1
rene-demonsters on Aug 24, 2020

@theAkito Yeah well aware it’s a bug, just sharing information in hopes that it help somebody else out that’s running into this.

The Rancher instance that’s having issues is actually using SSD Volumes from CloudVPS: https://www.cloudvps.com/openstack/block-volumes/, not HDDs. Before we were using the SSD of the system itself but that had become increasingly unstable over time. But going by your statement, I figure it sounds like my SSD solution is still rather slow? In that case, anything you can recommend (like a specific tier on Digital Ocean, AWS or something at TransIP) that I can migrate to in order to mitigate this issue? (Rather than polluting this ticket, I’ve temporarily set my email on my profile).

+1
rene-demonsters on Jun 22, 2020

I see the same thing happening on a test setup with v2.4.0-rc9 (also with 2.3.x) This is on a hosted Openstack environment where there is very limited control over disk speeds. etcdserver throws a LOT of ‘read-only range request “key:xyz” took too long (xxx.xxxms) to execute’ It’s on a dedicated instance with 4 cores (2physical), 8GB and SSD storage, there is a k8s test cluster with prometheus/grafana and isto running but I have seen it happen on a fresh install with 0 clusters.

+1
dannietjoh on Mar 25, 2020

The fix for this issue is assumed to correct the underlying cause of a massive number of writes to etcd resulting in the container failing. These writes are thought to come from nodesyncer.go which had a bug which could cause nodesyncer to loop extremely quickly and constantly update the v3.Node object. These high number of writes would then overwhelm etcd.

For testing: This needs to be tested multiple times since this bug is transient and hard to pin down. Please reopen if you think you see this issue again.

+1
daxmc99 on Sep 17, 2019
Read more comments on GitHub
← rancher: Sporadic driver error during K8s upgrade: listen tcp 127.0.0.1:0: bind: address already in use
rancher: Stuck on "Waiting to register with Kubernetes" →