rancher: Intermittent error seen while cluster is provisioning and provisioning is stalled for long time before it gets to active state.

I thought rancher will be an easier way to install k8s…

Hello, downstream cluster remains stuck in Provisioning state. Please assist. Thanks.

Rancher v2.7.4 Downstream : RKE1, K8s v1.25.9

Error while applying agent YAML, it will be retried automatically: exit status 1, Error from server (Forbidden): error when retrieving current configuration of: Resource: “rbac.authorization.k8s.io/v1, Resource=clusterroles”, GroupVersionKind: “rbac.authorization.k8s.io/v1, Kind=ClusterRole” Name: “proxy-clusterrole-kubeapiserver”, Namespace: “” from server for: “./management-statefile_path_redacted”: clusterroles.rbac.authorization.k8s.io “proxy-clusterrole-kubeapiserver” is forbidden: User “u-pl3h4p7xtj” cannot get resource “clusterroles” in API group “rbac.authorization.k8s.io” at the cluster scope Error from server (Forbidden): error when retrieving current configuration of: Resource: “rbac.authorization.k8s.io/v1, Resource=clusterrolebindings”, GroupVersionKind: “rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding” Name: “proxy-role-binding-kubernetes-master”, Namespace: “” from server for: “./management-statefile_path_redacted”: clusterrolebindings.rbac.authorization.k8s.io “proxy-role-binding-kubernetes-master” is forbidden: User “u-pl3h4p7xtj” cannot get resource “clusterrolebindings” in API group “rbac.authorization.k8s.io” at the cluster scope Error from server (Forbidden): error when retrieving current configuration of: Resource: “/v1, Resource=namespaces”, GroupVersionKind: “/v1, Kind=Namespace” Name: “cattle-system”, Namespace: “” from server for: “./management-statefile_path_redacted”: namespaces “cattle-system” is forbidden: User “u-pl3h4p7xtj” cannot get resource “namespaces” in API group “” in the namespace “cattle-system” Error from server (Forbidden): error when retrieving current configuration of: Resource: “/v1, Resource=serviceaccounts”, GroupVersionKind: “/v1, Kind=ServiceAccount” Name: “cattle”, Namespace: “cattle-system” from server for: “./management-statefile_path_redacted”: serviceaccounts “cattle” is forbidden: User “u-pl3h4p7xtj” cannot get resource “serviceaccounts” in API group “” in the namespace “cattle-system” Error from server (Forbidden): error when retrieving current configuration of: Resource: “rbac.authorization.k8s.io/v1, Resource=clusterrolebindings”, GroupVersionKind: “rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding” Name: “cattle-admin-binding”, Namespace: “” from server for: “./management-statefile_path_redacted”: clusterrolebindings.rbac.authorization.k8s.io “cattle-admin-binding” is forbidden: User “u-pl3h4p7xtj” cannot get resource “clusterrolebindings” in API group “rbac.authorization.k8s.io” at the cluster scope Error from server (Forbidden): error when retrieving current configuration of: Resource: “/v1, Resource=secrets”, GroupVersionKind: “/v1, Kind=Secret” Name: “cattle-credentials-df6de44”, Namespace: “cattle-system” from server for: “./management-statefile_path_redacted”: secrets “cattle-credentials-df6de44” is forbidden: User “u-pl3h4p7xtj” cannot get resource “secrets” in API group “” in the namespace “cattle-system” Error from server (Forbidden): error when retrieving current configuration of: Resource: “rbac.authorization.k8s.io/v1, Resource=clusterroles”, GroupVersionKind: “rbac.authorization.k8s.io/v1, Kind=ClusterRole” Name: “cattle-admin”, Namespace: “” from server for: “./management-statefile_path_redacted”: clusterroles.rbac.authorization.k8s.io “cattle-admin” is forbidden: User “u-pl3h4p7xtj” cannot get resource “clusterroles” in API group “rbac.authorization.k8s.io” at the cluster scope Error from server (Forbidden): error when retrieving current configuration of: Resource: “apps/v1, Resource=deployments”, GroupVersionKind: “apps/v1, Kind=Deployment” Name: “cattle-cluster-agent”, Namespace: “cattle-system” from server for: “./management-statefile_path_redacted”: deployments.apps “cattle-cluster-agent” is forbidden: User “u-pl3h4p7xtj” cannot get resource “deployments” in API group “apps” in the namespace “cattle-system” Error from server (Forbidden): error when retrieving current configuration of: Resource: “apps/v1, Resource=daemonsets”, GroupVersionKind: “apps/v1, Kind=DaemonSet” Name: “cattle-node-agent”, Namespace: “cattle-system” from server for: “./management-statefile_path_redacted”: daemonsets.apps “cattle-node-agent” is forbidden: User “u-pl3h4p7xtj” cannot get resource “daemonsets” in API group “apps” in the namespace “cattle-system” Error from server (Forbidden): error when retrieving current configuration of: Resource: “apps/v1, Resource=daemonsets”, GroupVersionKind: “apps/v1, Kind=DaemonSet” Name: “kube-api-auth”, Namespace: “cattle-system” from server for: “./management-statefile_path_redacted”: daemonsets.apps “kube-api-auth” is forbidden: User “u-pl3h4p7xtj” cannot get resource “daemonsets” in API group “apps” in the namespace “cattle-system” Error from server (Forbidden): error when retrieving current configuration of: Resource: “/v1, Resource=services”, GroupVersionKind: “/v1, Kind=Service” Name: “cattle-cluster-agent”, Namespace: “cattle-system” from server for: “./management-statefile_path_redacted”: services “cattle-cluster-agent” is forbidden: User “u-pl3h4p7xtj” cannot get resource “services” in API group “” in the namespace “cattle-system”

Seeing the same issue with Rancher on Exoscale. It can take anywhere between 10 to 60 minutes to recover, sometimes it doesn’t. The deployment is fully automated (using the terraform’s rancher driver) so there is no difference or manual changes between attempts.

rancher | v2.4.5 User Interface | v2.4.28 Helm | v2.16.8-rancher1 Machine | v0.15.0-rancher43

Update: seeing the same now on Azure & AWS. It does not appear to be a vendor issue but a Rancher (etcd/controld?) startup issue.

This error happens basically on every deployment (9 out of 10 are faulty).

Provider: Amazon EC2 K8s Version: v1.18.15 Rancher Version: v2.4.8 1 cp/etcd node + 1 worker node (t2.large each)

Error while applying agent YAML, it will be retried automatically: exit status 1, Error from server (Forbidden): error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "proxy-clusterrole-kubeapiserver", Namespace: "" Object: &{map["apiVersion":"rbac.authorization.k8s.io/v1" "kind":"ClusterRole" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"proxy-clusterrole-kubeapiserver"] "rules":[map["apiGroups":[""] "resources":["nodes/metrics" "nodes/proxy" "nodes/stats" "nodes/log" "nodes/spec"] "verbs":["get" "list" "watch" "create"]]]]} from server for: "management-statefile_path_redacted": clusterroles.rbac.authorization.k8s.io "proxy-clusterrole-kubeapiserver" is forbidden: User "u-7f4nygfh3f" cannot get resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterrolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" Name: "proxy-role-binding-kubernetes-master", Namespace: "" Object: &{map["apiVersion":"rbac.authorization.k8s.io/v1" "kind":"ClusterRoleBinding" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"proxy-role-binding-kubernetes-master"] "roleRef":map["apiGroup":"rbac.authorization.k8s.io" "kind":"ClusterRole" "name":"proxy-clusterrole-kubeapiserver"] "subjects":[map["apiGroup":"rbac.authorization.k8s.io" "kind":"User" "name":"kube-apiserver"]]]} from server for: "management-statefile_path_redacted": clusterrolebindings.rbac.authorization.k8s.io "proxy-role-binding-kubernetes-master" is forbidden: User "u-7f4nygfh3f" cannot get resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): error when retrieving current configuration of: Resource: "/v1, Resource=namespaces", GroupVersionKind: "/v1, Kind=Namespace" Name: "cattle-system", Namespace: "" Object: &{map["apiVersion":"v1" "kind":"Namespace" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"cattle-system"]]} from server for: "management-statefile_path_redacted": namespaces "cattle-system" is forbidden: User "u-7f4nygfh3f" cannot get resource "namespaces" in API group "" in the namespace "cattle-system" Error from server (Forbidden): error when retrieving current configuration of: Resource: "/v1, Resource=serviceaccounts", GroupVersionKind: "/v1, Kind=ServiceAccount" Name: "cattle", Namespace: "cattle-system" Object: &{map["apiVersion":"v1" "kind":"ServiceAccount" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"cattle" "namespace":"cattle-system"]]} from server for: "management-statefile_path_redacted": serviceaccounts "cattle" is forbidden: User "u-7f4nygfh3f" cannot get resource "serviceaccounts" in API group "" in the namespace "cattle-system" Error from server (Forbidden): error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1beta1, Resource=clusterrolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1beta1, Kind=ClusterRoleBinding" Name: "cattle-admin-binding", Namespace: "" Object: &{map["apiVersion":"rbac.authorization.k8s.io/v1beta1" "kind":"ClusterRoleBinding" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "labels":map["cattle.io/creator":"norman"] "name":"cattle-admin-binding"] "roleRef":map["apiGroup":"rbac.authorization.k8s.io" "kind":"ClusterRole" "name":"cattle-admin"] "subjects":[map["kind":"ServiceAccount" "name":"cattle" "namespace":"cattle-system"]]]} from server for: "management-statefile_path_redacted": clusterrolebindings.rbac.authorization.k8s.io "cattle-admin-binding" is forbidden: User "u-7f4nygfh3f" cannot get resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): error when retrieving current configuration of: Resource: "/v1, Resource=secrets", GroupVersionKind: "/v1, Kind=Secret" Name: "cattle-credentials-1552bb9", Namespace: "cattle-system" Object: &{map["apiVersion":"v1" "data":map["namespace":"Yy0ycThkZg==" "token":"REDACTED" "url":"aHR0cHM6Ly9yYW5jaGVyLWRldi5sa3YtZ2Vub2NlbGwuZGU="] "kind":"Secret" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"cattle-credentials-1552bb9" "namespace":"cattle-system"] "type":"Opaque"]} from server for: "management-statefile_path_redacted": secrets "cattle-credentials-1552bb9" is forbidden: User "u-7f4nygfh3f" cannot get resource "secrets" in API group "" in the namespace "cattle-system" Error from server (Forbidden): error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=clusterroles", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=ClusterRole" Name: "cattle-admin", Namespace: "" Object: &{map["apiVersion":"rbac.authorization.k8s.io/v1" "kind":"ClusterRole" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "labels":map["cattle.io/creator":"norman"] "name":"cattle-admin"] "rules":[map["apiGroups":["*"] "resources":["*"] "verbs":["*"]] map["nonResourceURLs":["*"] "verbs":["*"]]]]} from server for: "management-statefile_path_redacted": clusterroles.rbac.authorization.k8s.io "cattle-admin" is forbidden: User "u-7f4nygfh3f" cannot get resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): error when retrieving current configuration of: Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment" Name: "cattle-cluster-agent", Namespace: "cattle-system" Object: &{map["apiVersion":"apps/v1" "kind":"Deployment" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"cattle-cluster-agent" "namespace":"cattle-system"] "spec":map["selector":map["matchLabels":map["app":"cattle-cluster-agent"]] "template":map["metadata":map["labels":map["app":"cattle-cluster-agent"]] "spec":map["affinity":map["nodeAffinity":map["preferredDuringSchedulingIgnoredDuringExecution":[map["preference":map["matchExpressions":[map["key":"node-role.kubernetes.io/controlplane" "operator":"In" "values":["true"]]]] "weight":'d'] map["preference":map["matchExpressions":[map["key":"node-role.kubernetes.io/etcd" "operator":"In" "values":["true"]]]] "weight":'\x01']] "requiredDuringSchedulingIgnoredDuringExecution":map["nodeSelectorTerms":[map["matchExpressions":[map["key":"beta.kubernetes.io/os" "operator":"NotIn" "values":["windows"]]]]]]]] "containers":[map["env":[map["name":"CATTLE_FEATURES" "value":"dashboard=true"] map["name":"CATTLE_SERVER" "value":"https://domain.tld"] map["name":"CATTLE_CA_CHECKSUM" "value":""] map["name":"CATTLE_CLUSTER" "value":"true"] map["name":"CATTLE_K8S_MANAGED" "value":"true"]] "image":"rancher/rancher-agent:v2.4.8" "imagePullPolicy":"IfNotPresent" "name":"cluster-register" "volumeMounts":[map["mountPath":"/cattle-credentials" "name":"cattle-credentials" "readOnly":%!q(bool=true)]]]] "serviceAccountName":"cattle" "tolerations":[map["operator":"Exists"]] "volumes":[map["name":"cattle-credentials" "secret":map["defaultMode":'\u0140' "secretName":"cattle-credentials-1552bb9"]]]]]]]} from server for: "management-statefile_path_redacted": deployments.apps "cattle-cluster-agent" is forbidden: User "u-7f4nygfh3f" cannot get resource "deployments" in API group "apps" in the namespace "cattle-system" Error from server (Forbidden): error when retrieving current configuration of: Resource: "apps/v1, Resource=daemonsets", GroupVersionKind: "apps/v1, Kind=DaemonSet" Name: "cattle-node-agent", Namespace: "cattle-system" Object: &{map["apiVersion":"apps/v1" "kind":"DaemonSet" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"cattle-node-agent" "namespace":"cattle-system"] "spec":map["selector":map["matchLabels":map["app":"cattle-agent"]] "template":map["metadata":map["labels":map["app":"cattle-agent"]] "spec":map["affinity":map["nodeAffinity":map["requiredDuringSchedulingIgnoredDuringExecution":map["nodeSelectorTerms":[map["matchExpressions":[map["key":"beta.kubernetes.io/os" "operator":"NotIn" "values":["windows"]]]]]]]] "containers":[map["env":[map["name":"CATTLE_NODE_NAME" "valueFrom":map["fieldRef":map["fieldPath":"spec.nodeName"]]] map["name":"CATTLE_SERVER" "value":"https://domain.tld"] map["name":"CATTLE_CA_CHECKSUM" "value":""] map["name":"CATTLE_CLUSTER" "value":"false"] map["name":"CATTLE_K8S_MANAGED" "value":"true"] map["name":"CATTLE_AGENT_CONNECT" "value":"true"]] "image":"rancher/rancher-agent:v2.4.8" "imagePullPolicy":"IfNotPresent" "name":"agent" "securityContext":map["privileged":%!q(bool=true)] "volumeMounts":[map["mountPath":"/cattle-credentials" "name":"cattle-credentials" "readOnly":%!q(bool=true)] map["mountPath":"/etc/kubernetes" "name":"k8s-ssl"] map["mountPath":"/var/run" "name":"var-run"] map["mountPath":"/run" "name":"run"] map["mountPath":"/etc/docker/certs.d" "name":"docker-certs"]]]] "hostNetwork":%!q(bool=true) "serviceAccountName":"cattle" "tolerations":[map["operator":"Exists"]] "volumes":[map["hostPath":map["path":"/etc/kubernetes" "type":"DirectoryOrCreate"] "name":"k8s-ssl"] map["hostPath":map["path":"/var/run" "type":"DirectoryOrCreate"] "name":"var-run"] map["hostPath":map["path":"/run" "type":"DirectoryOrCreate"] "name":"run"] map["name":"cattle-credentials" "secret":map["defaultMode":'\u0140' "secretName":"cattle-credentials-1552bb9"]] map["hostPath":map["path":"/etc/docker/certs.d" "type":"DirectoryOrCreate"] "name":"docker-certs"]]]] "updateStrategy":map["rollingUpdate":map["maxUnavailable":"25%"] "type":"RollingUpdate"]]]} from server for: "management-statefile_path_redacted": daemonsets.apps "cattle-node-agent" is forbidden: User "u-7f4nygfh3f" cannot get resource "daemonsets" in API group "apps" in the namespace "cattle-system" Error from server (Forbidden): error when retrieving current configuration of: Resource: "apps/v1, Resource=daemonsets", GroupVersionKind: "apps/v1, Kind=DaemonSet" Name: "kube-api-auth", Namespace: "cattle-system" Object: &{map["apiVersion":"apps/v1" "kind":"DaemonSet" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "name":"kube-api-auth" "namespace":"cattle-system"] "spec":map["selector":map["matchLabels":map["app":"kube-api-auth"]] "template":map["metadata":map["labels":map["app":"kube-api-auth"]] "spec":map["affinity":map["nodeAffinity":map["requiredDuringSchedulingIgnoredDuringExecution":map["nodeSelectorTerms":[map["matchExpressions":[map["key":"beta.kubernetes.io/os" "operator":"NotIn" "values":["windows"]] map["key":"node-role.kubernetes.io/controlplane" "operator":"In" "values":["true"]]]]]]]] "containers":[map["image":"rancher/kube-api-auth:v0.1.4" "imagePullPolicy":"IfNotPresent" "name":"kube-api-auth" "securityContext":map["privileged":%!q(bool=true)] "volumeMounts":[map["mountPath":"/etc/kubernetes" "name":"k8s-ssl"]]]] "hostNetwork":%!q(bool=true) "serviceAccountName":"cattle" "tolerations":[map["operator":"Exists"]] "volumes":[map["hostPath":map["path":"/etc/kubernetes" "type":"DirectoryOrCreate"] "name":"k8s-ssl"]]]] "updateStrategy":map["rollingUpdate":map["maxUnavailable":"25%"] "type":"RollingUpdate"]]]} from server for: "management-statefile_path_redacted": daemonsets.apps "kube-api-auth" is forbidden: User "u-7f4nygfh3f" cannot get resource "daemonsets" in API group "apps" in the namespace "cattle-system"

Is there any workaround possible yet?

I am consistently running into this issue when deploying a custom cluster. I can confirm that it does take quite a few minutes but eventually will recover and the new cluster will become active. Rancher version is 2.4.8 and the K8 version of the cluster is 1.17.4. Rancher runs on 3 nodes on AWS spread among 3 AZ’s and downstream masters are the same configuration.