rancher: failed to call leader func: settings.management.cattle.io "install-uuid" not found

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible): install k3s on single host:

# kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-7ff9579c6-np754   1/1     Running     0          11m
kube-system   coredns-66c464876b-2xrfj                 1/1     Running     0          11m
kube-system   metrics-server-7b4f8b595-4s6jb           1/1     Running     0          11m
kube-system   helm-install-traefik-nhpxp               0/1     Completed   0          11m
kube-system   svclb-traefik-5mqlf                      2/2     Running     0          10m
kube-system   traefik-5dd496474-4xtcj                  1/1     Running     0          10m

register a cluster on rancher ui, copy the command, then execute it on k3s server node :

kubectl apply -f https://rancher.xxx.xx/v3/import/kq8fmcwvc4c7thj7dlmzng6fddbsdkhbczgd6gkcbjd9sc9qh8dzpt.yaml
kubectl get pods -A
NAMESPACE       NAME                                     READY   STATUS      RESTARTS   AGE
kube-system     metrics-server-7b4f8b595-4s6jb           1/1     Running     0          132m
kube-system     helm-install-traefik-nhpxp               0/1     Completed   0          132m
kube-system     svclb-traefik-lj6bl                      2/2     Running     0          117m
kube-system     coredns-66c464876b-2xrfj                 1/1     Running     1          132m
kube-system     traefik-5dd496474-4xtcj                  1/1     Running     0          131m
cattle-system   cattle-cluster-agent-7595fcc4b5-qt744    1/1     Running     0          38m
kube-system     local-path-provisioner-7ff9579c6-np754   1/1     Running     2          132m

Result: When executing the command kubectl -n cattle-system logs -l app=cattle-cluster-agent -f , the following error is continuously displayed in the log.

time="2021-02-28T10:12:49Z" level=error msg="failed to call leader func: settings.management.cattle.io \"install-uuid\" not found"
time="2021-02-28T10:12:54Z" level=error msg="failed to call leader func: settings.management.cattle.io \"install-uuid\" not found"
time="2021-02-28T10:13:00Z" level=error msg="failed to call leader func: settings.management.cattle.io \"install-uuid\" not found"
......

The rancher server is no problem.

# curl https://rancher.xxx.xx/ping
pong

Access the racher server inside the container,as follows:

# docker exec -it  d2234d175000 sh
sh-4.4# curl https://rancher.xxx.xx/ping
pongsh-4.4# curl https://rancher.xxx.xx/apis/management.cattle.io/v3/settings/install-uuid
not authorized

Log in to rancher ui, and open https://rancher.xxx.xx/apis/management.cattle.io/v3/settings/install-uuid in the browser, the response is as follows:

{
	apiVersion: "management.cattle.io/v3",
	customized: false,
	default: "",
	kind: "Setting",
	metadata: {
		creationTimestamp: "2021-02-26T10:45:40Z",
		generation: 2,
		managedFields: [{
			apiVersion: "management.cattle.io/v3",
			fieldsType: "FieldsV1",
			fieldsV1: {
				f: customized: {},
				f: default: {},
				f: source: {},
				f: value: {}
			},
			manager: "rancher",
			operation: "Update",
			time: "2021-02-26T10:45:40Z"
		}],
		name: "install-uuid",
		resourceVersion: "3459",
		selfLink: "/apis/management.cattle.io/v3/settings/install-uuid",
		uid: "0016c7c8-86bc-4966-a20e-2d1c54c433f9"
	},
	source: "",
	value: "a9d1eb14-475a-4ec7-bc4f-5c7312d2286e"
}

The log of rancher server pod is as follows (kubectl -n cattle-system -l app=rancher logs -f):

2021/02/28 00:32:54 [INFO] Stopping cluster agent for c-p2fqw
2021/02/28 00:32:54 [ERROR] failed to start cluster controllers c-p2fqw: context canceled

Other details that may be helpful:

Environment information

  • Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): v2.5.5
  • Installation option (single install/HA): single install

Cluster information

  • Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Custom
  • Machine type (cloud/VM/metal) and specifications (CPU/memory): cloud, 1 core, 1GB
  • OS version: CentOS Linux release 7.9.2009 (Core)
  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.8+k3s1", GitCommit:"95fc76b206b1455db7d50e23fb45c98d6ea97dde", GitTreeState:"clean", BuildDate:"2021-02-22T19:46:43Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.8+k3s1", GitCommit:"95fc76b206b1455db7d50e23fb45c98d6ea97dde", GitTreeState:"clean", BuildDate:"2021-02-22T19:46:43Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
  • Docker version (use docker version):
Client: Docker Engine - Community
 Version:           20.10.4
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        d3cb89e
 Built:             Thu Feb 25 07:06:20 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:25:42 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 6
  • Comments: 20 (1 by maintainers)

Commits related to this issue

Most upvoted comments

To reproduce this issue when utilizing the APIs. Create the cluster using this format

curl -s "$RANCHER_SERVER/v3/cluster" -H 'content-type: application/json' -H "Authorization: Bearer $APITOKEN" --data-binary '{"dockerRootDir":"/var/lib/docker","enableNetworkPolicy":false,"type":"cluster","rancherKubernetesEngineConfig":{"addonJobTimeout":30,"ignoreDockerVersion":true,"sshAgentAuth":false,"type":"rancherKubernetesEngineConfig","authentication":{"type":"authnConfig","strategy":"x509"},"network":{"type":"networkConfig","plugin":"canal"},"ingress":{"type":"ingressConfig","provider":"nginx"},"monitoring":{"type":"monitoringConfig","provider":"metrics-server"},"services":{"type":"rkeConfigServices","kubeApi":{"podSecurityPolicy":false,"type":"kubeAPIService"},"etcd":{"snapshot":false,"type":"etcdService","extraArgs":{"heartbeat-interval":500,"election-timeout":5000}}}},"name":"'$CLUSTERNAME'"}' 

And then I get the kubectl command to import the cluster like this.

curl -s $RANCHER_SERVER'/v3/clusterregistrationtokens?id="'$CLUSTERID'"&limit=-1' -H 'content-type: application/json' -H "Authorization: Bearer $APITOKEN" -X POST --data-binary '{"clusterId":"'$CLUSTERID'","name":"'$CLUSTERNAME'"}' $INSECURE | jq -r '.command'

This results in errors as indicated above time="2021-02-28T10:12:49Z" level=error msg="failed to call leader func: settings.management.cattle.io \"install-uuid\" not found"

Workaround that I found is to use this format to generate the cluster

curl -s "$RANCHER_SERVER/v3/cluster?_replace=true" -H 'content-type: application/json' -H "Authorization: Bearer $APITOKEN" --data-binary '{"name":"'$CLUSTERNAME'"}'

I can import the cluster successfully with the same command

curl -s $RANCHER_SERVER'/v3/clusterregistrationtokens?id="'$CLUSTERID'"&limit=-1' -H 'content-type: application/json' -H "Authorization: Bearer $APITOKEN" -X POST --data-binary '{"clusterId":"'$CLUSTERID'","name":"'$CLUSTERNAME'"}' | jq -r '.command'