kubernetes: a Node with name and status "Ready" already exists in the cluster.

What happened:

After upgrading to v1.18.0 not possible to join existing node into cluster:

# /usr/bin/kubeadm join 10.28.36.157:8443 --token <redacted> --discovery-token-ca-cert-hash 
...
I0326 11:57:39.638919   12149 kubelet.go:145] [kubelet-start] Checking for an existing Node in the cluster with name "m1c4" and status "Ready"
a Node with name "m1c4" and status "Ready" already exists in the cluster. You must delete the existing Node or change the name of this new joining Node
Full log 
W0326 11:57:39.322862   12149 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
I0326 11:57:39.322941   12149 join.go:371] [preflight] found NodeName empty; using OS hostname as NodeName
I0326 11:57:39.322968   12149 initconfiguration.go:103] detected and using CRI socket: /var/run/dockershim.sock
[preflight] Running pre-flight checks
I0326 11:57:39.323001   12149 preflight.go:90] [preflight] Running general checks
I0326 11:57:39.323031   12149 checks.go:249] validating the existence and emptiness of directory /etc/kubernetes/manifests
I0326 11:57:39.323070   12149 checks.go:286] validating the existence of file /etc/kubernetes/kubelet.conf
I0326 11:57:39.323077   12149 checks.go:286] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I0326 11:57:39.323086   12149 checks.go:102] validating the container runtime
I0326 11:57:39.407093   12149 checks.go:128] validating if the service is enabled and active
I0326 11:57:39.487777   12149 checks.go:335] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0326 11:57:39.487834   12149 checks.go:335] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0326 11:57:39.487861   12149 checks.go:649] validating whether swap is enabled or not
I0326 11:57:39.487890   12149 checks.go:376] validating the presence of executable conntrack
I0326 11:57:39.487922   12149 checks.go:376] validating the presence of executable ip
I0326 11:57:39.487954   12149 checks.go:376] validating the presence of executable iptables
I0326 11:57:39.487982   12149 checks.go:376] validating the presence of executable mount
I0326 11:57:39.488009   12149 checks.go:376] validating the presence of executable nsenter
I0326 11:57:39.488030   12149 checks.go:376] validating the presence of executable ebtables
I0326 11:57:39.488054   12149 checks.go:376] validating the presence of executable ethtool
I0326 11:57:39.488076   12149 checks.go:376] validating the presence of executable socat
I0326 11:57:39.488096   12149 checks.go:376] validating the presence of executable tc
I0326 11:57:39.488118   12149 checks.go:376] validating the presence of executable touch
I0326 11:57:39.488146   12149 checks.go:520] running all checks
I0326 11:57:39.560686   12149 checks.go:406] checking whether the given node name is reachable using net.LookupHost
I0326 11:57:39.560800   12149 checks.go:618] validating kubelet version
I0326 11:57:39.598654   12149 checks.go:128] validating if the service is enabled and active
I0326 11:57:39.609631   12149 checks.go:201] validating availability of port 10250
I0326 11:57:39.609728   12149 checks.go:286] validating the existence of file /etc/kubernetes/pki/ca.crt
I0326 11:57:39.609743   12149 checks.go:432] validating if the connectivity type is via proxy or direct
I0326 11:57:39.609764   12149 join.go:441] [preflight] Discovering cluster-info
I0326 11:57:39.609782   12149 token.go:78] [discovery] Created cluster-info discovery client, requesting info from "10.28.36.157:8443"
I0326 11:57:39.616900   12149 token.go:116] [discovery] Requesting info from "10.28.36.157:8443" again to validate TLS against the pinned public key
I0326 11:57:39.622631   12149 token.go:133] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.28.36.157:8443"
I0326 11:57:39.622650   12149 discovery.go:51] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process
I0326 11:57:39.622659   12149 join.go:455] [preflight] Fetching init configuration
I0326 11:57:39.622667   12149 join.go:493] [preflight] Retrieving KubeConfig objects
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
I0326 11:57:39.635104   12149 interface.go:400] Looking for default routes with IPv4 addresses
I0326 11:57:39.635123   12149 interface.go:405] Default route transits interface "vmbr0"
I0326 11:57:39.635884   12149 interface.go:208] Interface vmbr0 is up
I0326 11:57:39.636606   12149 interface.go:256] Interface "vmbr0" has 3 addresses :[10.28.36.164/16 2a02:2b88:1:2:9657:a5ff:fed3:b6f2/64 fe80::9657:a5ff:fed3:b6f2/64].
I0326 11:57:39.636663   12149 interface.go:223] Checking addr  10.28.36.164/16.
I0326 11:57:39.636681   12149 interface.go:230] IP found 10.28.36.164
I0326 11:57:39.636696   12149 interface.go:262] Found valid IPv4 address 10.28.36.164 for interface "vmbr0".
I0326 11:57:39.636710   12149 interface.go:411] Found active IP 10.28.36.164 
I0326 11:57:39.636959   12149 preflight.go:101] [preflight] Running configuration dependant checks
I0326 11:57:39.636986   12149 controlplaneprepare.go:211] [download-certs] Skipping certs download
I0326 11:57:39.637019   12149 kubelet.go:111] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I0326 11:57:39.638010   12149 kubelet.go:119] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt
I0326 11:57:39.638919   12149 kubelet.go:145] [kubelet-start] Checking for an existing Node in the cluster with name "m1c4" and status "Ready"
a Node with name "m1c4" and status "Ready" already exists in the cluster. You must delete the existing Node or change the name of this new joining Node
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runKubeletStartJoinPhase
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join/kubelet.go:152
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdJoin.func1
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/join.go:170
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
k8s.io/kubernetes/cmd/kubeadm/app.Run
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
	_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1357
error execution phase kubelet-start
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdJoin.func1
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/join.go:170
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
k8s.io/kubernetes/cmd/kubeadm/app.Run
	/workspace/anago-v1.18.0-rc.1.21+8be33caaf953ac/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
	_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1357

The problem is that the check introduced in https://github.com/kubernetes/kubernetes/commit/b117a928a6c3f650931bdac02a41fca6680548c4 violates the existing diskless node approach.

Eg. we are having cluster of hundreds servers which are booding over PXE and have no any local storage. After the boot they just running kubeadm join command to connect to the cluster, they can be rebooted ay any time, and after reboot they will run join procedure again.

Thus node object in cluster may already have some settings, labels and annotation for this node. After the joining the node just reuses them, this is working as a charm.

What you expected to happen:

Existing node successful joined

How to reproduce it (as minimally and precisely as possible):

kubeadm join
kubeadm reset
kubeadm join

Anything else we need to know?:

Currently if you try to join the node you will face with https://github.com/kubernetes/kubernetes/issues/89501, but I created cluster role to check this chage.

Environment:

  • Kubernetes version: v1.18.0
  • Cloud provider or hardware configuration: Bare Metal
  • OS: Ubuntu 18.04.3 LTS
  • Kernel: 4.15.18-24-pve
  • Install tools: kubeadm
  • Network plugin and version kube-router v0.4.0
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 19 (19 by maintainers)

Most upvoted comments

ok, i will send a cherry pick for the PR to 1.18

+1
neolit123 on Mar 27, 2020

You’re right it is 40 seconds:

  {
    "lastHeartbeatTime": "2020-03-27T23:11:23Z",
    "lastTransitionTime": "2020-03-27T23:10:53Z",
    "message": "kubelet is posting ready status. AppArmor enabled",
    "reason": "KubeletReady",
    "status": "True",
    "type": "Ready"
  }
  {
    "lastHeartbeatTime": "2020-03-27T23:11:23Z",
    "lastTransitionTime": "2020-03-27T23:12:33Z",
    "message": "Kubelet stopped posting node status.",
    "reason": "NodeStatusUnknown",
    "status": "Unknown",
    "type": "Ready"
  }
+1
kvaps on Mar 27, 2020

it did work in my tests. i guess, it would not work for the case where the condition is still present but not True, so your PR makes sense.

+1
neolit123 on Mar 27, 2020
Read more comments on GitHub
← kubernetes: _output/bin/deepcopy-gen: Permission denied
kubernetes: A single resource cache sync failure shouldn't break the entire ResourceQuota controller logic →