kubeadm: Unable to join workers with BGP Unnumbered RFC 5549

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version): 1.17.12

Environment:

  • Kubernetes version (use kubectl version): 1.17.12
  • Cloud provider or hardware configuration: baremetal
  • OS (e.g. from /etc/os-release): Centos 7
  • Kernel (e.g. uname -a): 3.10.0-1127.19.1.el7.x86_64
  • Others: Calico 3.13.3

What happened?

Related to the following: https://github.com/kubernetes/kubeadm/issues/1156 https://github.com/kubernetes/kubernetes/pull/83475

When I attempt to kubeadm join a worker with routing to the host (unicast IPv4 on the loopback, IPv6 link locals on interfaces), kubeadm fails to join workers with the following output:

$ kubeadm join --config=/etc/kubernetes/kubeadm-join.yaml
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
error execution phase preflight: unable to fetch the kubeadm-config ConfigMap: unable to select an IP from default routes.

verbose error:

[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
I1009 00:03:38.372308    8018 interface.go:400] Looking for default routes with IPv4 addresses
I1009 00:03:38.372350    8018 interface.go:405] Default route transits interface "ens15f0.300"
I1009 00:03:38.372793    8018 interface.go:208] Interface ens15f0.300 is up
I1009 00:03:38.372911    8018 interface.go:256] Interface "ens15f0.300" has 1 addresses :[fe80::a236:9fff:fe80:8258/64].
I1009 00:03:38.372959    8018 interface.go:223] Checking addr  fe80::a236:9fff:fe80:8258/64.
I1009 00:03:38.372984    8018 interface.go:236] fe80::a236:9fff:fe80:8258 is not an IPv4 address
I1009 00:03:38.373017    8018 interface.go:400] Looking for default routes with IPv6 addresses
I1009 00:03:38.373034    8018 interface.go:405] Default route transits interface "ens15f1.300"
I1009 00:03:38.373375    8018 interface.go:208] Interface ens15f1.300 is up
I1009 00:03:38.373465    8018 interface.go:256] Interface "ens15f1.300" has 1 addresses :[fe80::a236:9fff:fe80:825a/64].
I1009 00:03:38.373502    8018 interface.go:223] Checking addr  fe80::a236:9fff:fe80:825a/64.
I1009 00:03:38.373523    8018 interface.go:233] Non-global unicast address found fe80::a236:9fff:fe80:825a
I1009 00:03:38.373542    8018 interface.go:405] Default route transits interface "ens15f0.300"
I1009 00:03:38.374553    8018 interface.go:208] Interface ens15f0.300 is up
I1009 00:03:38.374644    8018 interface.go:256] Interface "ens15f0.300" has 1 addresses :[fe80::a236:9fff:fe80:8258/64].
I1009 00:03:38.374696    8018 interface.go:223] Checking addr  fe80::a236:9fff:fe80:8258/64.
I1009 00:03:38.374720    8018 interface.go:233] Non-global unicast address found fe80::a236:9fff:fe80:8258
I1009 00:03:38.374741    8018 interface.go:416] No active IP found by looking at default routes
unable to select an IP from default routes.
unable to fetch the kubeadm-config ConfigMap
k8s.io/kubernetes/cmd/kubeadm/app/cmd.fetchInitConfiguration
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/join.go:526
k8s.io/kubernetes/cmd/kubeadm/app/cmd.fetchInitConfigurationFromJoinConfiguration
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/join.go:494
k8s.io/kubernetes/cmd/kubeadm/app/cmd.(*joinData).InitCfg
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/join.go:456
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runPreflight
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join/preflight.go:95
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdJoin.func1
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/join.go:170
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
k8s.io/kubernetes/cmd/kubeadm/app.Run
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
	_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1357
error execution phase preflight
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdJoin.func1
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/join.go:170
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
k8s.io/kubernetes/cmd/kubeadm/app.Run
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
	_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1357

What you expected to happen?

It looks like the fix introduced in this PR only works for control plane nodes. I’d like to be able to join workers with the node’s unicast address being on the loopback interface.

How to reproduce it (as minimally and precisely as possible)?

On a node with BGP Unnumbered via RFC 5549 + a loopback unicast, run kubeadm join --config=/etc/kubernetes/kubeadm-join.yaml.

kubeadm-join.yaml:

apiVersion: kubeadm.k8s.io/v1beta2
kind: JoinConfiguration
discovery:
  file:
    kubeConfigPath: /etc/kubernetes/discovery.conf
  tlsBootstrapToken: 'boostrap-token-here'
nodeRegistration:
  name: worker-1.cluster.example.com
  kubeletExtraArgs:
    node_ip: 10.x.x.x

discovery.conf:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: b64-encoded-ca-cert-here
    server: https://api.cluster.example.com
  name: ""
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []

Anything else we need to know?

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 31 (23 by maintainers)

Most upvoted comments

we can log a separate issue (again) in kubernetes/kubernetes about ResolveBindAddress and tag it with /sig api-machinery proposing the change. you can create this one and explain the use case it good detail for the SIG API Machinery maintainers.

also please ping me and @aojea on that ticket.

showing the full kubeadm output and the kubeadm use case is not directly relevant, only this part:

I1009 00:03:38.372308    8018 interface.go:400] Looking for default routes with IPv4 addresses
I1009 00:03:38.372350    8018 interface.go:405] Default route transits interface "ens15f0.300"
I1009 00:03:38.372793    8018 interface.go:208] Interface ens15f0.300 is up
I1009 00:03:38.372911    8018 interface.go:256] Interface "ens15f0.300" has 1 addresses :[fe80::a236:9fff:fe80:8258/64].
I1009 00:03:38.372959    8018 interface.go:223] Checking addr  fe80::a236:9fff:fe80:8258/64.
I1009 00:03:38.372984    8018 interface.go:236] fe80::a236:9fff:fe80:8258 is not an IPv4 address
I1009 00:03:38.373017    8018 interface.go:400] Looking for default routes with IPv6 addresses
I1009 00:03:38.373034    8018 interface.go:405] Default route transits interface "ens15f1.300"
I1009 00:03:38.373375    8018 interface.go:208] Interface ens15f1.300 is up
I1009 00:03:38.373465    8018 interface.go:256] Interface "ens15f1.300" has 1 addresses :[fe80::a236:9fff:fe80:825a/64].
I1009 00:03:38.373502    8018 interface.go:223] Checking addr  fe80::a236:9fff:fe80:825a/64.
I1009 00:03:38.373523    8018 interface.go:233] Non-global unicast address found fe80::a236:9fff:fe80:825a
I1009 00:03:38.373542    8018 interface.go:405] Default route transits interface "ens15f0.300"
I1009 00:03:38.374553    8018 interface.go:208] Interface ens15f0.300 is up
I1009 00:03:38.374644    8018 interface.go:256] Interface "ens15f0.300" has 1 addresses :[fe80::a236:9fff:fe80:8258/64].
I1009 00:03:38.374696    8018 interface.go:223] Checking addr  fe80::a236:9fff:fe80:8258/64.
I1009 00:03:38.374720    8018 interface.go:233] Non-global unicast address found fe80::a236:9fff:fe80:8258
I1009 00:03:38.374741    8018 interface.go:416] No active IP found by looking at default routes
...

your interface setup and the proposed DIFF are relevant, i guess.

log a new ticket in kubernetes/kubeadm about the dynamic default / don’t download configuration for workers problem. i will log this ticket now.

this is now here:

https://github.com/kubernetes/kubeadm/issues/2328

close this issue.

/close

and sorry for the shuffle of issues, but your report surfaced a number of problems in old code. thanks.

+2
neolit123 on Oct 16, 2020

Yes, as said today there are two problems at stake

  1. autodetecting the IP address.
  2. the overall config fetch & defaults management.

For 1, @eknudtson provided a possible fix, and I will ask people to have eyes on IT (this is a tricky piece of code, so more eyes, the better).

For 2, my suggestion is to avoid to rush a solution, and possibly track the problem in a separate issue My initial reaction is that we should avoid changing the whole chain of fetch / defaulting because of the possible blast radio, instead I will explore if we can have a better distinction between join control plane (which requires init configuration) and join worker (which does not init configuration, but currently is reading it no matter of)

+1
fabriziopandini on Oct 14, 2020

why we should modify the kubernetes code to select the default route if BGP can install the routes?

i had to Google BGP when we logged this ticket so i’m really not familiar with this.

+1
neolit123 on Oct 14, 2020

cc @aojea do you happen to know if this use case is supported (k8s-wide, see OP)?

+1
neolit123 on Oct 14, 2020
Read more comments on GitHub
← kubeadm: Token not being added to configmap after kubeadm token create
kubeadm: Unable to read config path "/etc/kubernetes/manifests": path does not exist, ignoring  →