kubernetes: Apiserver to kubelet communication should not use NodeName as address.
In pkg/registry/pod/strategy.go the host address for kubelets seem to be derived from pod.Spec.nodeName. It should however use the Node.Status.Addresses field to find an IP (the docs seem to indicate InternalIP should be used: https://github.com/kubernetes/kubernetes/blob/master/docs/admin/node.md#node-addresses
When a NodeName is resolved to an address/IP which is not open to the apiserver for various reasons (in our case eth0 is blocked for all ingress traffic by security groups) several operations fail. This affects apiserver -> kubelet communication like exec, attach, port-forward.
I’ll try addressing this with a PR, when it’s acknowledged that this is a valid problem.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 6
- Comments: 54 (47 by maintainers)
Commits related to this issue
- Use nodeutil.GetHostIP consistently when talking to nodes Most of our communications from apiserver -> nodes used nodutil.GetNodeHostIP, but a few places didn't - and this meant that the node name ne... — committed to justinsb/kubernetes by justinsb 8 years ago
- Merge pull request #25532 from mkulke/resolve-nodename-in-kubelet-comm Automatic merge from submit-queue Populate Node.Status.Addresses with Hostname This PR is supposed to address #22063 Curre... — committed to kubernetes/kubernetes by deleted user 8 years ago
I think the ideal scenario would do something like this:
--override-hostname?)That enables topologies with the apiserver/nodes in the same internal network, or in different networks, with and without DNS.
sorry… was unclear. resolve node name to address by looking up the node object and inspecting the reported addresses in its status. That’s what #33718 is doing, defaulting to a reported IP, and what #25532 is doing, defaulting to a reported hostname.
I looked a bit more into the problem, it’s definitely not a low hanging fruit. Changing this behavior (from using
Pod.Spec.NodeNametoNode.Status.Addressesof the respective node) will require changes in a lot of places. It would mean that we have to defer the TLS cert generation in kubelet until the Addresses list is populated (so we can use those as alt subject in cert).This list in turn is populated by cloud-providers, which somewhat arbitrarily put addresses there, not necessarily the one to which we bound the kubelet server.
I’d suggest to use the kubelet’s
--addressparameter in the self-signed cert generation as an alternative subject ( the params we have available at this point). Then the kubelet logic for populatingNode.Status.Addressesneeds to be extended to add this address with a special typeKubeletIP. The apiserver would then connect to the kubelet using this field andPod.Spec.NodeNameas fallback.Makes some sense? I think it does not break anything. OTOH it’s pretty immersive for something which you could consider as a merely cosmetic issue (nodename == kubelet’s bound ip) or an issue in the realm of the cloud providers (should not rely on nodename for instance discovery).
This has come up many times (see #2462).
Yes, we shouldn’t try to resolve nodeName. Addresses are intended to be used. We’ve talked about adding more information to that array, to better indicate which address should be used by apiserver.