kubernetes: Incompatibility between kubelet and kubeadm and kubectl on which label prefix to use

There has to be an “official” straightforward way to assign node roles via the kubeadm config file. If node-labels is not going to support this use case, some other mechanism/key should. Assume that you have something like an auto-scaling group for “worker” nodes. The most natural thing to do would be to provision them with a kubeadm config file that assigns “worker” roles to these nodes at join time, without jumping any unnecessary hoops. IMO, a solution that first assigns a temporary taint and then labels the node later on in the provisioning process via kubectl looks like a temporary work-around, not a permanent idiom.

A variety of use cases with autoscaling groups, managed instance groups, or scale sets rely on the node-role ability to create sets of nodes labled beforehand for different purposes. It seems to me unlikely these cases will introduce an operator-style components that just-in-time labels nodes as they come and go. I’m not sure how the control plane initialization process could be enough - especially in preemptible worker clusters, nodes are added after control plane bringup, so it would need to be a longer running process.

I get what we’re trying to achieve here security-wise (adversary takeover of kubelet, change role). But I suspect a great many folks, will find less than intended workarounds, that will ultimately undermine those efforts. They’ll run a process with credentials kubelet lacks. Any solutions involving temporary labels or taints seem to miss the original point - if we’re worried about an adversary changing the flag, that same adversary can change the temporary label / taint flag.

This is going to be interesting or I’m missing something. Hopefully the later.

Doesn’t it make sense that kubectl should support a ROLE label having been set in one of the valid node domains, e.g. node.kubernetes.io/role=“CustomerA”` so that customers can continue to define specialist node roles (except ‘master’) and not break current functionality?

NAME                                         STATUS   ROLES        AGE    VERSION
ip-10-29-11-143.us-west-2.compute.internal   Ready    nodepool-a   105m   v1.14.1
ip-10-29-12-157.us-west-2.compute.internal   Ready    master       112m   v1.14.1
ip-10-29-20-101.us-west-2.compute.internal   Ready    master       112m   v1.14.1
ip-10-29-21-209.us-west-2.compute.internal   Ready    nodepool-b   105m   v1.14.1
ip-10-29-30-121.us-west-2.compute.internal   Ready    nodepool-c   106m   v1.14.1
ip-10-29-31-1.us-west-2.compute.internal     Ready    master       112m   v1.14.1

If I transition to using node.kubernetes.io/role=“xyz” I get a degraded kubectl experience: -

NAME                                         STATUS   ROLES    AGE     VERSION
ip-10-29-15-152.us-west-2.compute.internal   Ready    master   2d18h   v1.14.1
ip-10-29-21-204.us-west-2.compute.internal   Ready    master   2d18h   v1.14.1
ip-10-29-22-212.us-west-2.compute.internal   Ready    <none>   2d17h   v1.14.1
ip-10-29-26-56.us-west-2.compute.internal    Ready    master   2d17h   v1.14.1
ip-10-29-30-211.us-west-2.compute.internal   Ready    <none>   2d17h   v1.14.1
ip-10-29-9-185.us-west-2.compute.internal    Ready    <none>   2d17h   v1.14.1

@liggitt: Question about the third option: If --node-labels functionality is not there, how would one label their nodes at setup time (not after join) via kubeadm? Today, the most convenient way I know of is to use node-labels in kubeletExtraArgs when using kubeadm. Note that this concern applies to all nodes, not just control plane nodes.

I also think node-labels in kubeletExtraArgs is a very useful feature from the perspective of Cluster API. node-role.kubernetes.io label is very useful in that it is the default printcolumm of kubectl get node.

I’m looking for a solution to this issue. I use Kube-aws setup with ASG. The kubelet fails to start because of node-role.

Use Case:

I want to keep the Same node label for Master and Worker nodes as I’m using in v1.15 .

When I tried to upgrade the cluster to V1.16 the --node-labels is restricted to use ‘node-role’

If I keep the node role as “node-role.kubernetes.io/master” the kubelet fails to start after upgrade… if I remove the label, ‘kubectl get node’ output shows <none> for the upgraded node.

Yes, there are indeed many places where the node-role prefix is used. It seems like a pragmatic solution could be to add this prefix to the allowed list (hence not breaking anyone), but disallowing certain label(s) when self-labeling so that one won’t be able to arbitrarily label itself as a master node.

the picture around the node-role label limitations on the kubelet side were explained above. TL;DR:

• kubeadm is using an API client with privileged credentials to set node-roles. it would only do that on control-plane nodes using the common “master” label (to be changed to “control-plane” in the future).
• kubeadm has no plans to apply node-roles for other node types.
• high level tools / admins can do that as long as they have credentials.
• the kubelet will not allow you to self-set node roles which is by design.
• kubectl special casing the display for node-role labels is a topic for sig/cli

if you have arguments around the node-role limitations this is a topic for sig/node (kubelet) and sig/architecture (see this)

/remove-sig cluster-lifecycle /remove-area kubeadm

Finally, I don’t see more references to node-roles in kubeadm?

deployers like kubeadm, kops and kubespray will continue relying on the "node-role.kubernetes.io/master" labels.

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/constants/constants.go#L206

@liggitt How about this label prefix node-role.kubernetes.io which used across projects ? Could you give some guideline about whether add it to allowed list or change it to node.kubernetes.io ?

I can work on this ticket with your guideline, thanks. /assign

Ref: https://github.com/kubernetes/kubernetes/pull/68267