kubernetes: kubelet: lookup node address logic for external provider assign wrong PodIPs to hostNetwork pods

The kubelet logic used to get the Node addresses when using an external provider, temporary assigns an address discovered from the node , this address can be overriden later by the external cloud provider

https://github.com/kubernetes/kubernetes/blob/0241da314e0e69817d66313b45a69c19d1ce7327/pkg/kubelet/nodestatus/setters.go#L123-L131

This was implemented in https://github.com/kubernetes/kubernetes/pull/75229 to improve the bootstrapping problems with some external cloud providers

The side effect of this behavior is that if the external cloud provider sets a different address/es, then the pods with HostNetwork: true will be showing the “first” addresses as their PodIPs, since we don’t mutate those IPs once set to avoid problems

https://github.com/kubernetes/kubernetes/blob/0241da314e0e69817d66313b45a69c19d1ce7327/pkg/kubelet/kubelet_pods.go#L1770-L1786

I think that we should revert the existing logic for external cloud providers, since is causing a temporary and inconsistent state and it seem to address mainly a problem with kubeadm https://github.com/kubernetes/kubernetes/pull/75229#issuecomment-516592142 but kubeadm folks seems to not need this functionality https://github.com/kubernetes/kubernetes/pull/75229#issuecomment-516580449

/kind bug /sig network /sig node

About this issue

  • Original URL
  • State: closed
  • Created 9 months ago
  • Comments: 29 (29 by maintainers)

Most upvoted comments

For current behavior, the workaround would be to delete hostNetwork:true pods and wait for recreation if I understand correctly.

I don’t know if there are any cloud provider installers that depend on this behavior. I prefer to fix the weird behavior to avoid potential problems. Or a problem detector has to cover this to check if a hostNet pod is not using the correct node IP.

@lubronzhan that should not be a problem if having node addresses is a requirement of node readiness https://github.com/kubernetes/kubernetes/pull/120753

This is why we taint nodes initially and only remove the taint once cloud provider has initialized the node. So this is reallly an issue for pods tolerating this taint

yeah, but I think that is fair to give those pods the responsibility to deal with the known behavior (node will not have addresses until cloud provider external initializes) , than make the kubelet have two different addresses during its lifetime (that we discussed extensively https://github.com/kubernetes/kubernetes/issues/108281 and https://github.com/kubernetes/kubernetes/pull/93898) We discarded this behavior because most components, apps and features like downward API depend on the existing behavior of not changing the original IP

For the temporal status without addresses we can just set the node as notReady https://github.com/kubernetes/kubernetes/issues/120727

The side effect of this behavior is that if the external cloud provider sets a different address/es, then the pods with HostNetwork: true will be showing the “first” addresses as their PodIPs, since we don’t mutate those IPs once set to avoid problems

This is why we taint nodes initially and only remove the taint once cloud provider has initialized the node. So this is reallly an issue for pods tolerating this taint