kubernetes: Nodes behind NAT not starting

I have a hetzner vserver with a static private IP of 172.31.1.100 which is 1:1 NAT’ed from its external internet routable IP. When the node registers itself with the master it exposes its private IP which is not routable from a master even though nodename is set to a resolvable fqdn resolving to its external ip. When i manually set nodeIP to its external internet routable ip the node fails to start with the following error:

Oct 20 11:45:16 n2 origin-node: W1020 11:45:16.882798    9085 kubelet_node_status.go:934] Failed to set some node status fields: failed to validate nodeIP: Node IP: "x.x.x.x" not found in the host's network interfaces

I tracked this error down to https://github.com/kubernetes/kubernetes/blob/48da128d83a3a1dd94e4bcc1bfec997c23138a93/pkg/kubelet/kubelet_node_status.go#L1004

One workaround i applied and seems to be working is to set an ip alias on the node using the external ip but this is too hackish and in my opinion wrong.

Is the above scenario supported in any way?

openshift origin rpm: origin-node-3.6.0-1.0.c4dd4cf kubernetes:

kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1+5115d708d7", GitCommit:"fff65cf", GitTreeState:"clean", BuildDate:"2017-08-02T21:42:22Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

@kubernetes/sig-node-bugs

About this issue

Original URL
State: open
Created 7 years ago
Comments: 20 (15 by maintainers)

Commits related to this issue

Add kubelet flag to disable nodeIP validation May be desired if the Kubelet is running on a server behind a NAT. Fixes #54337 — committed to alvaroaleman/kubernetes by alvaroaleman 6 years ago

Most upvoted comments

When the node registers itself with the master it exposes its private IP which is not routable from a master

This is not uncommon. There are plenty of environments where this is true, and in fact there’s an explict API for the control planes to talk back to nodes: https://github.com/kubernetes-sigs/apiserver-network-proxy

The downside of using the public IP is that you can end up sending traffic out the NAT and back in when you do node-to-node traffic.

thockin on Sep 14, 2023

I have a hetzner vserver with a static private IP of 172.31.1.100 which is 1:1 NAT’ed from its external internet routable IP. When the node registers itself with the master it exposes its private IP which is not routable from a master

So, given #42125, this would be solvable with kube-apiserver --kubelet-preferred-address-types ExternalIP,InternalIP

danwinship on Sep 14, 2023