kubeadm: kubeadm init “--apiserver-advertise-address=publicIP” not working, private IP works 1.13 version

I found a solution.

Instead of using --apiserver-advertise-address=publicIp use --apiserver-cert-extra-sans=publicIp

Don’t forget to replace the private ip for the public ip in your .kube/config if you use kubectl from remote.

Hi, I have reproduced your issue and found the root cause. Please find my analysis below.

During kubeadm init, kubelet will try to create kube-proxy, kube-apiserver, kube-controller, and kube-scheduller. It will try to bind all these services with the public IP address(GCP assigned) of the VM.

But the problem in GCP is, the public IP address does not rest on the VM, but it rather a NAT function. The tricky part here is to understand is, if you are sending a packet to the NAT it will forward the packet to the VM and vice-versa. But your process/application cannot bind to that NAT IP address. The IP address with which you intend to create the cluster has to reside on the VM.

That is why it is working with internal IP address but not with public IP address.

You can verify this by checking ‘tail -f /var/log/syslog’ while creating the cluster.

Please let me know if this addressed your issue.

-M

@Zhang21 after a lot of day and night research… i finally found a way to let k8s work over WAN.

short answer

1. --apiserver-advertise-address=publicIP is necessary, this tell k8s worker communicate to master over publicIP. default is privateIP, will lead to 10.96.0.1:443: i/o timeout.
2. node annotation flannel.alpha.coreos.com/public-ip-overwrite=publicIP is necessary，this set flannel pod node-ip to publicIP

full answer

1. first, run ifconfig on master, check whether there have publicIP info on master interfaces. some cloud provider use Elastic IP, they dont have any publicIP interface info. you must add publicIP interface info yourself. follow Cloud_floating_IP_persistent. if you dont add publicIP interface, kubeadm init --apiserver-advertise-address=publicIP will not success.
2. kubeadm init with --apiserver-advertise-address=publicIP, i use --control-plane-endpoint=publicIP --upload-certs --apiserver-advertise-address=publicIP for my own. i think just apiserver-advertise-address also will be ok.
3. apply flannel.yaml and join worker node.
4. on worker node，add kubelet args --node-ip

vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --node-ip=publicIP
systemctl restart kubelet

1. set worker node public-ip-overwrite annotate, and restart worker kube-flannel pod

kubectl annotate node NODE_NAME flannel.alpha.coreos.com/public-ip-overwrite=publicIP

# on worker node
systemctl restart docker

I solved this problem by forwarding the private IP of the master node to the public IP of the master node on the worker node. Specifically, this was the command that I ran on worker node before running kubeadm join: sudo iptables -t nat -A OUTPUT -d <Private IP of master node> -j DNAT --to-destination <Public IP of master node>

I found a solution.

Instead of using --apiserver-advertise-address=publicIp use --apiserver-cert-extra-sans=publicIp

Don’t forget to replace the private ip for the public ip in your .kube/config if you use kubectl from remote.

@x22n Your solution results in a verification error.

$ kubectl get nodes
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

Update

It turns out that the verification issue is due to leftover credentials (in $HOME/.kube) from the last master, before kubeadm reset was performed.

This can be resolved by,

mv  $HOME/.kube $HOME/.kube.bak
mkdir $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

And, the following seems to be a clean solution.

kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=<PRIVATE_IP>[,<PUBLIC_IP>,...]

I just tested the solution above on Compute Engine.

--apiserver-cert-extra-sans=publicIp do not solve the problem. Yes, it adds public IP to the Certs, but do not affects connection procedure. The worker nodes will look for apiserver-advertise-address during join. So they will not connect to the ptivateIP, if no route exist. Api-server itself have two params --advertise-address ip and --bind-address ip. It looks reasonable. But, how can this addresses configured during kubeadm init?

I manged to solve the problem by enabling inbound traffic in port 6443 and used the flag --control-plane-endpoint instead of --apiserver-advertise-address

Even am facing the same issue kubeadm init is failing with public ip and saying that kubelet is misconfigured This error is likely caused by:

• The kubelet is not running
• The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

@Zhang21 after a lot of day and night research… i finally found a way to let k8s work over WAN. short answer

1. --apiserver-advertise-address=publicIP is necessary, this tell k8s worker communicate to master over publicIP. default is privateIP, will lead to 10.96.0.1:443: i/o timeout.
2. node annotation flannel.alpha.coreos.com/public-ip-overwrite=publicIP is necessary，this set flannel pod node-ip to publicIP

full answer

1. first, run ifconfig on master, check whether there have publicIP info on master interfaces. some cloud provider use Elastic IP, they dont have any publicIP interface info. you must add publicIP interface info yourself. follow Cloud_floating_IP_persistent. if you dont add publicIP interface, kubeadm init --apiserver-advertise-address=publicIP will not success.
2. kubeadm init with --apiserver-advertise-address=publicIP, i use --control-plane-endpoint=publicIP --upload-certs --apiserver-advertise-address=publicIP for my own. i think just apiserver-advertise-address also will be ok.
3. apply flannel.yaml and join worker node.
4. on worker node，add kubelet args --node-ip

vim /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --node-ip=publicIP
systemctl restart kubelet

1. set worker node public-ip-overwrite annotate, and restart worker kube-flannel pod

kubectl annotate node NODE_NAME flannel.alpha.coreos.com/public-ip-overwrite=publicIP

# on worker node
systemctl restart docker

warning I tried this solution. It can solve the apiserver timeout problem. but, It will prevent flannel packets from being sent to the destination in Alibaba Cloud. I suggest not to try [Cloud_floating_IP_persistent].

I spend a lot of time in this solution. It work great in the beginning, I can create nodes in two vm (which do not have public ip, all rely on port forwarding ) across two wan, flannel and ingress create successfully. however, when come to udp connection between two node, udp package seen like dropped by router because of the src ip is a WAN IP. so I believe unless you have two machine directly connect to wan or you can counter the anti spoofing rule in your router, this is not a good attend in this topic. I also wonder if there anyway we can send a udp package with WAN src ip from a machine in LAN. Any trick?

yes… still have problem… I am tired, It’s not worth to speed lot of time for it, I finally buy an another machine under the lan, I found spending money makes me happy.

I am trying on different cloud provider but no luck . Any suggestion or solution ?

Thank you @spockmang for the explanation. Just for the record, i faced the same issue on Openstack.

I’m for closing the issue here because this is not a kubeadm problem

The IP address with which you intend to create the cluster has to reside on the VM.

Probably the best option to get a suggestion here is to reopen the issue in k/k and tag SIG network and SIG cloud provider @neolit123 opinions?

Hi, Even i have the exact problem with trying on Azure VMs (Redhat 7 OS). Anyone has suggestions to resolve the issue?

This is what i get every time i do kubeadm init … I tried re-installing kubelet and kubeadm… tried yum update and restarted VM.

[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred: timed out waiting for the condition

This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - ‘systemctl status kubelet’ - ‘journalctl -xeu kubelet’

Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker. Here is one example how you may list all Kubernetes containers running in docker: - ‘docker ps -a | grep kube | grep -v pause’ Once you have found the failing container, you can inspect its logs with: - ‘docker logs CONTAINERID’ error execution phase wait-control-plane: couldn’t initialize a Kubernetes cluster

Thanks Sagar

thanks, @spockmang for the insight. do you have the recommendation or step to make it work with public IP add ?

Iam able to access curl and ping.

On Thu 7 Feb, 2019, 7:37 PM Lubomir I. Ivanov <notifications@github.com wrote:

i cannot see the log for some reason. but if you can’t connect using tools like curl and ping then this is not a kubeadm bug.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubeadm/issues/1390#issuecomment-461432540, or mute the thread https://github.com/notifications/unsubscribe-auth/AINnY4CcC0fNGf4oX4vf4r4Q-OpuToR4ks5vLDMugaJpZM4alDvn .