kubeadm: Kubeadm fails to bring up a HA cluster due to EOF error when uploading configmap

@ereslibre and others, just wanted to say a big thank you for investigating this! It’s great to see such commitment on making kubeadm a great tool to use.

@ereslibre could you kindly check again the EOF problem on your vagrant setup?

I will have a look at it tonight, the race condition fixed by https://github.com/kubernetes/kubernetes/pull/72030 was slightly different, I never saw this one, but I’m happy to try to reproduce the issue and try to find the root cause. I will report back.

It seems that the HA of either etcd or control plane is not working properly.

If you have grown your cluster to 3 masters etcd was also grown to 3. As per the etcd admin guide in a cluster of 3 the fault tolerance is of 1. If you shut down 2 out of 3 etcd instances your etcd cluster will be unavailable.

Please, let’s keep this issue from now on for the certificate issue, since the reporter provided a repository that contained this problem and we are keeping this issue open because of that. This issue is split in 3 different cases:

1. Bad etcd certificates: this issue.
2. Not pre-pulling images when joining a control plane: https://github.com/kubernetes/kubeadm/issues/1341 (fix already merged in master)
3. Not explicitly waiting for etcd to be healthy when we grow the cluster: https://github.com/kubernetes/kubeadm/issues/1353

For any related issue please refer to the explicit issues linked above, otherwise please let’s open a new bug report since this one is already mixing very different things. Thank you!

My problem was not related to k8s. My nginx-lb config had a very short timeout.

I changed my proxy_timeout from 3s to 24h.

Here is my config user nginx; worker_processes 1;

error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid;

events { worker_connections 1024; }

http { include /etc/nginx/mime.types; default_type application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

access_log  /var/log/nginx/access.log  main;

sendfile        on;
#tcp_nopush     on;

keepalive_timeout  65;

#gzip  on;

include /etc/nginx/conf.d/*.conf;

}

stream { upstream apiserver { #server IP1:6443 weight=5 max_fails=9 fail_timeout=30s; server IP2:6443 weight=5 max_fails=9 fail_timeout=30s; #server IP3:6443 weight=5 max_fails=9 fail_timeout=30s; }

server {
    listen 16443;
    proxy_connect_timeout 1s;
    proxy_timeout 24h;
    proxy_pass apiserver;
}
log_format proxy '$remote_addr [$time_local] '
             '$protocol $status $bytes_sent $bytes_received '
             '$session_time "$upstream_addr" '
             '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';
access_log  /var/log/nginx/access.log  proxy;

}

I can see something that will be probably related. When joining the second master to the cluster the https://172.28.128.25:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config endpoint takes far longer to answer than on the third.

Master 2 (~14 seconds):

[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I0115 12:52:38.633739    2027 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.14.0 (linux/amd64) kubernetes/1b28775" 'https://172.28.128.25:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config'
I0115 12:52:52.960533    2027 round_trippers.go:438] GET https://172.28.128.25:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config 200 OK in 14325 milliseconds
I0115 12:52:52.960568    2027 round_trippers.go:444] Response Headers:
I0115 12:52:52.960576    2027 round_trippers.go:447]     Content-Type: application/json
I0115 12:52:52.960593    2027 round_trippers.go:447]     Content-Length: 1149
I0115 12:52:52.960604    2027 round_trippers.go:447]     Date: Tue, 15 Jan 2019 12:52:54 GMT

Master 3 (~40 milliseconds):

[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I0115 12:54:02.519284    2118 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.14.0 (linux/amd64) kubernetes/1b28775" 'https://172.28.128.25:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config'
I0115 12:54:02.557663    2118 round_trippers.go:438] GET https://172.28.128.25:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config 200 OK in 38 milliseconds
I0115 12:54:02.557732    2118 round_trippers.go:444] Response Headers:
I0115 12:54:02.557741    2118 round_trippers.go:447]     Content-Type: application/json
I0115 12:54:02.557747    2118 round_trippers.go:447]     Content-Length: 1218
I0115 12:54:02.557753    2118 round_trippers.go:447]     Date: Tue, 15 Jan 2019 12:54:02 GMT

So I can confirm that the first control plane join takes a bit longer, so it seems that we have yet another cause for this issue and this would match with @masantiago and @fmehrdad descriptions.

I did the kubeadm join with -v10 in order to find what was the request that was blocking. I’ll keep digging to see the root cause of this, but we have 3 different issues causing the same visible problem, maybe it’s time to split this issue 😃

So, two things after discussing with @fabriziopandini:

1. Copying the whole /etc/kubernetes/pki from one machine to another will lead to this problem.

   • When we do a kubeadm join we are only checking if the certificates are present, not checking if the SANs match what we expect. @fabriziopandini is +1 to not only check for the presence of the certificate, but to check also the SANs of it, and if it doesn’t match what we expect, we regenerate them. Priority #0 is etcd, then #1 is apiserver.

2. The lack of image pre-pullling is addressed in issue https://github.com/kubernetes/kubeadm/issues/1341 with a PR in the works here: https://github.com/kubernetes/kubernetes/pull/72870.

As for 1., despite we name the explicit certificates that need copying on the documentation (https://kubernetes.io/docs/setup/independent/high-availability/#steps-for-the-rest-of-the-control-plane-nodes), I think we can expect more people to try to copy /etc/kubernetes/pki directly between machines, basically because it’s handier. If we address the issue of the SAN checking when doing a kubeadm join this wouldn’t be a problem, because the certificates that don’t match what we expect would simply be recreated.

I see this as a temporary solution until we have completely addressed the automatic transmission of secrets between masters when creating an HA cluster with kubeadm. Until we have a proper solution this would make it easier to copy things around between machines.

I’m hitting the same issue regardless of pre-pulling the images. The first master bootstraps successfully whereas the second master fails with unexpected EOF when uploading some configuration. The weirdest part is that when I try adding another master node, it bootstraps successfully!

waffles@kube-master-1:~$ kubectl get nodes
NAME            STATUS   ROLES    AGE   VERSION
kube-master-1   Ready    master   28m   v1.13.1
kube-master-2   Ready    <none>   18m   v1.13.1
kube-master-3   Ready    master   86s   v1.13.1
kube-worker-1   Ready    <none>   26m   v1.13.1
kube-worker-2   Ready    <none>   25m   v1.13.1
kube-worker-3   Ready    <none>   25m   v1.13.1

Apparently, the second master isn’t a master, but the third master has no trouble. The images were pre-pulled in all machines, so I guess pre-pulling doesn’t affect anything.

My kubeadm-config.yaml was the same as the one in docs (only difference is in the pod subnet because I was using Flannel):

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
networking:
  podSubnet: 10.244.0.0/16
apiServer:
  certSANs:
  - "PUBLIC_IP"
controlPlaneEndpoint: "PUBLIC_IP:PORT"

@iverberk Could you confirm this by adding another master after the second one fails?