kops: Unable to connect to the server: EOF - Kops rolling update of 1.10.11

1. What kops version are you running? The command kops version, will display this information. 1.10.0

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-04T07:48:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}
Unable to connect to the server: EOF
server had 1.9.6```

**3. What cloud provider are you using?** AWS

**4. What commands did you run?  What is the simplest way to reproduce this issue?**
 `kops rolling-update cluster prod-xxx.k8s.local --state s3://prod-xxx-store --yes`

**5. What happened after the commands executed?**
master node running in AWS but not hittable via kubectl

**6. What did you expect to happen?** get the nodes!

**7. Please provide your cluster manifest. Execute
  `kops get --name my.example.com -o yaml` to display your cluster manifest.
  You may want to remove your cluster name and other sensitive information.**

**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
  Paste the logs into this report, or in a gist and provide the gist link here.**

**9. Anything else do we need to know?**

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 6
Comments: 30 (5 by maintainers)

Most upvoted comments

Hi @ssoroka

Could you describe your problems more specifically? And where are those error message come from?

In most cases I faced when try to rolling-update cluster can be classified into two parts and all solved by update ip address in Route 53:

Master node not available when validate master node by using kubectl or kops, usually cause by the public ip address in api.{$your_domain_name} in Route 53 hosted zones not update properly. In this case, you can fix it by update the ip address in Route 53 manually. The address in api.{$your_domain_name} should be the same as your public address of master node in EC2.
Worker node can’t reach to master node. This could be tricky because there are so many reasons can cause those problems. You can try to check the ip address of api.internal.{$your_domain_name}, etcd.internal.{$your_domain_name} and etcd-event.internal.{$your_domain_name} is as same as private ip in master node.

cychiang on Dec 6, 2018

It was kops version 1.10. It was a development tier cluster with private topology. I’ve postponed production until I know more about this.

mpagnucco on Dec 11, 2018

Same error for me upgrading from 1.9.10 to 1.10.11, switched to TCP healthcheck on ELB manually and now all is good. But SSL does not work.

mpagnucco on Dec 10, 2018

Part of my problem is updating from 1.8.x -> 1.13.0. 😕 I downgraded as advised, and did a rolling update and it seems to have resolved most of my problems.

If anyone’s curious, I switched kubernetes-cli versions like so:

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.10.11/bin/darwin/amd64/kubectl
chmod 555 kubectl
sudo mv kubectl /usr/local/bin/kubectl

Thanks!

ssoroka on Dec 6, 2018

I’m seeing an issue that smells the same: the API ELB has SSL:443 “ping” healthchecks that fail with a fresh cluster created with ‘kops create’. Switching to TCP:443 makes the healthchecks pass. This is only an issue in us-east-2; us-east-1 works fine.

markine on Dec 6, 2018

Just wanted to add that I just ran into the same issue following the guide from https://github.com/kubernetes/kops/blob/master/docs/aws.md

Seems that the master node is failing ELB health checks and never getting put into service.

Update: I’m seeing the following line in journalctl -u kops-configuration.service

Dec 06 15:56:42 ip-172-20-41-62 nodeup[702]: I1206 15:56:42.065540     702 s3fs.go:219] Reading file "s3://<redacted>/<redacted>.k8s.local/cluster.spec: AccessDenied: Access Denied

So, yeah, that probably has something to do with it for me. Investigating…

Update 2: Yeah, I put my S3 bucket in US East and we’re working fine now.

scjudd on Dec 6, 2018

@ssoroka your attached cluster.txt indicates you have set kubernetesVersion: 1.13.0 Kops does not support 1.13.0, and probably won’t for an extended period of time, or until a corporation invests in paying for someone to work full-time on upgrades.

I would roll back to 1.10.11 which is the most recent known-good combination of kops and Kubernetes. Make sure to also roll back your versions of the image on the InstanceGroup

BrianChristie on Dec 6, 2018