kops: Cert errors creating cluster

Thanks for submitting an issue! Please fill in as much of the template below as you can.

------------- BUG REPORT TEMPLATE --------------------

What kops version are you running? The command kops version, will display this information.

on a feature branch

What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

1.8.x

What cloud provider are you using? aws
What commands did you run? What is the simplest way to reproduce this issue?

KOPS=kops

$KOPS create -f $KOPS_FILE
$KOPS create secret --name $CLUSTER_NAME sshpublickey admin -i ~/.ssh/id_rsa.pub
$KOPS update cluster --phase assets $CLUSTER_NAME --yes
$KOPS update cluster --phase network $CLUSTER_NAME --yes
$KOPS update cluster --phase security $CLUSTER_NAME --yes
$KOPS update cluster --phase cluster $CLUSTER_NAME --yes

What happened after the commands executed?

instances are created, but I am getting cert errors.

What did you expect to happen?

Cluster should be created

Please provide your cluster manifest. Execute
Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: I1227 21:58:38.478038     726 s3fs.go:213] Listing objects in S3 bucket "aws.k8spro.com" with prefix "us-west-2.aws.k8spro.com/pki/issued/ca/"
Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: I1227 21:58:38.574952     726 s3fs.go:239] Listed files in s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca: [s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca/6504343549136392374237039546.crt s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca/keyset.yaml]
Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: I1227 21:58:38.574981     726 s3fs.go:176] Reading file "s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca/6504343549136392374237039546.crt"
Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: I1227 21:58:38.663684     726 certificate.go:103] Parsing pem block: "CERTIFICATE"
Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: I1227 21:58:38.663901     726 s3fs.go:176] Reading file "s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca/keyset.yaml"
Dec 27 21:58:38 ip-172-60-0-196 nodeup[726]: W1227 21:58:38.751769     726 main.go:141] got error running nodeup (will retry in 30s): error building loader: error fetching CA certificate from keystore: error in 'FindCert' attempting to load cert "ca": error loading certificate "s3://aws.k8spro.com/us-west-2.aws.k8spro.com/pki/issued/ca/keyset.yaml": could not parse certificate

Anything else do we need to know?

I am trying to re-create.

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 2
Comments: 49 (22 by maintainers)

Commits related to this issue

Ignore keyset.yaml files These are written by later versions of kops (1.9), but if we try to parse them as a certificate/key we will error out. We're going to do a 1.8.1 release which will just igno... — committed to justinsb/kops by justinsb 6 years ago
Ignore keyset.yaml files These are written by later versions of kops (1.9), but if we try to parse them as a certificate/key we will error out. We're going to do a 1.8.1 release which will just igno... — committed to justinsb/kops by justinsb 6 years ago

Most upvoted comments

@mmacfadden shortest answer I can give (for me on OSX): Requirements:

golang 1.9.2 from homebrew
docker
write access to an S3 bucket (export AWS_ACCESS_KEY=<your key> AWS_SECRET_KEY=<your secret>)

go get k8s.io/kops
cd go/src/k8s.io/kops
git checkout master # (at least, that's what I'm building from)
make upload S3_BUCKET=s3://<your AWS S3 bucket> # (does not need to be a public bucket)
make upload S3_BUCKET=s3://<your AWS S3 bucket>
cat .build/upload/kops/1.8.0/linux/amd64/nodeup.sha1 # this sha will be needed later

Build time takes me 45 minutes (though, I comment out windows build targets in the Makefile to speed things along a bit.) Once the binaries are built, make will push the artifacts to your S3 bucket. During kops create you will need to then export the new S3 location and sha for nodeup:

export NODEUP_URL="https://s3.amazonaws.com/<your AWS S3 bucket>/kops/1.8.0/linux/amd64/nodeup"  NODEUP_HASH="<NODEUP.SHA1 value>"
kops create cluster --name=newcluster

@francoran mmacfadden is correct, to identify that it’s a keyset.yaml bootstrap issue, you have to ssh to the node that isn’t joining, and look at the /var/log/daemon.log for the events that KashifSaadat reported.

StephanX on Jan 24, 2018

Any pointers on how we can get kops to generate the keyset.yaml files in S3 (if it’s detected as missing) during a kops update cluster.. command? I was trying to follow through the logic and got a little lost!

Doing so should cover the user journey on upgrading an existing cluster.

KashifSaadat on Jan 8, 2018

@chrislovecnm I don’t think this is OSX go bug, I got hit by this on Linux with kops built from master with both go 1.9.2 and go 1.8.5.

nebril on Jan 15, 2018

The 1.8.1 release and master have been fixed to use the new nodeup in 1.8.1. This was part of # 4375, and allows the keyset.yaml to be ignored.

chrislovecnm on Feb 13, 2018

I also got a cluster up and running by uploading a new version of nodeup to S3. However, I think we will hold off on moving forward until this issue is resolve. It was suggested that an 1.9.0 alpha release might solve the problem by referencing the more recent nodeup. Do we still think this is the case? If so, is there any approximate timeline for a 1.9 alpha just for planning purposes? Thanks for the great work!

mmacfadden on Feb 2, 2018

Just following up - I eventually had to rebuild the cluster with the new make upload binary and the new nodeup with hash.

Thanks all for your help!

kaijparo on Jan 31, 2018

i am guessing you need a newer version of nodeup.

make upload with the s3 bucket name will help with that.

Let me know how it goes. I am concerned about upgrades with this as well.

chrislovecnm on Jan 30, 2018

@KashifSaadat I got past this by using the same golang version for kops and nodeup.

chrislovecnm on Dec 28, 2017