kops: Private DNS + private topology on AWS: problems regarding the certificate
Hi there,
I created a new cluster on AWS:
- setting it to use a private zone on Route 53
- setting it to belong to a private subnet (
--topology private)
As I’m using a private zone on Route 53, my laptop cannot resolve the server name (https://api.cluster.k8s) set on kubecfg.
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: *****
server: https://api.cluster.k8s
[...]
Looking at Route 53 I noticed that https://api.cluster.k8s is a alias to the load balancer api-cluster-k8s-177050000.us-east-1.elb.amazonaws.com. Therefore I changed kubecfg to use this load balancer endpoint instead of the server name. When I do this change, whenever I run kubectl I get an error regarding the certificate:
Unable to connect to the server: x509: certificate is valid for api.internal.cluster.k8s, api.cluster.k8s, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not api-cluster-k8s-177050000.us-east-1.elb.amazonaws.com
So I need to use insecure-skip-tls-verify=true to get it working.
Do you guys know how to overcome this issue? Is it possible to tell kops to generate the certificate without constraining domains, for instance?
Thanks in advance!
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 3
- Comments: 17 (4 by maintainers)
Hey @thaniyarasu, thanks for replying!
I wish I could use a private zone instead of a public one. Everything works fine on the snippet you provided, but names could be resolved from anywhere on the internet, even if it makes sense only inside a VPC - several of its entries are maped to private ips (nodes within the cluster). I wish I could use a private DNS without having problem with the certificate. It could be done either by inputting my own certificate or specifying I don’t want kops to constrain it to any domain, for instance. I have no idea if its possible to do so.
Stumbled on the same issue today as I was setting up a cluster. There’s a small trick I found here at the very bottom, which basically tells you to create records in your public zone for the ELBs created by kops (bastion and api).
This only works if you have a public and private route53 zone with the same domain. And in that case it works fine. But it probably won’t work if you have an existing VPC with an internal domain that doesn’t match the public one, which is probably the most likely scenario (I use internal domains that are different from the public ones for all my VPCs…).
Ideally I would like to be able to tell kops to either:
Hmm, only things I now can come up with to use private DNS are,
What an user of kops can do:
8.8.8.8so that your private domain is resolvable from your laptop. <- this is what I’m doingIF kops should resolve this: 3. kops needs to add an ELB’s domain name when creating SSL certificates and then make it possible to specify it in k8s config. 4. Or we would add a cluster parameter so that an user can pass any domains for SSL certificate. (A problem is the user doesn’t have ELB’s domain name until a cluster is created though)
P.S. Not sure everyone does know this, but when creating AWS internal ELB, the DNS name of an internal load balancer is publicly resolvable to the private IP addresses of the nodes. http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-internal-load-balancers.html This means AWS already reveals part of our using private ips. (I assume this is the same for AWS RDS’s endpoint as well) So if you have a control of domain names on your company, I don’t think it’s totally a bad idea to use publicly resolvable domain name for your AWS private resources, considering AWS already does it 😅