cluster-api: Inconsistent provider behaviour with API server ports
What steps did you take and what happened: [A clear and concise description on how to REPRODUCE the bug.]
A common refrain I have heard from customers:
“As a cluster operator, I may be working in a multi-cloud environment, and also have a directive to always have https endpoints exposed on 443. In the Kubernetes context, this means clients should only ever connect to the API server on port 443.”
In Cluster API, we have a number of places where the port can be changed:
-
cluster.spec.clusterNetwork.APIServerPortThis should be used by infrastructure providers to configure the load balancer port. -
kcp.spec.kubeadmConfigSpec.initConfiguration.localAPIEndpoint.bindPortThis will change the port that kube-apiserver will listen on each individual control plane node, and is used by kubeadm
In AWS, CAPA will use cluster.spec.clusterNetwork.APIServerPort to set the LB port and assume kube-apiserver continues to listen on 6443.
In Azure, CAPZ assumes that if cluster.spec.clusterNetwork.APIServerPort isn’t 6443, that kube-apiserver is also listening on the different port. This is kind of necessary due to the way the Azure LBs work and the workarounds required for hairpin NAT.
Importantly, in bare-metal / on-prem environments like vSphere/Tinkerbell, we might be using kube-vip to make our control plane highly available. In this circumstance, there is no LB, and cluster.spec.clusterNetwork.APIServerPort is ignored.
What did you expect to happen:
One consistent place.
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
Enforcing any particular implied behaviour may break one or more providers as well as clusters on upgrade, so should not be done within a v1beta1 release IMO.
Also, not suggesting we change the default port, but make it easier to change the port and have a defined behaviour.
Environment:
- Cluster-api version:
- Minikube/KIND version:
- Kubernetes version: (use
kubectl version): - OS (e.g. from
/etc/os-release):
/kind bug [One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 20 (19 by maintainers)
Ideally the users need to be able to configure the local apiServer port through a single field. Today we can’t really make the various
bindPortin the init and join as part of the bootstrappers contract due to the fact that they’re nested into APIs and structs that are kubeadm specific.we probably also need to disambiguate between
cluster.spec.clusterNetwork.APIServerPortandcluster.spec.ControlPlaneEndpoint.PortAn idea could be:
cluster.spec.clusterNetwork.APIServerPorttocluster.spec.ControlPlaneEndpoint.Portand ensure provider reads from it to figure the LB portcluster.spec.clusterNetwork.APIServerPortcluster.spec.clusterNetwork.LocalAPIServerPortI don’t think this needs to behave this way due to any particular Azure infra constraint. We should be able to solve for this. I’ll create an issue to expose the API server LB port while keeping the API server backend port 6443, as done in CAPA.