kubernetes: Masters should not be excluded from service load balancers
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug
What happened: I’m running a single node cluster on AWS EC2, ie. scheduling work on the master as well. Now that I tried to add an ELB for my service, the EC2 instance is not associated with the ELB. The ELB gets created but there are 0 EC2 instances associated with it. I’m using ELB for SSL termination, if you wonder why I’d like to load balance a single node cluster.
The service controller logs this message:
I0628 17:54:48.853175 1 event.go:221] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx", UID:"59980933-7afc-11e8-9a8c-0621b993d602", APIVersion:"v1", ResourceVersion:"2052", FieldPath:""}): type: 'Warning' reason: 'UnAvailableLoadBalancer' There are no available nodes for LoadBalancer service default/nginx
What you expected to happen: The EC2 instance is associated with the ELB.
How to reproduce it (as minimally and precisely as possible):
- Boot an EC2 instance with correct IAM permissions
- Install a single node cluster using
kubeadm
andcloud-provider=aws
- Untaint the node
- Schedule eg. nginx
- Create an external load balancer for nginx
- View the ELB in AWS console and note the
Status: 0 of 0 instances in service
Anything else we need to know?:
The reason for this seems to be the node-role.kubernetes.io/master
label. This blocks associating a load balancer with the node. On the other hand this changed what is included, because includeNodeFromNodeList
did not check if a node is a master. I’m not sure what would be correct fix. I could try submit a PR, if you guide me how this should behave. Is my scenario even a supported one?
I think this bug should be reproducible on other clouds as well.
Environment:
- Kubernetes version (use
kubectl version
):
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration: AWS
- OS (e.g. from /etc/os-release):
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
- Kernel (e.g.
uname -a
):
Linux ip-x.eu-central-1.compute.internal 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux
- Install tools:
kubeadm
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 51 (38 by maintainers)
Commits related to this issue
- docs/user/*/install_upi: Drop compute replicas zeroing We grew this in c22d042fe1 (docs/user/aws/install_upi: Add 'sed' call to zero compute replicas, 2019-05-02, #1649) to set the stage for changing... — committed to wking/openshift-installer by wking 5 years ago
- docs/user/*/install_upi: We grew replicas-zeroing in c22d042 (docs/user/aws/install_upi: Add 'sed' call to zero compute replicas, 2019-05-02, #1649) to set the stage for changing the 'replicas: 0' se... — committed to wking/openshift-installer by wking 5 years ago
- docs/user/*/install_upi: Explicitly set control-plane unschedulable We grew replicas-zeroing in c22d042 (docs/user/aws/install_upi: Add 'sed' call to zero compute replicas, 2019-05-02, #1649) to set ... — committed to wking/openshift-installer by wking 5 years ago
- docs/user/*/install_upi: Explicitly set control-plane unschedulable We grew replicas-zeroing in c22d042 (docs/user/aws/install_upi: Add 'sed' call to zero compute replicas, 2019-05-02, #1649) to set ... — committed to wking/openshift-installer by wking 5 years ago
- docs/user/*/install_upi: Explicitly set control-plane unschedulable We grew replicas-zeroing in c22d042 (docs/user/aws/install_upi: Add 'sed' call to zero compute replicas, 2019-05-02, #1649) to set ... — committed to alaypatel07/installer by wking 5 years ago
- docs/user/*/install_upi: Explicitly set control-plane unschedulable We grew replicas-zeroing in c22d042 (docs/user/aws/install_upi: Add 'sed' call to zero compute replicas, 2019-05-02, #1649) to set ... — committed to jhixson74/installer by wking 5 years ago
- Bug 2096376: Remove limitation on single node deployments There should no longer be any issues running router pods on control plane nodes (i.e. https://github.com/kubernetes/kubernetes/issues/65618 w... — committed to shiftstack/installer by stephenfin 2 years ago
- Bug 2096376: [openstack] Remove limitation on single node deployments There should no longer be any issues running router pods on control plane nodes (i.e. https://github.com/kubernetes/kubernetes/is... — committed to shiftstack/installer by stephenfin 2 years ago
Thanks @jiatongw! We’ll need a KEP as per https://github.com/kubernetes/enhancements/issues/724 and there’s more context from SIG network here https://groups.google.com/forum/#!topic/kubernetes-sig-network/erULVdDkLNw
The implementation of Services and LB needs to be distinct from the API around it. Not every implementation uses the same techniques or even uses kube-proxy. This is why I have historically been against such API changes, but I admit we have not come up with a workable compromise.
In other words please don’t do dev work on this because there’s not a clear design.
I’ll pick up this soon @andrewsykim