kubernetes: aws cloud provider tries to get ELB for services that do not have Type=LoadBalancer

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

Note: Please file issues for subcomponents under the appropriate repo

Component	Repo
cloud_provider/aws.go	kubernetes/aws.go

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): disable elb

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE-REQUEST

Kubernetes version (use kubectl version): 1.6.4

Environment:

Cloud provider or hardware configuration: AWS
OS (e.g. from /etc/os-release):

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1353.8.0
VERSION_ID=1353.8.0
BUILD_ID=2017-05-30-2322
PRETTY_NAME="Container Linux by CoreOS 1353.8.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"

Kernel (e.g. uname -a): Linux ip-10-14-6-231.eu-west-1.compute.internal 4.9.24-coreos #1 SMP Tue May 30 23:12:01 UTC 2017 x86_64 Intel® Xeon® CPU E5-2670 v2 @ 2.50GHz GenuineIntel GNU/Linux
Install tools: cloud-config/custom
Others:

What happened:

I want to use the AWS cloud provider for node labelling and EBS volumes but I do not want the ELB’s being created for every kubernetes service. When I create ANY service with ClusterIP the kube-controller tries to create ELB load-balancers for them. e.g.

E0619 16:25:57.416401       1 servicecontroller.go:779] Failed to process service. Retrying in 5m0s: Error getting LB for service sandbox/quotation: AccessDenied: User: arn:aws:sts::659222956801:assumed-role/kubernetes-server-sand1/i-0851b6157302ce7c9 is not authorized to perform: elasticloadbalancing:DescribeLoadBalancers
	status code: 403, request id: f9535a57-550b-11e7-9325-cd3b2a462e89
E0619 16:25:57.421229       1 servicecontroller.go:779] Failed to process service. Retrying in 5m0s: Error getting LB for service sandbox/quote-composer: AccessDenied: User: arn:aws:sts::659222956801:assumed-role/kubernetes-server-sand1/i-0851b6157302ce7c9 is not authorized to perform: elasticloadbalancing:DescribeLoadBalancers
	status code: 403, request id: f953f698-550b-11e7-9325-cd3b2a462e89
E0619 16:25:57.426800       1 servicecontroller.go:779] Failed to process service. Retrying in 5m0s: Error getting LB for service kube-system/kube-dns: AccessDenied: User: arn:aws:sts::659222956801:assumed-role/kubernetes-server-sand1/i-0851b6157302ce7c9 is not authorized to perform: elasticloadbalancing:DescribeLoadBalancers
	status code: 403, request id: f954b9e9-550b-11e7-9325-cd3b2a462e89
E0619 16:25:57.431282       1 servicecontroller.go:779] Failed to process service. Retrying in 5m0s: Error getting LB for service sandbox/depositapi: AccessDenied: User: arn:aws:sts::659222956801:assumed-role/kubernetes-server-sand1/i-0851b6157302ce7c9 is not authorized to perform: elasticloadbalancing:DescribeLoadBalancers
	status code: 403, request id: f955a44a-550b-11e7-9325-cd3b2a462e89
E0619 16:25:57.435969       1 servicecontroller.go:779] Failed to process service. Retrying in 5m0s: Error getting LB for service sandbox/kafkastreamtest: AccessDenied: User: arn:aws:sts::659222956801:assumed-role/kubernetes-server-sand1/i-0851b6157302ce7c9 is not authorized to perform: elasticloadbalancing:DescribeLoadBalancers
	status code: 403, request id: f956408b-550b-11e7-9325-cd3b2a462e89
E0619 16:25:57.441247       1 servicecontroller.go:779] Failed to process service. Retrying in 5m0s: Error getting LB for service sandbox/p2ps-console: AccessDenied: User: arn:aws:sts::659222956801:assumed-role/kubernetes-server-sand1/i-0851b6157302ce7c9 is not authorized to perform: elasticloadbalancing:DescribeLoadBalancers
	status code: 403, request id: f95703dc-550b-11e7-9325-cd3b2a462e89

I prevented the controller from being able to access the ELB related API calls on amazon but I am concerned by the repeated attempts and the possibility of AWS disabling the API through these repeated errors.

I don’t understand why it is trying to do anything here seen as I am not creating services of type load-balancer. Does it always try to list the elb’s even if the service is of type ClusterIP?

What you expected to happen:

Different aspects of the cloud-provider could be configurable via feature flags: e.g. elb-creation: false ebs-provisioning: true node-labelling: true auto-scaling: false

How to reproduce it (as minimally and precisely as possible):

Present in any cluster.

Anything else we need to know:

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 44 (31 by maintainers)

Commits related to this issue

Revert "Upgraded k8s version to 1.9" This reverts commit 52b00d4103ca0345cc857a0cab71d47c7e2a4643. There is still a bug in 1.9 regarding services that don't use a loadbalancer. https://github.com/k... — committed to edevil/kubernetes-deployment by edevil 6 years ago
Allow controller nodes to DescribeLoadBalancers This shouldn't be required but is because of a bug[1]. This still ends up producing an API error, but with less noise in the kube-controller logs and w... — committed to alphagov/gsp by samcrang 5 years ago
Allow controller nodes to DescribeLoadBalancers This shouldn't be required but is because of a bug[1]. This still ends up producing an API error, but with less noise in the kube-controller logs and w... — committed to alphagov/gsp by samcrang 5 years ago

Most upvoted comments

Why are we doing it this way here: https://github.com/kubernetes/kubernetes/blob/v1.14.5/pkg/controller/service/service_controller.go#L284-L285

When we have previous state and can just check to see if the previous service type was of type LoadBalancer?

This example makes a lot of sense: https://github.com/kubernetes/kubernetes/blob/v1.14.5/pkg/controller/service/service_controller.go#L444-L447

stockhausenj on Oct 11, 2019

I’m still having this issue with kubernetes 1.14.5 on EC2. My digging thus far suggests that with the implementation of service-controller in 1.14.5 that for every update to a service not of type LoadBalancer the cloud provider’s GetLoadBalancer method is invoked, which in the case of the aws cloud provider always attempts to describe a load balancer who’s name is derived from the service name by the default implementation of CloudProvider.DefaultLoadBalancerName.

I’ve been able to mitigate the warnings by granting elasticloadbalancing:DescribeLoadBalancers to the instance role of my nodes, but this should not be a required permission when I’m not using LoadBalancer services.

marcus-jameson on Oct 11, 2019

This bug needs an appropriate name, please

I would rename it, but neither did I open it, nor am I a maintainer, so no rights. I think it should be:

“aws cloud provider tries to get ELB for services that do not have Type=LoadBalancer”

deitch on Dec 13, 2017

Been looking for this. I just enabled --cloud-provider=aws solely to get EBS persistent volumes. All of the sudden every service of any Type gives those errors. I don’t mind expanding the IAM perms so that it can describe the load balancers, but why is it possibly trying to create them in the first place?

deitch on Aug 29, 2017