agones: agones-ping-udp-service brings up incompatible load balancer in EKS

What happened:

When installing Agones in an EKS cluster, the agones-ping-udp-service fails to create with the following error:

Warning CreatingLoadBalancerFailed 1s (x4 over 37s) service-controller Error creating load balancer (will retry): failed to ensure load balancer for service agones-system/agones-ping-udp-service: Only TCP LoadBalancer is supported for AWS ELB

What you expected to happen: The service should have created successfully.

How to reproduce it (as minimally and precisely as possible): Build an EKS cluster, follow installation instructions for Agones.

Anything else we need to know?: Yes, we already know the fix! The service needs the following annotation: annotations: service.beta.kubernetes.io/aws-load-balancer-type: nlb

Environment:

Agones version: 1.0.0
Kubernetes version (use kubectl version): 1.12
Cloud provider or hardware configuration: AWS/EKS
Install method (yaml/helm): YAML, but I checked the Helm chart and it would have the same issue
Troubleshooting guide log(s): n/a
Others:

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 28 (12 by maintainers)

Most upvoted comments

Yeah, also that’s totally a separate thing that affects more Pod creation for GameServers (although, very interesting! 😄 )

Question though: Is this a documentation issue?

If we added some kind of AWS installation docs that said what agones.ping.udp.annotations and agones.ping.http.annotations should be on installation, would that solve the issue?

markmandel on Nov 18, 2022

@markmandel Yes, the following annotations needs to be added if we have to replace Classic Load balancers with Network Load Balancers. NLB supports UDP s

Can someone who clearly knows EKS way better than I do file a PR for the docs? That would be aces.

Sounds like you’ve got it worked out @vara-bonthu ?

markmandel on Jun 21, 2021

I consider using pkg/cloudproduct since the NLB Health Checks must be aware of the Service nodePort and this isn’t available at install time using Helm/YAML.

After typing this, I realize it might be overkill if MixedProtocolLBService is going to be supported in EKS (and other k8s integrations for AWS). Maybe just updating the YAML is best so the UDP ping service at least starts.

bostrt on Nov 18, 2022

service.beta.kubernetes.io/aws-load-balancer-type does bring up a valid NLB with a target group, but the nodes in the target group fail healthchecks as NLBs do not support UDP healthchecks and would fail TCP healthchecks on the traffic port. LoadBalancer services w/ mixed UDP and TCP ports are not currently supported, but I was able to get this working by doing some manual setup of the Agones ping service and setting the following annotations on the UDP ping service:

      "service.beta.kubernetes.io/aws-load-balancer-type"                 = "nlb"
      "service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol" = "HTTP"
      "service.beta.kubernetes.io/aws-load-balancer-healthcheck-path"     = "/"
      "service.beta.kubernetes.io/aws-load-balancer-healthcheck-port"     = {NodePortForHTTPPingService}
      "service.beta.kubernetes.io/aws-load-balancer-target-node-labels"   = "agones.dev/agones-system=true"

This will be cleaner in the future when MixedProtocolLBService is suported in EKS.

ajjohnston on May 2, 2022

The upstream fix for this has landed in k8s master, so will be in 1.19, which should be available in EKS sometime in 2021.

TBBle on Jun 25, 2020

@aLekSer based on your output, it looks like the service is still trying to build without the annotation. You need to completely destroy the service and rebuild it for that to work. When I tested this it was in fact on 1.12.

You’re getting the same error message that indicates that the load balancer was built sans annotation: “Only TCP LoadBalancer is supported for AWS ELB.” There’s a brief AWS doc on the annotation here.

structurefall on Nov 4, 2019