aws-load-balancer-controller: Internal load balancer throws curl: (52) Empty reply from server

I exposed a k8s service using internal load balancer using annotation kubectl patch svc argo-server -n argo -p '{"metadata": {"annotations": {"service.beta.kubernetes.io/aws-load-balancer-internal": "true"}}}'

It worked for past 6 months with EKS k8s 1.17 but stopped working with EKS K8s upgrade to 1.19 and giving me (52) Empty reply from server for curl.

I tried service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0" and service.beta.kubernetes.io/aws-load-balancer-scheme: "internal" both gives same error.

I see external ip is exposed with service.beta.kubernetes.io/aws-load-balancer-internal: "true" but unable to access it

kubectl get svc argo-server -n argo                                                                                                          
NAME          TYPE           CLUSTER-IP       EXTERNAL-IP                                                                        PORT(S)          AGE
argo-server   LoadBalancer   172.20.135.242   internal-ae99b272671214b38bba978529ccc81f-1242061336.us-east-1.elb.amazonaws.com   2746:32105/TCP   11h
❯ curl http://internal-ae99b272671214b38bba978529ccc81f-1242061336.us-east-1.elb.amazonaws.com:2746/
curl: (52) Empty reply from server

P.S - EKS k8s version - 1.19

Any suggestion?

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 20

Most upvoted comments

@mikeinton Glad to hear you were able to narrow down to the root cause of your issue.

Yes, there is indeed limits on number of NLBs per cluster due to worker security group rule limitation. We are aware of this and work is underway to address this limitation, stay tuned 😄.