ingress-nginx: Reoccurrence of Service does not have any active Endpoint [when it actually does]

What happened: The ingress controller reported that the “Service <service-name> does not have any active Endpoint” when in fact the service did have active endpoints.

I was able to verify the service was active by execing into the nginx pod and curling the health check endpoint of the service.

The only way I was able to recover was to reinstall the helm chart.

What you expected to happen:

The service to be added to ingress controller

NGINX Ingress controller version:

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.6.4
  Build:         69e8833858fb6bda12a44990f1d5eaa7b13f4b75
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version): Server Version: version.Info{Major:“1”, Minor:“25+”, GitVersion:“v1.25.6-eks-48e63af”, GitCommit:“9f22d4ae876173884749c0701f01340879ab3f95”, GitTreeState:“clean”, BuildDate:“2023-01-24T19:19:02Z”, GoVersion:“go1.19.5”, Compiler:“gc”, Platform:“linux/amd64”}

Environment: AWS EKS

Server Version: version.Info{Major:“1”, Minor:“25+”, GitVersion:“v1.25.6-eks-48e63af”, GitCommit:“9f22d4ae876173884749c0701f01340879ab3f95”, GitTreeState:“clean”, BuildDate:“2023-01-24T19:19:02Z”, GoVersion:“go1.19.5”, Compiler:“gc”, Platform:“linux/amd64”}

How was the ingress-nginx-controller installed: nginx nginx 1 2023-05-06 16:52:09.643618809 +0000 UTC deployed ingress-nginx-4.5.2 1.6.4

Values:

  ingressClassResource:
    default: true
  service:
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-type: nlb

How to reproduce this issue: Unknown. There was a single replica of the pod, and it was deployed for 42 days before exhibiting this problem.

However, others have recently reported this issue in https://github.com/kubernetes/ingress-nginx/issues/6135.

Anything else we need to know:

The problem was previously reported in https://github.com/kubernetes/ingress-nginx/issues/6135, but the defect was closed.

About this issue

Original URL
State: open
Created a year ago
Reactions: 2
Comments: 76 (29 by maintainers)

Most upvoted comments

@matthewbrumpton thanks for the update. @rdb0101 hope this comment from @matthewbrumpton helps.

I need data and logs to co-relate the problem and solution.

longwuyuan on Jun 12, 2023

@rdb0101 I think its important to establish if the controller has a problem because of which the routing to app fails. If your HTTP/HTTPS request to your frontend breaks, it does not directly mean that the controller is causing the problem.

So first we need the kubectl describe ingress .... output when your HTTP/HTTPS request to frontend fails. Then we need to match the timestamp when you sent request to the timestamp of log messages related to your HTTP/HTTPS request. Then we need to check if pod had problems like networking or cpu or memory temporarily. Or some other proof that without any other problems in the cluster, only the ingress-nginx controller was the cause of broken routing and error message.

longwuyuan on Jun 3, 2023

Installed without Azure annotations, with same error:

I0601 17:37:38.119381 7 controller.go:207] “Backend successfully reloaded” I0601 17:37:38.119590 7 controller.go:218] “Initial sync, sleeping for 1 second” I0601 17:37:38.119733 7 event.go:285] Event(v1.ObjectReference{Kind:“Pod”, Namespace:“bmde-ne-nginx”, Name:“nginx-ingress-ingress-nginx-controller-6679b95c85-tzwk9”, UID:“d039bd8f-5901-492d-9767-7485515bf49d”, APIVersion:“v1”, ResourceVersion:“12177”, FieldPath:“”}): type: ‘Normal’ reason: ‘RELOAD’ NGINX reload triggered due to a change in configuration W0601 17:37:41.937126 7 controller.go:1152] Service “httpbin/httpbin” does not have any active Endpoint. W0601 17:37:46.303769 7 controller.go:1152] Service “httpbin/httpbin” does not have any active Endpoint. W0601 17:37:49.640181 7 controller.go:1152] Service “httpbin/httpbin” does not have any active Endpoint. W0601 17:37:52.974091 7 controller.go:1152] Service “httpbin/httpbin” does not have any active Endpoint.

helm install nginx-ingress ingress-nginx/ingress-nginx --version 4.6.1 –create-namespace --namespace bmde-ne-nginx –set controller.replicaCount=1 --set controller.metrics.enabled=true –set controller.nodeSelector.“kubernetes.io/os”=linux ` –set controller.admissionWebhooks.patch.nodeSelector.“kubernetes.io/os”=linux

matthewbrumpton on Jun 1, 2023