ingress-nginx: nginx_ingress_controller_requests is missing for Ingress that has had no requests

NGINX Ingress controller version: 0.30.0

Kubernetes version (use kubectl version): 1.15

Environment:

  • EKS k8s cluster

What happened:

The nginx_ingress_controller_requests metric was missing for Ingresses that have had 0 requests. I queried nginx_ingress_controller_requests == 0 for our prometheus metrics and found no time series.

What you expected to happen:

I expected ingresses that have had no requests sent to them to have a nginx_ingress_controller_requests metric with a count of 0 rather than not being present.

How to reproduce it:

  • I deployed a test Ingress object (very basic config) with the host name set
  • I waited for the Ingress controller to show the host and address, I’ve seen other bugs raised that state this metric won’t show if the host isn’t set but I don’t think that applies here
$ kubectl --context <CONTEXT> -n <NAMESPACE> get ing test
NAME       HOSTS          ADDRESS                 PORTS     AGE
test       <DOMAIN>  <LB_ADDRESS>      <PORTS>   7h40m
  • I looked for the metric in prometheus for this ingress but couldn’t find it nginx_ingress_controller_requests{ingress="test"}
  • I also exec’d to a pod on the cluster and hit the NGINX controller’s /metrics endpoint to check if our prometheus stack was filtering out the metric, but it didn’t appear there either

Anything else we need to know:

Please let me know if you need any other info from me, thanks for your time 😃

/kind bug

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 4
  • Comments: 27 (11 by maintainers)

Most upvoted comments

The issue seems to be a bit old, but I’m presenting my thoughts here to provide some closure to others who might see this.

I feel the problem at hand is that the metric itself could have multiple labels, and those labels could have multiple values. This makes it harder to initialize nginx_ingress_controller_requests for all combination of labels and values (maybe, some values might not even be known until the request arrives). Something like https://github.com/prometheus/client_golang/issues/190 might be helpful but even that might have problems with dynamic values of labels.

Hi @strongjz sorry for the delayed response. I upgrade to 0.47.0 and repeated my above test (created new debug ingress, waited for host and address to appear, checked if metric appeared) and could still not see any nginx_ingress_controller_requests metrics from the /metrics endpoint of any NGINX pod in the cluster. When I curled my new debug ingress and checked the NGINX pods /metrics endpoint again it showed me an nginx_ingress_controller_requests metric for that ingress. Please let me know if you need any more info from me 😃