ingress-nginx: nginx_ingress_controller_requests is missing for Ingress that has had no requests
NGINX Ingress controller version: 0.30.0
Kubernetes version (use kubectl version
): 1.15
Environment:
- EKS k8s cluster
What happened:
The nginx_ingress_controller_requests
metric was missing for Ingresses that have had 0 requests. I queried nginx_ingress_controller_requests == 0
for our prometheus metrics and found no time series.
What you expected to happen:
I expected ingresses that have had no requests sent to them to have a nginx_ingress_controller_requests
metric with a count of 0 rather than not being present.
How to reproduce it:
- I deployed a test Ingress object (very basic config) with the host name set
- I waited for the Ingress controller to show the host and address, I’ve seen other bugs raised that state this metric won’t show if the host isn’t set but I don’t think that applies here
$ kubectl --context <CONTEXT> -n <NAMESPACE> get ing test
NAME HOSTS ADDRESS PORTS AGE
test <DOMAIN> <LB_ADDRESS> <PORTS> 7h40m
- I looked for the metric in prometheus for this ingress but couldn’t find it
nginx_ingress_controller_requests{ingress="test"}
- I also
exec
’d to a pod on the cluster and hit the NGINX controller’s/metrics
endpoint to check if our prometheus stack was filtering out the metric, but it didn’t appear there either
Anything else we need to know:
Please let me know if you need any other info from me, thanks for your time 😃
/kind bug
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 4
- Comments: 27 (11 by maintainers)
The issue seems to be a bit old, but I’m presenting my thoughts here to provide some closure to others who might see this.
I feel the problem at hand is that the metric itself could have multiple labels, and those labels could have multiple values. This makes it harder to initialize
nginx_ingress_controller_requests
for all combination of labels and values (maybe, some values might not even be known until the request arrives). Something like https://github.com/prometheus/client_golang/issues/190 might be helpful but even that might have problems with dynamic values of labels.Hi @strongjz sorry for the delayed response. I upgrade to 0.47.0 and repeated my above test (created new debug ingress, waited for host and address to appear, checked if metric appeared) and could still not see any
nginx_ingress_controller_requests
metrics from the/metrics
endpoint of any NGINX pod in the cluster. When I curled my new debug ingress and checked the NGINX pods/metrics
endpoint again it showed me annginx_ingress_controller_requests
metric for that ingress. Please let me know if you need any more info from me 😃