ingress-nginx: Metrics does not count requests on ingresses without host

Hi @jonaz Any news with this issue? It’s very important for our use-case…

@jonaz @aledbf any progress on this? We’re essentially flying blind with our ingress right now because of this bug…

Are there any concrete plans on the timeline for this? We also have the issue, that we basically are blind in terms of nginx metrics, as we give every customer a custom subdomain (so we use a wildcard domain and hence have not metrics). IMHO it would be a good solution, to just record the wildcard host. For example:

We have an ingress for host *.domain.com. If a customer accesses our system via customer1.domain.com, then the nginx could just record this request for *.domain.com. The same for every other xxx.domain.com. All requests handled by the one wildcard ingress go into one bucket. Hence there would be no problem with unbounded cardinality in the metrics and so no DDOS problem. But we could still distinguish requests for the different ingresses.

I would try to help improve this, but I have no experience with Go 😦

Just wanted to thank @jonaz for opening this issue, as this was the reason Grafana wasn’t showing anything for me.

It might be worth mentioning this in the monitoring section of the docs. I’d imagine others might assume a fan-out without a host would generate metrics.

I also cannot see metrics of hosts with wildcard, there is a workaround?

I am also experiencing the same issue not getting nginx_ingress_controller_requests metrics if I don’t define the host e.g.

This doesn’t generate metrics for nginx_ingress_controller_requests

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: service-nginx
  name: observability-es
  namespace: capabilities
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: elasticsearch-es-http
          servicePort: 9200
        path: /

This generate metrics for nginx_ingress_controller_requests

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: service-nginx
  name: observability-es
  namespace: capabilities
spec:
  rules:
  - host: dev.microservices.test.test
    http:
      paths:
      - backend:
          serviceName: elasticsearch-es-http
          servicePort: 9200
        path: /

Would be great if we could get the fix for this.

My solution above would protect against DDOS and also support metrics on all paths+hosts. But no word from maintainers here yet.

Would it make sense to change the code here to export all ingress metrics when per-host metrics are disabled though? We’re losing a lot of metrics that are essential to monitoring this in production because the metrics are just not passed on.

Here’s the flag I’m talking about: https://github.com/kubernetes/ingress-nginx/pull/3594

Here’s the code in question I’m suggesting we change: https://github.com/kubernetes/ingress-nginx/blob/ddedd165b2a457607e70e37d3d7ce613d1aa5307/internal/ingress/metric/collectors/socket.go#L224-L227

I even lose my metrics when I made my host a wildcard. So the host exists it just changed from e.g. “prod.something.com” to “*.something.com”.

@aledbf I also just ran into this issue (today as a matter-of-fact). IMO the linked PR (#4139) is a reasonable way to go. In the case that someone has a wildcard ingress IMO it makes sense to just exclude the host label – since then I can still get metrics based on the ingress name (which is also a fixed cardinality).

Is there anything remaining to get the PR merged in?

Very unlikely to ever happen. I’d recommend you name all of your ingresses (this is the approach I took to get around this issue). It isn’t that hard, especially when using some form of automation to deploy Kubernetes apps (Helm or in my case Terraform). Sticky Session also starts working again when doing this 😃

Long term, traefik might be the way to go, depending on if or not ingress-nginx picks up a new maintainer with aledbf stepping down.

Disclaimer: this isn’t a dig, ingress-nginx is a great piece of work and I highly appreciate the work aledbf has done on it. Just trying to point out a workaround and the fact things are bleak right now but hopefully someone/some others will step up.

@aledbf I also just ran into this issue (today as a matter-of-fact). IMO the linked PR (https://github.com/kubernetes/ingress-nginx/pull/4139) is a reasonable way to go. In the case that someone has a wildcard ingress IMO it makes sense to just exclude the host label – since then I can still get metrics based on the ingress name (which is also a fixed cardinality).

Is there anything remaining to get the PR merged in?

@jonaz @aledbf any progress on this? We’re essentially flying blind with our ingress right now because of this bug…

@hairyhenderson since no reply from @aledbf regarding my implementation suggestion we are migrating to traefik 2.0 instead.