serving: The queue-proxy's memory slowly and constantly increases even if there is no request.

What version of Knative?

0.23.0

Expected Behavior

When there is no request sent to the pod, the memory usage of queue-proxy should be a constant or at least a stable value.

Actual Behavior

When there is no request sent to the pod, the memory usage slowly increases. Specifically, the queue-proxy’s memory is increasing. Don’t know if it is a memory leak. 5261627439879_ pic_hd

Steps to Reproduce the Problem

  1. In config-observability, set metrics.backend-destination: prometheus and metrics.request-metrics-backend-destination: prometheus
  2. Set annotations of prometheus.io for ksvc you want at ksvc.spec.template.metadata.annotation where port is 9091 and path is /metrics
  3. Create ksvc and watch the pod.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (8 by maintainers)

Most upvoted comments

no problem

I recompiled queue-proxy only using golang 1.16 and here is a monitoring statistics for 2 hours. It seems like a little bit better. Don’t know if they are related… comparison This happened to all of my test pods. I will continuously run my tests for another 12 hours.

@skonto No problem

FWIW: I totally expect the leak to be in the prometheus realm. I should’ve spelled that out clearly I guess 😂 .

I can take a look.