prometheus-operator: High memory usage on prometheus operator pods

What happened? The cached memory usage of the prometheus operator pods has been gradually increasing over time. Currently it’s ~15GB.

Did you expect to see some different? Expected lower memory usage, considering the number of pods monitored and timeseries ibeing ingested.

How to reproduce it (as minimally and precisely as possible): Deploy prometheus operator with helm chart. Configure autodiscovery for application services. Memory usage starts gradually increasing.

Environment EKS k8s v1.11 Platform version eks.5

• Prometheus Operator version:

Helm chart: prometheus-operator-6.1.1 v0.31.1 Image: quay.io/coreos/prometheus-operator:v0.30.1

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"archive", BuildDate:"1970-01-01T00:00:01Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.10-eks-7f15cc", GitCommit:"7f15ccb4e58f112866f7ddcfebf563f199558488", GitTreeState:"clean", BuildDate:"2019-08-19T17:46:02Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

Terraformed with terraform-eks-module

• Manifests:

N/A

• Prometheus Operator Logs:

Only seeing logs with:

invalid metric type

Anything else we need to know?: Attaching screenshots of prometheus metrics from grafana: