VictoriaMetrics: Potential Memory Leak in VMAgent v1.105 / v1.108
Describe the bug
We observe container memory to increase over time (days) in case of vmagent component. Gathered data to help with investigation, pod is running only vmagent container and we use official vmagent image in version 1.105.
helm-chart: 0.14.7 image-tag: v1.105.0
Gathered pprof, pmap, metrics, cgroup mem-in-use, config (there was a small delay between them):
pmap
rss usage over time
mem over time by container
cgroup mem
chart config (few things redacted due to confidentiality):
image:
tag: v1.105.0
fullnameOverride: vmagent-scrape
replicaCount: 3
statefulSet:
enabled: true
clusterMode: true
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: <<REDACTED>>
name: vmagent-scrape
service:
enabled: true
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '8429'
labels: {}
type: NodePort
ingress:
enabled: false
resources:
limits:
cpu: 400m
memory: 450Mi
requests:
cpu: 240m
memory: 280Mi
ephemeral-storage: 4Gi
rbac:
create: true
pspEnabled: true
podDisruptionBudget:
enabled: true
minAvailable: 40%
selector:
matchLabels:
app.kubernetes.io/instance: vmagent-scrape
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: vmagent-scrape
topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/instance: vmagent-scrape
remoteWrite:
- url: <<REDACTED>>
extraArgs:
remoteWrite.disableOnDiskQueue: true
remoteWrite.dropSamplesOnOverload: true
loggerLevel: WARN
promscrape.streamParse: true
promscrape.suppressScrapeErrors: true
promscrape.suppressDuplicateScrapeTargetErrors: true
loggerErrorsPerSecondLimit: 20
promscrape.discovery.concurrency: 200
promscrape.maxDroppedTargets: 100000
config:
global:
scrape_interval: 30s
scrape_timeout: 10s
scrape_configs:
- job_name: kubernetes-nodes
kubernetes_sd_configs:
- role: node
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
- action: replace
replacement: $1
source_labels:
- __meta_kubernetes_node_label_kubernetes_io_hostname
target_label: hostname
metric_relabel_configs:
- action: labelmap
regex: exported_(.+)
- regex: k8s_io_cloud_provider_aws|failure_domain_beta_kubernetes_io_.*|kubernetes_io_hostname|topology_kubernetes_.*|beta_kubernetes_.*
action: labeldrop
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
source_labels: [__meta_kubernetes_pod_container_init]
regex: true
- action: keep_if_equal
source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels:
[__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- source_labels: [__meta_kubernetes_pod_container_name]
target_label: container
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
- source_labels: [__meta_kubernetes_pod_node_name]
action: replace
target_label: hostname
metric_relabel_configs:
- action: labelmap
regex: exported_(.+)
- source_labels: [__name__]
regex: tf_upbound_io_ProviderConfig.*|aws_upbound_io_ProviderConfig.*
action: drop
- action: labeldrop
regex: label_.*
if: '{app_kubernetes_io_name="x-metrics"}'
- job_name: kubernetes-kubelet
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc.cluster.local:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
- source_labels: [__meta_kubernetes_node_name]
action: replace
target_label: hostname
metric_relabel_configs:
- action: labelmap
regex: exported_(.+)
- job_name: kubernetes-cadvisor
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
- source_labels: [__meta_kubernetes_node_name]
action: replace
target_label: hostname
metric_relabel_configs:
- action: labelmap
regex: exported_(.+)
- regex: k8s_io_cloud_provider_aws|failure_domain_beta_kubernetes_io_.*|kubernetes_io_hostname|topology_kubernetes_.*|beta_kubernetes_.*
action: labeldrop
- job_name: kubernetes-service-endpoints
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_kubectl_kubernetes_io_last_applied_configuration]
action: replace
replacement: ""
target_label: __annotation
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
- __meta_kubernetes_endpoint_hostname
regex: ((.*);(.*))
replacement: $2$3
target_label: hostname
metric_relabel_configs:
- action: labelmap
regex: exported_(.+)
- action: labelmap
regex: node
replacement: hostname
- action: labeldrop
regex: label_.*
if: '{app_kubernetes_io_name="kube-state-metrics"}'
- job_name: vmstorage
metrics_path: /metrics
scheme: http
ec2_sd_configs:
- region: "us-east-1"
port: 8482
filters:
- name: tag:component
values:
- vmstorage
relabel_configs:
- source_labels: [__meta_ec2_availability_zone]
action: replace
target_label: availability_zone
- source_labels: [__meta_ec2_instance_id]
action: replace
target_label: instance_id
- source_labels: [__meta_ec2_instance_type]
action: replace
target_label: instance_type
- source_labels: [__meta_ec2_private_dns_name]
action: replace
target_label: private_dns_name
- source_labels: [__meta_ec2_tag_Name]
action: replace
target_label: name
- source_labels: [__meta_ec2_tag_component]
action: replace
target_label: component
- source_labels: [__meta_ec2_tag_purpose]
action: replace
target_label: purpose
To Reproduce
Run vmagent in that version for few weeks and observe memory grow
Version
vmagent-20241021-132821-tags-v1.105.0-0-g05f6ea621d
Logs
N/A
Screenshots
No response
Used command-line flags
No response
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 19 (5 by maintainers)
We no longer see this issue after deployment of v1.114.0. This issue can be closed.
We are still evaluating this following upgrade to 1.114. Will update here when we have enough data to compare.
@vhajdukd This issue might be related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7146. Could you please share the output of the following metrics?
These will give us insight into potential memory growth related to label compression.