VictoriaMetrics: Potential Memory Leak in VMAgent v1.105 / v1.108

Describe the bug

We observe container memory to increase over time (days) in case of vmagent component. Gathered data to help with investigation, pod is running only vmagent container and we use official vmagent image in version 1.105.

helm-chart: 0.14.7 image-tag: v1.105.0

Gathered pprof, pmap, metrics, cgroup mem-in-use, config (there was a small delay between them): Image

pmap Image

rss usage over time Image

mem over time by container Image

cgroup mem Image

chart config (few things redacted due to confidentiality):

image:
  tag: v1.105.0
fullnameOverride: vmagent-scrape

replicaCount: 3

statefulSet:
  enabled: true
  clusterMode: true

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: <<REDACTED>>
  name: vmagent-scrape

service:
  enabled: true
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: '8429'
  labels: {}
  type: NodePort

ingress:
  enabled: false

resources:
  limits:
    cpu: 400m
    memory: 450Mi
  requests:
    cpu: 240m
    memory: 280Mi
    ephemeral-storage: 4Gi

rbac:
  create: true
  pspEnabled: true

podDisruptionBudget:
  enabled: true
  minAvailable: 40%
  selector:
    matchLabels:
      app.kubernetes.io/instance: vmagent-scrape

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app.kubernetes.io/instance: vmagent-scrape
          topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/instance: vmagent-scrape

remoteWrite:
  - url: <<REDACTED>>

extraArgs:
  remoteWrite.disableOnDiskQueue: true
  remoteWrite.dropSamplesOnOverload: true
  loggerLevel: WARN
  promscrape.streamParse: true
  promscrape.suppressScrapeErrors: true
  promscrape.suppressDuplicateScrapeTargetErrors: true
  loggerErrorsPerSecondLimit: 20
  promscrape.discovery.concurrency: 200
  promscrape.maxDroppedTargets: 100000

config:
  global:
    scrape_interval: 30s
    scrape_timeout: 10s
  scrape_configs:
    - job_name: kubernetes-nodes
      kubernetes_sd_configs:
        - role: node
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics
        - action: replace
          replacement: $1
          source_labels:
            - __meta_kubernetes_node_label_kubernetes_io_hostname
          target_label: hostname

      metric_relabel_configs:
        - action: labelmap
          regex: exported_(.+)
        - regex: k8s_io_cloud_provider_aws|failure_domain_beta_kubernetes_io_.*|kubernetes_io_hostname|topology_kubernetes_.*|beta_kubernetes_.*
          action: labeldrop

    - job_name: kubernetes-pods
      kubernetes_sd_configs:
        - role: pod

      relabel_configs:
        - action: drop
          source_labels: [__meta_kubernetes_pod_container_init]
          regex: true
        - action: keep_if_equal
          source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port, __meta_kubernetes_pod_container_port_number]
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels:
            [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_pod_name]
          target_label: pod
        - source_labels: [__meta_kubernetes_pod_container_name]
          target_label: container
        - source_labels: [__meta_kubernetes_namespace]
          target_label: namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
        - source_labels: [__meta_kubernetes_pod_node_name]
          action: replace
          target_label: hostname

      metric_relabel_configs:
        - action: labelmap
          regex: exported_(.+)
        - source_labels: [__name__]
          regex: tf_upbound_io_ProviderConfig.*|aws_upbound_io_ProviderConfig.*
          action: drop
        - action: labeldrop
          regex: label_.*
          if: '{app_kubernetes_io_name="x-metrics"}'

    - job_name: kubernetes-kubelet
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
        - role: node
      relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc.cluster.local:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics
        - source_labels: [__meta_kubernetes_node_name]
          action: replace
          target_label: hostname

      metric_relabel_configs:
        - action: labelmap
          regex: exported_(.+)

    - job_name: kubernetes-cadvisor

      scheme: https

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
        - role: node

      relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
        - source_labels: [__meta_kubernetes_node_name]
          action: replace
          target_label: hostname

      metric_relabel_configs:
        - action: labelmap
          regex: exported_(.+)
        - regex: k8s_io_cloud_provider_aws|failure_domain_beta_kubernetes_io_.*|kubernetes_io_hostname|topology_kubernetes_.*|beta_kubernetes_.*
          action: labeldrop

    - job_name: kubernetes-service-endpoints

      kubernetes_sd_configs:
        - role: endpoints

      relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_kubectl_kubernetes_io_last_applied_configuration]
          action: replace
          replacement: ""
          target_label: __annotation
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_node_name
            - __meta_kubernetes_endpoint_hostname
          regex: ((.*);(.*))
          replacement: $2$3
          target_label: hostname

      metric_relabel_configs:
        - action: labelmap
          regex: exported_(.+)
        - action: labelmap
          regex: node
          replacement: hostname
        - action: labeldrop
          regex: label_.*
          if: '{app_kubernetes_io_name="kube-state-metrics"}'

    - job_name: vmstorage

      metrics_path: /metrics
      scheme: http

      ec2_sd_configs:
        - region: "us-east-1"
          port: 8482
          filters:
            - name: tag:component
              values:
                - vmstorage

      relabel_configs:
        - source_labels: [__meta_ec2_availability_zone]
          action: replace
          target_label: availability_zone
        - source_labels: [__meta_ec2_instance_id]
          action: replace
          target_label: instance_id
        - source_labels: [__meta_ec2_instance_type]
          action: replace
          target_label: instance_type
        - source_labels: [__meta_ec2_private_dns_name]
          action: replace
          target_label: private_dns_name
        - source_labels: [__meta_ec2_tag_Name]
          action: replace
          target_label: name
        - source_labels: [__meta_ec2_tag_component]
          action: replace
          target_label: component
        - source_labels: [__meta_ec2_tag_purpose]
          action: replace
          target_label: purpose

To Reproduce

Run vmagent in that version for few weeks and observe memory grow

Version

vmagent-20241021-132821-tags-v1.105.0-0-g05f6ea621d

Logs

N/A

Screenshots

No response

Used command-line flags

No response

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 19 (5 by maintainers)

Most upvoted comments

We no longer see this issue after deployment of v1.114.0. This issue can be closed.

We are still evaluating this following upgrade to 1.114. Will update here when we have enough data to compare.

@vhajdukd This issue might be related to https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7146. Could you please share the output of the following metrics?

sum(vm_streamaggr_labels_compressor_size_bytes) by (job,instance)
sum(vm_streamaggr_labels_compressor_items_count) by (job,instance)

These will give us insight into potential memory growth related to label compression.