vector: Possible Memory Leak for K8s logs

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Description of the environment

There is vector DaemonSet in the k8s cluster, each Pod collects k8s logs

Expectation

After upgrading Vector from 0.20.0 to 0.21.2 there is not much difference in memory usage

Actuall Outcome

On some nodes, Vector memory usage keeps linearly increasing causing them to be OOM-Killed by k8s. The same issue persists in all versions including 0.24.1

Memory usage graphs

Updated version at ~11:20 All pods image

Problematic Pods image

Configuration

# Vector's API for introspection
    [api]
      enabled = true
      address = "0.0.0.0:yyyy"

    # Read Kubernetes logs from files
    [sources.kube_logs_1]
      type = "kubernetes_logs"
      exclude_paths_glob_patterns = ["xxx"]
      annotation_fields.container_name = "container_name"
      annotation_fields.container_image = "container_image"
      annotation_fields.pod_ip = "pod_ip"
      annotation_fields.pod_name = "pod_name"
      annotation_fields.pod_namespace = "namespace_name"
      annotation_fields.pod_node_name = "pod_node_name"
      annotation_fields.pod_uid = "pod_uid"

    # Transform Kube logs and remove unused fields
    [transforms.kube_logs_remapped]
      type = "remap"
      inputs = ["kube_logs_1"]
      source = '''
      .full_message = .message
      .message = "pod_stderr_stdout"
      .source = .pod_name
      .node_name = .pod_node_name
      .facility = .container_name
      del(.file)
      del(.pod_node_name)
      del(.kubernetes)
      '''

    # Filter OUT k8s-event-logger events
    [transforms.kube_logs_filtered]
      type = "filter"
      inputs = ["kube_logs_remapped"]
      condition = '.container_name != "k8s-event-logger"'

    # Filter IN k8s-event-logger events
    [transforms.only_k8s_event_logs]
      type = "filter"
      inputs = ["kube_logs_1"]
      condition = '.container_name == "k8s-event-logger"'

    # Transform k8s event logs from JSON
    [transforms.k8s_events_remapped]
      type = "remap"
      inputs = ["only_k8s_event_logs"]
      source = '''
      if is_nullish(.message) == true {
        .message = "No log message attached for k8s-event-logger"
        .facility = "k8s-event-logger"
        abort
      }
      . = parse_json!(.message)
      .facility = "k8s-event-logger"
      .type = .type
      .reason = .reason
      .namespace_name = .metadata.namespace
      .node_name = .source.host
      .source = .involvedObject.name
      .object_kind = .involvedObject.kind
      .object_name = .involvedObject.name
      del(.metadata)
      del(.involvedObject)
      del(.count)
      del(.firstTimestamp)
      del(.lastTimestamp)
      del(.reportingComponent)
      del(.reportingInstance)
      del(.eventTime)
      '''

    # Forward logs to vector-aggregator service
    [sinks.vector_aggr]
      type = "vector"
      inputs = ["k8s_events_remapped", "kube_logs_filtered"]
      address = "x.x.x.x:yyyy"

Version

>=0.21.0-distroless-libc

Debug Output

No response

Example Data

No response

Additional Context

  • Vector is deployed as DaemonSet in k8s
  • It is collecting k8s logs

References

No response

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 18
  • Comments: 32 (10 by maintainers)

Most upvoted comments

Yes, I can first check the subversions, and then the nightly builds between them. I’ll divide periods into halves to speed up the process, though this still is going take some time.

We’re seeing this behavior for vector 0.29.1 on EKS 1.24. We are load testing vector by running it as a daemonset against pods writing files via the json-file log driver for containerd. There’s 32 pods writing 100 logs/s each, so we’re pushing 3200 logs/s through the vector daemon on the node. Log sizes vary from 1.25K-128K, with mostly smaller writes. We see unbounded memory growth consistently even when setting a blackhole sink (contrary to what @arch-xtof experienced). Based on log throughput and EKS’ rotation size I’d imagine that we’re rotating through a ton of files, so agree that Vector is probably hoarding open file handles.

Setup

Chart.yaml

apiVersion: v2
name: service
type: application
version: 1.0.0
dependencies:
  - name: vector
    version: 0.21.1
    repository: "https://helm.vector.dev"

Values.yaml

podLabels:
  tags.datadoghq.com/version: 0.29.1
resources:
  limits:
    cpu: 3000m
    memory: 1Gi
  requests:
    cpu: 3000m
    memory: 1Gi

Vector Config

The redacted remaps below are just adding additional static tags.

data_dir: /vector-data-dir
expire_metrics_secs: 60

api:
  enabled: true
  address: 127.0.0.1:8686
  playground: false

sources:
  internal_logs:
    type: internal_logs
  internal_metrics:
    type: internal_metrics
    scrape_interval_secs: 30
    tags:
      host_key: host
  k8s_logs:
    type: kubernetes_logs
    glob_minimum_cooldown_ms: 15000

transforms:
  vector_tagging:
    type: remap
    inputs:
      - internal_metrics
    source: |
      .tags.region = get_env_var!("AWS_REGION")
      ...(REDACTED)...
  log_tagging:
    type: remap
    inputs:
      - internal_logs
    source: |
      .service = "vector-daemonset"
      ...(REDACTED)...
sinks:
  datadog_metrics:
    type: datadog_metrics
    inputs:
      - vector_tagging
    default_api_key: ${DATADOG_API_KEY}
  datadog_logs:
    type: datadog_logs
    inputs:
      - log_tagging
    default_api_key: ${DATADOG_API_KEY}
  null_out:
    type: blackhole
    inputs:
      - k8s_logs

Results

image

Thanks @spencergilbert!

I will update to 0.24.2 when it becames available and check the behavior.

OK, so I talked with @jszwedko and it turns out I’ve completely mixed up when fixes are/were released. I think this should be fixed in 0.24.2 (cutting this week) between https://github.com/vectordotdev/vector/issues/14394 and https://github.com/vectordotdev/vector/pull/14423. This should be testable on a nightly image without any custom builds required.

It is possible to entirely disable the generation of these metrics to see if this solve the issue?

Right now that would require a custom build, I believe. CC @jszwedko and @bruceg as another report of metrics not expiring issue.

I have a simular issue of memory leak, opened an issue a couple of months ago, #11025, but the issue persists after upgrading to the last version.

I’m running 0.24.1 and have expire_metrics_secs set to 30, my source are files and my sinks are kafka, the vector memory keeps increasing.

It is possible to entirely disable the generation of these metrics to see if this solve the issue?

The Y-axis graph refers to memory usage in MB. Here is the metric of internal_metric_cardinality: Top 5 Cardinality image

All image

For some reason, cardinality is still increasing all the time, but I don’t see a strong correlation between cardinality and memory usage.