vector: Vector-agent hits OOM every hour

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Vector-agent in our deployment shows constant memory growth till the pod hits OOM. This is happening continuously. I looked at a number of other issues open already on the same problem but it’s not clear how to resolve it. In prod we have a lot more pipelines than shown below though.

The metric vector_component_allocated_bytes shows remap-* components have the most memory allocation and constantly growing.

Configuration

data_dir: /var/lib/vector
expire_metrics_secs: 600
api:
  enabled: true
  address: 0.0.0.0:8686
  playground: true
sources:
  host_metric_source:
    type: host_metrics
    scrape_interval_secs: 60
  internal_metric_source:
    type: internal_metrics
    scrape_interval_secs: 60
  ip-node1:
    type: prometheus_scrape
    endpoints:
    - http://<ip>:9100/metrics
    scrape_interval_secs: 60
sinks:
  datadog_sink:
    type: datadog_metrics
    inputs:
    - remap_pod_*
    site: datadoghq.com
  prometheus_exporter:
    type: prometheus_exporter
    inputs:
    - internal_metric_source
    - host_metric_source
    address: 0.0.0.0:9598
  vector_sink:
    type: vector
    inputs:
    - tiering.tier1
    address: https://<host>:8903
    buffer:
      type: disk
      when_full: drop_newest
      max_size: 10737418240
    batch:
      max_bytes: 2500000
      timeout_secs: 15
    healthcheck: true
    compression: true
    headers:
      auth-version: "1"
transforms:
  remap_node_ip-node1:
    type: remap
    inputs:
    - ip-node1
    source: |-
      .tags.nodename = "ip-node1"
      .tags.service_name = "nodeExporter"
  tiering:
    type: route
    inputs:
    - remap_*
    route:
      tier1: .name == .name
    reroute_unmatched: false

Version

0.35.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 4
  • Comments: 40 (18 by maintainers)

Most upvoted comments

@jszwedko You’re right the egress volume’s not keeping up with the ingress causes the agent to hit OOM. However for block it seems to be a bug as it would cause data-loss for events which have not been flushed to disk. Right?

I think there are two bugs here:

  1. prometheus_scape causes OOM when there are requests timeouts.

Agreed. I think we need to verify that the prometheus_scrape is dropping requests that time out correctly. It seems like, based on your report, that they may be leaking.

  1. Again, the prometheus_scape would hit OOM with block buffer when ingress is greater than egress volume.

How could we track these two issues? Otherwise this thread could be closed.

Agreed. The prometheus_scrape source should not scrape again if a previous scrape is still pending to avoid stacking scrapes.

@pront I will open two separate issues for these bugs. One follow up though - when I set request.concurrency how does the vector-agent guarantee order of the events. Say one batch is send to the vector-server but fails and is retried but another batch with newer events is processed first by the vector-server. How will these two batches be ordered in the end?

Batches are sent concurrently so it is up to the downstream system to order events by timestamp. You are already sending concurrent requests, it is just using the default of “adaptive concurrency” where it tries to to algorithmically determine the optimal concurrency (which is reported by the vector_adaptive_concurrency_limit histogram we were looking at previously). I’m suggesting setting it to a static value in case the algorithm isn’t finding the optimal concurrency and you could be pushing it higher. https://vector.dev/docs/about/under-the-hood/networking/arc/ describes the adaptive concurrency mechanism that is used by default.

@jszwedko Looks like there a correlation with receive errors. I noticed some timeouts in the logs so adjusted the scrape-timeout which not only resolved the errors but also the memory growth.

Hmm, that is interesting. It sounds like some requests may have been stacking up in the prometheus_scape and not being reclaimed when they timed out, but that would be surprising.

thinking it would slow down the ingestion once the buffer was filled up. Unfortunately this again lead to the memory growth.

I think what is happening here is that the sink cannot send the data fast enough and so it is piling up first in the buffers and then in the scraping source, which I believe will continue scraping on the interval even if it can’t flush the data downstream (this is probably suboptimal; I think the source should stop scraping the previous request hasn’t been flushed downstream yet). To fix this, I think you’ll need to adjust the sink to meet the throughput requirements of your source. Some ideas to do this:

  • Set request.concurrency to a high static number
  • If that isn’t enough and you are bottlenecked on another resource, increase that resource (e.g. CPU, memory, or network bandwidth)

I got a heap profile and this is how a diff looks

Showing nodes accounting for 13.73GB, 94.84% of 14.47GB total
Dropped 190 nodes (cum <= 0.07GB)
      flat  flat%   sum%        cum   cum%
   13.73GB 94.84% 94.84%    13.73GB 94.84%  prof_backtrace_impl
         0     0% 94.84%     0.27GB  1.90%  <(FnA,FnB,FnC) as nom::sequence::Tuple<Input,(A,B,C),Error>>::parse
         0     0% 94.84%     0.22GB  1.52%  <F as nom::internal::Parser<I,O,E>>::parse
         0     0% 94.84%    13.11GB 90.58%  <T as futures_util::fns::FnMut1<A>>::call_mut
         0     0% 94.84%     0.33GB  2.30%  <alloc::collections::btree::map::BTreeMap<K,V,A> as core::clone::Clone>::clone::clone_subtree
         0     0% 94.84%     0.37GB  2.55%  <alloc::string::String as core::clone::Clone>::clone
         0     0% 94.84%     0.35GB  2.44%  <alloc::vec::Vec<T,A> as core::clone::Clone>::clone
         0     0% 94.84%    13.17GB 91.00%  <core::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
         0     0% 94.84%    13.12GB 90.68%  <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
         0     0% 94.84%    13.12GB 90.68%  <futures_util::stream::stream::Flatten<St> as futures_core::stream::Stream>::poll_next
         0     0% 94.84%    13.10GB 90.52%  <futures_util::stream::stream::filter_map::FilterMap<St,Fut,F> as futures_core::stream::Stream>::poll_next
         0     0% 94.84%    13.12GB 90.68%  <futures_util::stream::stream::flatten_unordered::FlattenUnorderedWithFlowController<St,Fc> as futures_core::stream::Stream>::poll_next
         0     0% 94.84%    13.12GB 90.68%  <futures_util::stream::stream::flatten_unordered::PollStreamFut<St> as core::future::future::Future>::poll
         0     0% 94.84%    13.73GB 94.85%  <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
         0     0% 94.84%     0.18GB  1.27%  <tower::limit::rate::service::RateLimit<S> as tower_service::Service<Request>>::call
         0     0% 94.84%     0.18GB  1.27%  <tower::retry::Retry<P,S> as tower_service::Service<Request>>::call
         0     0% 94.84%    13.73GB 94.83%  <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
         0     0% 94.84%     0.18GB  1.27%  <vector::sinks::util::retries::FibonacciRetryPolicy<L> as tower::retry::policy::Policy<Req,Res,alloc::boxed::Box<dyn core::error::Error+core::marker::Send+core::marker::Sync>>>::clone_request
         0     0% 94.84%     0.19GB  1.30%  <vector::sinks::vector::sink::VectorSink<S> as vector_core::sink::StreamSink<vector_core::event::Event>>::run::{{closure}}
         0     0% 94.84%    13.11GB 90.57%  <vector::sources::prometheus::scrape::PrometheusScrapeContext as vector::sources::util::http_client::HttpClientContext>::on_response
         0     0% 94.84%     0.19GB  1.30%  <vector::transforms::remap::Remap<Runner> as vector_core::transform::SyncTransform>::transform
         0     0% 94.84%     2.87GB 19.81%  <vector_core::event::metadata::EventMetadata as core::default::Default>::default
         0     0% 94.84%     1.60GB 11.03%  <vector_core::event::metadata::Inner as core::default::Default>::default
         0     0% 94.84%     0.16GB  1.11%  <vector_core::event::metric::Metric as core::clone::Clone>::clone
         0     0% 94.84%     7.04GB 48.67%  <vector_core::event::metric::tags::MetricTags as core::convert::From<alloc::collections::btree::map::BTreeMap<alloc::string::String,alloc::string::String>>>::from
         0     0% 94.84%     0.18GB  1.24%  <vector_core::event::vrl_target::VrlTarget as vrl::compiler::target::Target>::target_insert
         0     0% 94.84%     0.17GB  1.19%  <vector_core::sink::EventStream<T> as vector_core::sink::StreamSink<vector_core::event::array::EventArray>>::run::{{closure}}
         0     0% 94.84%     0.18GB  1.24%  <vrl::compiler::expression::Expr as vrl::compiler::expression::Expression>::resolve
         0     0% 94.84%     0.18GB  1.24%  <vrl::compiler::expression::assignment::Assignment as vrl::compiler::expression::Expression>::resolve
         0     0% 94.84%    13.73GB 94.84%  [libc.so.6]
         0     0% 94.84%    11.05GB 76.35%  _rjem_je_malloc_default
         0     0% 94.84%    13.73GB 94.84%  _rjem_je_prof_backtrace
         0     0% 94.84%    13.73GB 94.84%  _rjem_je_prof_tctx_create
         0     0% 94.84%     0.13GB  0.89%  alloc::collections::btree::map::BTreeMap<K,V,A>::insert
         0     0% 94.84%     7.17GB 49.55%  alloc::collections::btree::map::entry::VacantEntry<K,V,A>::insert
         0     0% 94.84%     0.13GB  0.89%  alloc::collections::btree::node::Handle<alloc::collections::btree::node::NodeRef<alloc::collections::btree::node::marker::Mut,K,V,alloc::collections::btree::node::marker::Leaf>,alloc::collections::btree::node::marker::Edge>::insert_recursing
         0     0% 94.84%     2.67GB 18.41%  alloc::raw_vec::RawVec<T,A>::grow_one
         0     0% 94.84%     0.24GB  1.65%  alloc::raw_vec::RawVec<T,A>::reserve::do_reserve_and_handle
         0     0% 94.84%     2.90GB 20.06%  alloc::raw_vec::finish_grow
         0     0% 94.84%    13.73GB 94.85%  core::ops::function::FnOnce::call_once{{vtable.shim}}
         0     0% 94.84%     2.68GB 18.49%  do_rallocx
         0     0% 94.84%    13.13GB 90.73%  futures_util::stream::stream::StreamExt::poll_next_unpin
         0     0% 94.84%     0.27GB  1.90%  prometheus_parser::line::Line::parse
         0     0% 94.84%     0.27GB  1.90%  prometheus_parser::parse_text
         0     0% 94.84%    13.73GB 94.84%  std::sys::backtrace::_rust_begin_short_backtrace
         0     0% 94.84%    13.73GB 94.85%  std::sys::pal::unix::thread::Thread::new::thread_start
         0     0% 94.84%    13.73GB 94.85%  tokio::runtime::blocking::pool::Inner::run
         0     0% 94.84%    13.73GB 94.85%  tokio::runtime::context::runtime::enter_runtime
         0     0% 94.84%    13.73GB 94.85%  tokio::runtime::scheduler::multi_thread::worker::Context::run
         0     0% 94.84%    13.73GB 94.85%  tokio::runtime::scheduler::multi_thread::worker::Context::run_task
         0     0% 94.84%    13.73GB 94.85%  tokio::runtime::scheduler::multi_thread::worker::run
         0     0% 94.84%    13.73GB 94.85%  tokio::runtime::task::core::Core<T,S>::poll
         0     0% 94.84%    13.73GB 94.85%  tokio::runtime::task::harness::Harness<T,S>::poll
         0     0% 94.84%    13.18GB 91.06%  vector::source_sender::SourceSender::send_event_stream::{{closure}}
         0     0% 94.84%    12.83GB 88.67%  vector::sources::prometheus::parser::reparse_groups
         0     0% 94.84%    13.18GB 91.06%  vector::sources::util::http_client::call::{{closure}}
         0     0% 94.84%     0.17GB  1.19%  vector::topology::builder::Builder::build_sinks::{{closure}}::{{closure}}
         0     0% 94.84%    13.18GB 91.04%  vector::topology::builder::Builder::build_sources::{{closure}}::{{closure}}

Here - vector gets to 55-65 and dd-metrics gets to 1k before the pod hit OOM. Screenshot 2024-10-30 at 3 10 35 PM

Interesting, thanks for sharing that graph. It seems likely to me that the issue is that the concurrency limit is never finding a max. You could try to configure a max via request.adaptive_concurrency.max_concurrency_limit (e.g. https://vector.dev/docs/reference/configuration/sinks/vector/#request.adaptive_concurrency.max_concurrency_limit).