vector: Vector-agent hits OOM every hour
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
Vector-agent in our deployment shows constant memory growth till the pod hits OOM. This is happening continuously. I looked at a number of other issues open already on the same problem but it’s not clear how to resolve it. In prod we have a lot more pipelines than shown below though.
The metric vector_component_allocated_bytes shows remap-* components have the most memory allocation and constantly growing.
Configuration
data_dir: /var/lib/vector
expire_metrics_secs: 600
api:
enabled: true
address: 0.0.0.0:8686
playground: true
sources:
host_metric_source:
type: host_metrics
scrape_interval_secs: 60
internal_metric_source:
type: internal_metrics
scrape_interval_secs: 60
ip-node1:
type: prometheus_scrape
endpoints:
- http://<ip>:9100/metrics
scrape_interval_secs: 60
sinks:
datadog_sink:
type: datadog_metrics
inputs:
- remap_pod_*
site: datadoghq.com
prometheus_exporter:
type: prometheus_exporter
inputs:
- internal_metric_source
- host_metric_source
address: 0.0.0.0:9598
vector_sink:
type: vector
inputs:
- tiering.tier1
address: https://<host>:8903
buffer:
type: disk
when_full: drop_newest
max_size: 10737418240
batch:
max_bytes: 2500000
timeout_secs: 15
healthcheck: true
compression: true
headers:
auth-version: "1"
transforms:
remap_node_ip-node1:
type: remap
inputs:
- ip-node1
source: |-
.tags.nodename = "ip-node1"
.tags.service_name = "nodeExporter"
tiering:
type: route
inputs:
- remap_*
route:
tier1: .name == .name
reroute_unmatched: false
Version
0.35.0
Debug Output
No response
Example Data
No response
Additional Context
No response
References
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 4
- Comments: 40 (18 by maintainers)
Agreed. I think we need to verify that the
prometheus_scrapeis dropping requests that time out correctly. It seems like, based on your report, that they may be leaking.Agreed. The
prometheus_scrapesource should not scrape again if a previous scrape is still pending to avoid stacking scrapes.Batches are sent concurrently so it is up to the downstream system to order events by timestamp. You are already sending concurrent requests, it is just using the default of “adaptive concurrency” where it tries to to algorithmically determine the optimal concurrency (which is reported by the
vector_adaptive_concurrency_limithistogram we were looking at previously). I’m suggesting setting it to a static value in case the algorithm isn’t finding the optimal concurrency and you could be pushing it higher. https://vector.dev/docs/about/under-the-hood/networking/arc/ describes the adaptive concurrency mechanism that is used by default.Hmm, that is interesting. It sounds like some requests may have been stacking up in the
prometheus_scapeand not being reclaimed when they timed out, but that would be surprising.I think what is happening here is that the sink cannot send the data fast enough and so it is piling up first in the buffers and then in the scraping source, which I believe will continue scraping on the interval even if it can’t flush the data downstream (this is probably suboptimal; I think the source should stop scraping the previous request hasn’t been flushed downstream yet). To fix this, I think you’ll need to adjust the sink to meet the throughput requirements of your source. Some ideas to do this:
request.concurrencyto a high static numberI got a heap profile and this is how a diff looks
Here - vector gets to 55-65 and dd-metrics gets to 1k before the pod hit OOM.
Interesting, thanks for sharing that graph. It seems likely to me that the issue is that the concurrency limit is never finding a max. You could try to configure a max via
request.adaptive_concurrency.max_concurrency_limit(e.g. https://vector.dev/docs/reference/configuration/sinks/vector/#request.adaptive_concurrency.max_concurrency_limit).