VictoriaMetrics: Takes too long time to start the newly added scrapers
Describe the bug The VictoriaMetrics takes too long time to start the newly added scrapers if there are scrapers that should be removed but still trying to scrape data from a target but does not respond. Please see below scenario.
- There are 100 scrapers.
- I need to stop existing 100 scrapers but need to add 1 scraper.
- By current implementation, the VictoriaMetrics should wait for the stop for all the scrapers.
- But if the targets are not responding, a scraper waits for the response until expiration of timeout.
- So, it the timeout is 5 seconds, the VictoriaMetrics should wait for total 500 seconds to add 1 scraper.
So, I want to fix this issue by myself and contribute the code to this community. Please guide me how to do it. Any procedures, governance of this community or anything. 😃
To Reproduce Please refer to above scenario.
Expected behavior The VictoriaMetrics should not wait for expiration of timeout but stop the scrapers immediately.
Logs None.
Screenshots None.
Version
# /usr/sbin/victoriametrics --version
victoria-metrics-20221026-123908-pmm-6401-v1.77.1
Used command-line flags
–promscrape.config=/etc/victoriametrics-promscrape.yml --retentionPeriod=4d --storageDataPath=/srv/victoriametrics/data --httpListenAddr=127.0.0.1:9090 --search.disableCache=true --search.maxQueryLen=1MB --search.latencyOffset=5s --search.maxUniqueTimeseries=100000000 --search.maxSamplesPerQuery=1500000000 --search.maxQueueDuration=30s --search.logSlowQueryDuration=30s --search.maxQueryDuration=90s --promscrape.streamParse=true --prometheusDataPath=/srv/prometheus/data --http.pathPrefix=/prometheus --envflag.enable --envflag.prefix=VM_
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (2 by maintainers)
@zekker6 OK. I see. I will wait for a fix for the default mode as well. Feel free to let me know once a fix is ready. I could validate your fixes. Thanks!
FYI, the fix, which improves config reload speed for the disabled stream parsing mode, has been included in v1.88.0 - see this issue for details. Closing this issue as fixed then.
@chadr123 , please re-open this issue or create new one if the latest release of
vmagentstill experience the original issue you posted here.FYI, configuration reload should take less time in
vmagentstarting from v1.87.1 when stream parsing mode is enabled. The fix for the disabled stream parsing mode will be provided in the future releases.@zekker6 It seems that once I ran the Victoriametrics with
-promscrape.streamParse="true"option, it works as expected, I mean that without any delay, like below but does not without the-promscrape.streamParse="true"option.