VictoriaMetrics: Wrong vmagent query with dockerswarm_sd_configs filters
Describe the bug I use docker swarm and try to migrate to VM from Prometheus (as full replacement) with dockerswarm_sd_configs filters, but vmagent stop working if i specify any filters. Without filters vmagent is working, but prometheus working with filters too.
To Reproduce My docker-compose.yml file
version: '3.8'
vmagent:
image: victoriametrics/vmagent:v1.85.3
expose:
- 8429
volumes:
- monitoring_vmagent_data:/vmagentdata
- /var/run/docker.sock:/var/run/docker.sock:ro
configs:
- source: monitoring_prometheus_config
target: /etc/prometheus/prometheus.yml
command:
- "--promscrape.config=/etc/prometheus/prometheus.yml"
- "--promscrape.config.strictParse=false"
- "--remoteWrite.url=http://victoriametrics:8428/api/v1/write"
networks:
- monitoring_network
restart: always
My prometheus.yml file
- job_name: 'swarm_task_exporter'
scrape_interval: 15s
dockerswarm_sd_configs:
- host: unix:///var/run/docker.sock
role: tasks
filters:
- name: "desired-state"
values: ["running"]
Expected behavior
If i connect to docker container and using curl create request, everything is ok
curl --silent -XGET --unix-socket /run/docker.sock http://localhost/services?filters%5Bdesired-state%5D%5B0%5D=running
But vmagent generate wrong request if i specify any filters
Logs
error VictoriaMetrics/lib/promscrape/config.go:1007 skipping dockerswarm_sd_config targets for job_name "swarm_task_exporter" because of error: cannot query dockerswarm api for services: unexpected status code returned from "http:///services?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%7D": 400; expecting 200; response body: "{\"message\":\"Invalid filter 'desired-state'\"}\n"
The logs show that there is a request for
http:///services?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%7D
If you decode the parameters, you get ‘filters’ like this: {"desired-state":{"running":true}}
The correct parameters should be ‘filters’: {"desired-state": "running"}
Version Swarm 20.10.22 vmagent v1.85.3
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 28 (9 by maintainers)
Thanks a lot, it helped!
@fannyfan414 This container was added into discovery with 2 different networks. Since relabeling will produce two identical sets of labels vmagent will write this warning every time. I can suggest you to keep one of targets since you only need to scrape targets from monitoring network:
Hi @fannyfan414, thank you for the report.
This issue was related to the label discovery for the tasks and have been fixed at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3632
In the meantime, you can build vmagent from sources by using instruction here Or use pre-built docker image:
victoriametrics/vmagent:heads-fix-dockerswarm-discovery-filters-0-gf1c98e190