VictoriaMetrics: Wrong vmagent query with dockerswarm_sd_configs filters

Describe the bug I use docker swarm and try to migrate to VM from Prometheus (as full replacement) with dockerswarm_sd_configs filters, but vmagent stop working if i specify any filters. Without filters vmagent is working, but prometheus working with filters too.

To Reproduce My docker-compose.yml file

version: '3.8'
  vmagent:
    image: victoriametrics/vmagent:v1.85.3
    expose:
      - 8429
    volumes:
      - monitoring_vmagent_data:/vmagentdata
      - /var/run/docker.sock:/var/run/docker.sock:ro
    configs:
      - source: monitoring_prometheus_config
        target: /etc/prometheus/prometheus.yml
    command:
      - "--promscrape.config=/etc/prometheus/prometheus.yml"
      - "--promscrape.config.strictParse=false"
      - "--remoteWrite.url=http://victoriametrics:8428/api/v1/write"
    networks:
      - monitoring_network
    restart: always

My prometheus.yml file

  - job_name: 'swarm_task_exporter'
    scrape_interval: 15s
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: tasks
        filters:
          - name: "desired-state"
            values: ["running"]

Expected behavior If i connect to docker container and using curl create request, everything is ok curl --silent -XGET --unix-socket /run/docker.sock http://localhost/services?filters%5Bdesired-state%5D%5B0%5D=running But vmagent generate wrong request if i specify any filters

Logs error VictoriaMetrics/lib/promscrape/config.go:1007 skipping dockerswarm_sd_config targets for job_name "swarm_task_exporter" because of error: cannot query dockerswarm api for services: unexpected status code returned from "http:///services?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%7D": 400; expecting 200; response body: "{\"message\":\"Invalid filter 'desired-state'\"}\n"

The logs show that there is a request for

http:///services?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%7D

If you decode the parameters, you get ‘filters’ like this: {"desired-state":{"running":true}} The correct parameters should be ‘filters’: {"desired-state": "running"}

Version Swarm 20.10.22 vmagent v1.85.3

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 28 (9 by maintainers)

Most upvoted comments

@fannyfan414Этот контейнер был добавлен в обнаружение с двумя разными сетями. Поскольку перемаркировка создаст два идентичных набора меток, vmagent будет каждый раз выводить это предупреждение. Я могу предложить вам сохранить одну из целей, так как вам нужно только очищать цели из сети мониторинга:

- action: keep
  if: "{__meta_dockerswarm_network_name='monitoring_network'}"

Thanks a lot, it helped!

@fannyfan414 This container was added into discovery with 2 different networks. Since relabeling will produce two identical sets of labels vmagent will write this warning every time. I can suggest you to keep one of targets since you only need to scrape targets from monitoring network:

- action: keep
  if: "{__meta_dockerswarm_network_name='monitoring_network'}"

Hi @fannyfan414, thank you for the report.

This issue was related to the label discovery for the tasks and have been fixed at https://github.com/VictoriaMetrics/VictoriaMetrics/pull/3632

In the meantime, you can build vmagent from sources by using instruction here Or use pre-built docker image: victoriametrics/vmagent:heads-fix-dockerswarm-discovery-filters-0-gf1c98e190