agent: loki.source.docker error: could not set up a wait request to the Docker client

What’s wrong?

I have a very basic Docker log scraping config with the agent running in flow mode. The logs seem to be transmitted normally, however I get constant error/warning messages from the grafana-agent complaining about the inability to read container logs. The error messages are produced and repeated for all targeted containers on a regular basis.

Steps to reproduce

Simply run AGENT_MODE=flow grafana-agent --server.http.listen-addr=0.0.0.0:12345 run /etc/grafana-agent.river with the provided config on the affected host.

Agent is running as user grafana-agent who is a member of the following groups: grafana-agent adm systemd-journal docker

System information

Linux 4.4.0-92-generic x86_64

Software version

Grafana Agent v0.34.3

Configuration

logging {
  level  = "warn"
  format = "logfmt"
}

prometheus.scrape "default" {
	targets    = [{"__address__" = "localhost:12345"}]
	forward_to = [prometheus.remote_write.prom.receiver]
}

prometheus.exporter.unix {
	include_exporter_metrics = true
	disable_collectors       = ["mdadm"]
}

prometheus.remote_write "prom" { }

discovery.docker "linux" {
	host = "unix:///var/run/docker.sock"
	filter {
		name = "status"
		values = ["running"]
	}
}

loki.source.docker "default" {
	host    = "unix:///var/run/docker.sock"
	targets = discovery.relabel.docker.output
	forward_to = [loki.process.final.receiver]
}

discovery.relabel "docker" {
	targets = discovery.docker.linux.targets

	rule {
		source_labels = ["__meta_docker_container_name"]

		target_label = "container_name"
	}

	rule {
		source_labels = ["__meta_docker_container_id"]
		target_label  = "container_id"
	}
}

loki.relabel "journal" {
	forward_to = []

	rule {
		source_labels = ["__journal__systemd_unit"]
		target_label  = "unit"
	}

	rule {
		source_labels = ["__journal_priority_keyword"]
		target_label  = "level"
	}

	rule {
		source_labels = ["__journal__transport"]
		target_label  = "transport"
	}

	rule {
		source_labels = ["__journal__boot_id"]
		target_label  = "boot_id"
	}
}

loki.source.journal "read" {
	forward_to    = [loki.process.final.receiver]
	relabel_rules = loki.relabel.journal.rules
}

loki.process "final" {
	forward_to = [loki.write.grafana_cloud.receiver]

	stage.drop {
		older_than = "1h"
	}
}

loki.write "grafana_cloud" {
	external_labels = {instance = constants.hostname}

	endpoint {
		url = "https://USER:SECRET@logs-prod3.grafana.net/loki/api/v1/push"
	}
}

Logs

# the following logs repeat many times, once for each container; then repeat every few minutes

2023-07-11 19:00:04	ts=2023-07-11T23:00:04.735108377Z component=loki.source.docker.default level=warn target=docker/4f8edb61c3d75c606ddf0159ba4892aee758a53a0a58e9b4faec13fa63457538 msg="could not transfer logs" written=0 container=4f8edb61c3d75c606ddf0159ba4892aee758a53a0a58e9b4faec13fa63457538 err="context canceled"

2023-07-11 19:00:04	ts=2023-07-11T23:00:04.735116325Z component=loki.source.docker.default level=error target=e2a20d798dbb394b802179ed42fd1001ab4a5d0ed66c69649e65df3a65be0221 msg="could not set up a wait request to the Docker client" error=

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 9
  • Comments: 16 (2 by maintainers)

Most upvoted comments

Hey folks, happy new year! 🎉

We’ve merged #6055 which allows loki.source.docker to automatically deduplicate targets with matching container IDs which should alleviate the problem based on our tests.

The fix is out on main, and will be available on the Agent’s v0.40 release in about 6 weeks. In the meantime, you can try using one of the latest images in our dockerhub repo, or build your own from main.

Let us know how this works for you, and thank you for your patience!

The documentation gives an example of using loki.source.docker in combination with discovery.docker. In that setup I frequently get errors similar to this issue. https://grafana.com/docs/agent/latest/flow/reference/components/loki.source.docker/

It seems that the workaround can be even more complicated because in my environment a single container belonged to multiple networks. This did not occur when I collect logs in static mode. It is now a blocker for migration to flow mode 😢

If a solution is not available soon, it might be a good idea to mention the drawbacks of this approach in the documentation.

I’m having exactly the same problem as you. In addition, I see an increasing memory consumption, it seems to be the docker-driver responsible. I have to restart the docker engine

LGTM (I picked the latest commit on main 6965ae2 for testing)

This bug is absolutely insane. I have containers without exposed ports, with only one network, and only an IPv4 address. The bug still appears, and the Grafana Agent Flow does not push logs from some containers.

Solved it via removing the Docker integration completely (I need only logs).

River example:

loki.write "local" {
  endpoint {
    url = "...."
  }
}

loki.process "drop_ip" {
  forward_to = [loki.write.local.receiver]
  stage.label_drop {
    values = [ "ip_addr","filename",]
  }
}

loki.process "geoip" {    
  forward_to = [loki.relabel.level.receiver]
  stage.geoip {
    source  = "ip_addr"
    db      = "/geoip/GeoLite2-City.mmdb"
    db_type = "city"
  }
  stage.labels {
    values = {
      geoip_city_name          = "",
      geoip_country_name       = "",
    }
  }
}
loki.process "docker_json" {
  forward_to = [loki.process.ip_and_drop.receiver]
  stage.json {
    expressions = {message = "log", timestamp = "time", tag = "attrs.tag" }
  }
  stage.regex {
    expression = ".*\\/(?P<container>.*)"
    source = "tag"
  }
  stage.output {
    source = "message"
  }
  stage.labels {
    values = {
      tag  = "",
      container = "",
    }
  }
}
loki.process "ip_and_drop" {
  forward_to = [loki.process.geoip.receiver]
  stage.regex {
    expression = "^(?P<ip_addr>[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+) "
  }
  stage.labels {
    values = {
      ip_addr = "",
    }
  }
  
  stage.drop {
    older_than          = "1h"
    drop_counter_reason = "too old"
  }
}


local.file_match "docker" {
  path_targets = [{"__path__" = "/var/lib/docker/containers/**/*.log", "instance" = "my_instance", "job" = "docker/my_instance"}]
}
loki.source.file "files" {
  targets    = local.file_match.docker.targets
  forward_to = [loki.process.docker_json.receiver]
}

Also you need to add option to docker logging driver: tag: "my_instance/{{.Name}}"

Btw without Docker integration - logs appears much faster (almost isntantelly). With integration I have delay about 8-12 sec (even if I change interval to 1sec).

One way to resolve this would be to give us a way to simply take the exported targets, which is a list of maps, and then be able to create a new list of maps excluding all of the duplicate items.

For example, we could iterate over the list of maps based on the __meta_docker_container_id and keep only the first instance of each __meta_docker_container_id.

Currently the standard library is quite limited in terms of data manipulation functions, but maybe this could be implemented in another way. Allow us to set one of the returned fields as a “filter” where only the first instance is returned and others discarded.

I’ve try to apply filter to make sure that loki.source.docker has only one instance of each container in the targets and errors could not set up a wait request to the Docker client gone.

@mai-rnd Can you share an example of a working filter setup?

I have filtered by port, for example:

discovery.docker "scrapable" {
    host = "unix:///var/run/docker.sock"
}
discovery.relabel "scrapable" {
    targets = discovery.docker.scrapable.targets
    rule {
        source_labels = ["__meta_docker_port_private" ]
        regex         = "8080"
        action        = "keep"
    }
....

But it’s not universal solution. Still try to find some “univeral” way. Also checked how it’s works in Promtail with docker_sd_configs component - looks like Promtail filters targets internally and we have only uniq container_id in the targets list.

Can confirm. Additionally I see a CPU spike in the agent and loki containers every minute. Goes away when loki.source.docker is removed from the agent flow config.

Interestingly, the default refresh interval for the prometheus docker SD object is 1 minute as well. https://github.com/grafana/agent/blob/52d855ee5f881cdae595698f607ae1f8533b4195/component/discovery/docker/docker.go#L57