agent: loki.source.docker error: could not set up a wait request to the Docker client
What’s wrong?
I have a very basic Docker log scraping config with the agent running in flow mode. The logs seem to be transmitted normally, however I get constant error/warning messages from the grafana-agent complaining about the inability to read container logs. The error messages are produced and repeated for all targeted containers on a regular basis.
Steps to reproduce
Simply run AGENT_MODE=flow grafana-agent --server.http.listen-addr=0.0.0.0:12345 run /etc/grafana-agent.river with the provided config on the affected host.
Agent is running as user grafana-agent who is a member of the following groups: grafana-agent adm systemd-journal docker
System information
Linux 4.4.0-92-generic x86_64
Software version
Grafana Agent v0.34.3
Configuration
logging {
level = "warn"
format = "logfmt"
}
prometheus.scrape "default" {
targets = [{"__address__" = "localhost:12345"}]
forward_to = [prometheus.remote_write.prom.receiver]
}
prometheus.exporter.unix {
include_exporter_metrics = true
disable_collectors = ["mdadm"]
}
prometheus.remote_write "prom" { }
discovery.docker "linux" {
host = "unix:///var/run/docker.sock"
filter {
name = "status"
values = ["running"]
}
}
loki.source.docker "default" {
host = "unix:///var/run/docker.sock"
targets = discovery.relabel.docker.output
forward_to = [loki.process.final.receiver]
}
discovery.relabel "docker" {
targets = discovery.docker.linux.targets
rule {
source_labels = ["__meta_docker_container_name"]
target_label = "container_name"
}
rule {
source_labels = ["__meta_docker_container_id"]
target_label = "container_id"
}
}
loki.relabel "journal" {
forward_to = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
rule {
source_labels = ["__journal_priority_keyword"]
target_label = "level"
}
rule {
source_labels = ["__journal__transport"]
target_label = "transport"
}
rule {
source_labels = ["__journal__boot_id"]
target_label = "boot_id"
}
}
loki.source.journal "read" {
forward_to = [loki.process.final.receiver]
relabel_rules = loki.relabel.journal.rules
}
loki.process "final" {
forward_to = [loki.write.grafana_cloud.receiver]
stage.drop {
older_than = "1h"
}
}
loki.write "grafana_cloud" {
external_labels = {instance = constants.hostname}
endpoint {
url = "https://USER:SECRET@logs-prod3.grafana.net/loki/api/v1/push"
}
}
Logs
# the following logs repeat many times, once for each container; then repeat every few minutes
2023-07-11 19:00:04 ts=2023-07-11T23:00:04.735108377Z component=loki.source.docker.default level=warn target=docker/4f8edb61c3d75c606ddf0159ba4892aee758a53a0a58e9b4faec13fa63457538 msg="could not transfer logs" written=0 container=4f8edb61c3d75c606ddf0159ba4892aee758a53a0a58e9b4faec13fa63457538 err="context canceled"
2023-07-11 19:00:04 ts=2023-07-11T23:00:04.735116325Z component=loki.source.docker.default level=error target=e2a20d798dbb394b802179ed42fd1001ab4a5d0ed66c69649e65df3a65be0221 msg="could not set up a wait request to the Docker client" error=
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 9
- Comments: 16 (2 by maintainers)
Hey folks, happy new year! 🎉
We’ve merged #6055 which allows loki.source.docker to automatically deduplicate targets with matching container IDs which should alleviate the problem based on our tests.
The fix is out on main, and will be available on the Agent’s v0.40 release in about 6 weeks. In the meantime, you can try using one of the latest images in our dockerhub repo, or build your own from main.
Let us know how this works for you, and thank you for your patience!
The documentation gives an example of using
loki.source.dockerin combination withdiscovery.docker. In that setup I frequently get errors similar to this issue. https://grafana.com/docs/agent/latest/flow/reference/components/loki.source.docker/It seems that the workaround can be even more complicated because in my environment a single container belonged to multiple networks. This did not occur when I collect logs in static mode. It is now a blocker for migration to flow mode 😢
If a solution is not available soon, it might be a good idea to mention the drawbacks of this approach in the documentation.
I’m having exactly the same problem as you. In addition, I see an increasing memory consumption, it seems to be the docker-driver responsible. I have to restart the docker engine
LGTM (I picked the latest commit on main 6965ae2 for testing)
This bug is absolutely insane. I have containers without exposed ports, with only one network, and only an IPv4 address. The bug still appears, and the Grafana Agent Flow does not push logs from some containers.
Solved it via removing the Docker integration completely (I need only logs).
River example:
Also you need to add option to docker logging driver:
tag: "my_instance/{{.Name}}"Btw without Docker integration - logs appears much faster (almost isntantelly). With integration I have delay about 8-12 sec (even if I change interval to 1sec).
One way to resolve this would be to give us a way to simply take the exported targets, which is a list of maps, and then be able to create a new list of maps excluding all of the duplicate items.
For example, we could iterate over the list of maps based on the
__meta_docker_container_idand keep only the first instance of each__meta_docker_container_id.Currently the standard library is quite limited in terms of data manipulation functions, but maybe this could be implemented in another way. Allow us to set one of the returned fields as a “filter” where only the first instance is returned and others discarded.
I have filtered by port, for example:
But it’s not universal solution. Still try to find some “univeral” way. Also checked how it’s works in Promtail with docker_sd_configs component - looks like Promtail filters targets internally and we have only uniq container_id in the targets list.
Can confirm. Additionally I see a CPU spike in the agent and loki containers every minute. Goes away when
loki.source.dockeris removed from the agent flow config.Interestingly, the default refresh interval for the prometheus docker SD object is 1 minute as well. https://github.com/grafana/agent/blob/52d855ee5f881cdae595698f607ae1f8533b4195/component/discovery/docker/docker.go#L57