moby: Cannot reference syslog-address by service name

You cannot use the container-container network for logging; logging is handled by the docker daemon, not by the container; the container sends logs to stdout / stderr, and the docker daemon captures that output, and passes it to the logging driver. The daemon’s networking (and the logging driver) are not part of the container-container network, thus cannot send the logs there.

Here’s a simple example of the issue when trying to use overlay networks, ELK, and the GELF logging driver on a Swarm.

version: '2'

services:
  elasticsearch:
    container_name: elasticsearch
    image: elasticsearch:2.3
    restart: unless-stopped
    networks:
      - logging
    expose:
      - 9200
      - 9300

  logstash:
    container_name: logstash
    image: logstash:2.3
    restart: unless-stopped
    networks:
      - logging
    expose:
      - "12201/udp"
    depends_on:
      - elasticsearch
    command: logstash -e 'input { gelf { type => docker port => 12201 } } output { elasticsearch { hosts => ["elasticsearch:9200"] } stdout { codec => rubydebug } }'

  kibana:
    container_name: kibana
    image: kibana:4.5
    restart: unless-stopped
    networks:
      - logging
    ports:
      - 5601
    depends_on:
      - elasticsearch
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200

  nginx:
    container_name: nginx
    image: nginx:1.9
    restart: unless-stopped
    networks:
      - logging
    ports:
      - "80:80"
    logging:
      driver: gelf
      options:
        gelf-address: "udp://logstash:12201"
    depends_on:
      - logstash

networks:
  logging: {}

Running it produces the error

$ docker-compose up
Creating elasticsearch
Creating logstash
Creating kibana
Creating nginx

ERROR: for nginx  500 Internal Server Error: Failed to initialize logging driver: gelf: cannot connect to GELF endpoint: logstash:12201 dial udp: lookup logstash on xxx.xxx.xxx.xxx:53: no such host
Attaching to elasticsearch, logstash, kibana
...

I’ve confirmed that the ELK portion is working using this GELF Client. It’s the logging driver running on the host network stack as @mavenugo explained.

There are some solutions floating around that say to change gelf-address: "udp://logstash:12201" to gelf-address: "udp://localhost:12201" but that doesn’t work with an overlay network. The error message does go away but no logs are actually written.

Relevant versions

$ docker version
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 21:49:11 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      swarm/1.2.0
 API version:  1.22
 Go version:   go1.5.4
 Git commit:   a6c1f14
 Built:        Wed Apr 13 05:58:31 UTC 2016
 OS/Arch:      linux/amd64

$ docker-compose version
docker-compose version 1.7.0, build 0d7bf73
docker-py version: 1.8.0
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1j 15 Oct 2014

@gprivitera you can still use log-drivers, however, sending logs to a container won’t work if the container is only accessible over a docker network (as in the example @everett-toews showed). Logs are collected by the daemon. The daemon itself is not part of the docker-network, so cannot resolve / directly connect to a container, unless the container is exposing/publishing a port. See https://github.com/docker/docker/issues/20370#issuecomment-185229591

Though this is an interesting use-case, It is not easy to achieve (unless we deploy special tricks).

Reason : Name resolution between containers (via container name, links, aliases) are handled by the embedded DNS server (or the expanded server) in the context of the logical network formed between the containers. These networks are in isolated space managed and handled by the network driver. In some cases (such as overlay driver), the isolated network space is not even accessible from the host network namespace directly. Since the docker daemon operates in the host network stack, it may not even be able to access the service exposed within an application network stack. We could deploy some special tricks such as operating the daemon in container’s network stack. But it is trickier and complex.

The simpler alternative is to do port-mapping and access the mapped service from the docker daemon on the host-networking stack.

The solution was to publish ports: - “127.0.0.1:12201:12201/udp” instead of only 12201:12201. My problem is solved. Now I only have the problem that conntrack cashes the connection even after the receiving container dies.

Though this is an interesting use-case, It is not easy to achieve (unless we deploy special tricks).

Reason : Name resolution between containers (via container name, links, aliases) are handled by the embedded DNS server (or the expanded server) in the context of the logical network formed between the containers. These networks are in isolated space managed and handled by the network driver. In some cases (such as overlay driver), the isolated network space is not even accessible from the host network namespace directly. Since the docker daemon operates in the host network stack, it may not even be able to access the service exposed within an application network stack. We could deploy some special tricks such as operating the daemon in container’s network stack. But it is trickier and complex.

The simpler alternative is to do port-mapping and access the mapped service from the docker daemon on the host-networking stack.

On dockerCE for Windows it worked like

  demo:
    build: demo
    logging:
      driver: syslog
      options:
        syslog-address: tcp://host.docker.internal:5000  #-host.docker.internal resolved the port
    depends_on:
    - demo-logstash
    ports:
    - 8080:8080

I have been running in the same issue trying to run a similar compose file, this time on swarm (with a special network dedicated to logging). It felt like a good idea to leverage overlay network to spawn ELK in the same compose file than my app (not so sure on security point of view though …)

I have the same problem. I am using fluentd, but this probably applies to syslogd and journald as well. Containers with fluentd logging-driver can not connect to a fluentd container within the same stack in swarm mode.

As the fluentd architecture suggests you should have a fluentd proxy for the containers of your application, then “forward” it to a “global collector” that runs in front of elasticsearch and kibana. The recommended way to map the proxy to port 24224 of the host and let all containers log to this host port works, but it does not scale well.

If I want to run a stack on my dev machine it requires either a preinstalled fluentd daemon, or a free port 24224 which means I cannot run 2 stacks in parallel with default config. Both options are not desireable because they introduce an external dependency to my docker deployment. Workarounds with other ports require additional configuration and administration for every environment/swarm this stack is deployed to.

I’d love to see this working as a minimal configuration:

version: '3'
services:

  app:
    image: mycompany/app
    logging:
      driver: "fluentd"
      options:
        # this forwards logs to the own application stack, 
        # this should work once logger service is up (currently not supported)
        # would be even better if I could skip the default port
        fluentd-address: logger:24224

  logger:
    # listen to own port 24224
    # forwards to own stdout and host port 24224
    # if no daemon is running, the stack works anyway, the application is decoupled
    image: mycompany/fluentd-proxy

second stack in swarm might be the log analysis, which picks up logs from all stacks running in the swarm

version: '3'
services:

  logcollect:
    # listen to host port 24224
    # forwards to service elasticsearch
    image: mycompany/fluentd-collector

  elasticsearch:
    image: elasticsearch

  kibana:
    image: kibana
    ports:
      - "5601:5601"