moby: logdriver syslog tcp+tls - Failed to initialize logging driver: EOF

Trying to log to logstash tcp input endpoint with the docker syslog log driver failes with the rather unspecific error Failed to initialize logging driver: EOF

I’m a novice at tls, but here’s what I’m trying to do:

• Created a CA
• Created a certificate signed by this CA for the logstash host
• Created a certificate signed by this CA for the host which is to be logged
• Set the ca and logstash host certificate in the logstash configuration
• Send logs to logstash using the docker syslog log driver via an tcp+tls:// address using the ca and the host to be logged certificate

Connecting using the openssl s_client using the certificates works and test logs sent through it are processed so I’m fairly certain the logstash config is not the problem.

Output of docker version:

production host

Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:38:58 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:38:58 2016
 OS/Arch:      linux/amd64

local test

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64

Output of docker info:

production host

Containers: 93
 Running: 6
 Paused: 0
 Stopped: 87
Images: 17
Server Version: 1.10.3
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 4.6.0-0.bpo.1-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.6 MiB
Name: host001
ID: HS4V:K6WB:PY7B:MH3D:O2DP:IAHT:WLG2:ZDKM:ZYVR:OFDD:2KRL:MMRK
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No oom kill disable support

local test

Containers: 74
 Running: 3
 Paused: 0
 Stopped: 71
Images: 168
Server Version: 1.11.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 422
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.0-31-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 6.936 GiB
Name: himalia
ID: N4CG:BGIP:4OE5:AMYR:EXTG:OE4J:T5FT:5PWC:RBRG:N2LQ:QKW6:5NC3
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): true
 File Descriptors: 24
 Goroutines: 50
 System Time: 2016-07-27T14:21:00.171180294+02:00
 EventsListeners: 0
Username: svensp
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.): The production host runs on a vhost. The local test runs on physical. Logstash runs on a rancher managed environment. The external Port is provided by its Load Balancer Service(haproxy 1.5) as tcp port without ssl knowledge

Steps to reproduce the issue:

1. Set up ca + certificates
2. Set up logstash with the following config, this is most easily done trough the rancher catalog entry

input {
    tcp {
          port => 6000
          type => 'syslog'
          ssl_enable => true
          ssl_extra_chain_certs => [ '/etc/certs/ca.crt' ]
          ssl_cert => '/etc/certs/cert.crt'
          ssl_key => '/etc/certs/cert.key'
          ssl_verify => true
        }

}



output {
    stdout {
   codec => rubydebug
}
}

docker run --log-driver=syslog --log-opt syslog-address=tcp+tls://TCP_LOADBALANCER:PORT --log-opt syslog-tls-ca-cert=`readlink -f cacert.crt` --log-opt syslog-tls-cert=`readlink -f cert.cert` --log-opt syslog-tls-key=`readlink -f cert.key`  -p 10000:80 nginx

Describe the results you received: Starting the container fails with the error

docker: Error response from daemon: Failed to initialize logging driver: EOF.

Describe the results you expected: The container starts and sends syslog style logs to logstash OR the container does not start and provides an indication at what point the tls handshake failed.

Additional information you deem important (e.g. issue happens only occasionally): Logstash 1.5 warned about no cipher suites in common. I can’t provide the exact error message because among the things I tried to make this work was upgrading to Logstash 2.3.4 which no longer provides this message.

The test command

openssl s_client -key *.key -cert *.cert -CAfile *.crt -connect TCP_LOADBALANCER:PORT

works providing the information

SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-SHA
    Session-ID: 5798AA1AE42F0C243FB389E1202797C5C4BBEA8F3B3A04CD03F32B54E125B7F4
    Session-ID-ctx: 
    Master-Key: 06F707D23D0D642F1DBE393E89709D9AC26422DE2A555BFF115537C5C61DDC6F9C83862E40505C3B473D1D6F85DC4776
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1469622810
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)