moby: logdriver syslog tcp+tls - Failed to initialize logging driver: EOF
Trying to log to logstash tcp input endpoint with the docker syslog log driver failes with the rather unspecific error Failed to initialize logging driver: EOF
I’m a novice at tls, but here’s what I’m trying to do:
- Created a CA
- Created a certificate signed by this CA for the logstash host
- Created a certificate signed by this CA for the host which is to be logged
- Set the ca and logstash host certificate in the logstash configuration
- Send logs to logstash using the docker syslog log driver via an tcp+tls:// address using the ca and the host to be logged certificate
Connecting using the openssl s_client using the certificates works and test logs sent through it are processed so I’m fairly certain the logstash config is not the problem.
Output of docker version
:
production host
Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 15:38:58 2016
OS/Arch: linux/amd64
Server:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 15:38:58 2016
OS/Arch: linux/amd64
local test
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
Output of docker info
:
production host
Containers: 93
Running: 6
Paused: 0
Stopped: 87
Images: 17
Server Version: 1.10.3
Storage Driver: overlay
Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
Volume: local
Network: host bridge null
Kernel Version: 4.6.0-0.bpo.1-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 996.6 MiB
Name: host001
ID: HS4V:K6WB:PY7B:MH3D:O2DP:IAHT:WLG2:ZDKM:ZYVR:OFDD:2KRL:MMRK
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No oom kill disable support
local test
Containers: 74
Running: 3
Paused: 0
Stopped: 71
Images: 168
Server Version: 1.11.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 422
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 4.4.0-31-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 6.936 GiB
Name: himalia
ID: N4CG:BGIP:4OE5:AMYR:EXTG:OE4J:T5FT:5PWC:RBRG:N2LQ:QKW6:5NC3
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): true
File Descriptors: 24
Goroutines: 50
System Time: 2016-07-27T14:21:00.171180294+02:00
EventsListeners: 0
Username: svensp
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.): The production host runs on a vhost. The local test runs on physical. Logstash runs on a rancher managed environment. The external Port is provided by its Load Balancer Service(haproxy 1.5) as tcp port without ssl knowledge
Steps to reproduce the issue:
- Set up ca + certificates
- Set up logstash with the following config, this is most easily done trough the rancher catalog entry
input {
tcp {
port => 6000
type => 'syslog'
ssl_enable => true
ssl_extra_chain_certs => [ '/etc/certs/ca.crt' ]
ssl_cert => '/etc/certs/cert.crt'
ssl_key => '/etc/certs/cert.key'
ssl_verify => true
}
}
output {
stdout {
codec => rubydebug
}
}
docker run --log-driver=syslog --log-opt syslog-address=tcp+tls://TCP_LOADBALANCER:PORT --log-opt syslog-tls-ca-cert=`readlink -f cacert.crt` --log-opt syslog-tls-cert=`readlink -f cert.cert` --log-opt syslog-tls-key=`readlink -f cert.key` -p 10000:80 nginx
Describe the results you received: Starting the container fails with the error
docker: Error response from daemon: Failed to initialize logging driver: EOF.
Describe the results you expected: The container starts and sends syslog style logs to logstash OR the container does not start and provides an indication at what point the tls handshake failed.
Additional information you deem important (e.g. issue happens only occasionally):
Logstash 1.5 warned about no cipher suites in common
. I can’t provide the exact error message because among the things I tried to make this work was upgrading to Logstash 2.3.4 which no longer provides this message.
The test command
openssl s_client -key *.key -cert *.cert -CAfile *.crt -connect TCP_LOADBALANCER:PORT
works providing the information
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA
Session-ID: 5798AA1AE42F0C243FB389E1202797C5C4BBEA8F3B3A04CD03F32B54E125B7F4
Session-ID-ctx:
Master-Key: 06F707D23D0D642F1DBE393E89709D9AC26422DE2A555BFF115537C5C61DDC6F9C83862E40505C3B473D1D6F85DC4776
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1469622810
Timeout : 300 (sec)
Verify return code: 0 (ok)
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 1
- Comments: 35 (7 by maintainers)
@ripcurld0 yes we tried installing that on the JVM but logstash uses jruby, which appears to override the underlying JVMs JCE provider with its own list of ciphersuites.
That’s fine awaiting reply on the last edge from @svensp 😃 this is very useful topic and informative.
It turned out that my problem genuinely was an unexpected EOF - ie the server closed the connection without sending a response.