rancher: Load balancer logs are empty - error during parsing file /etc/haproxy/rsyslogd.conf

@jmoscato, I was able to resolve it in my setup by adding this code to rancher-compose.yml

   lb_config:
     certs: []
     config: |-
       global
         log 127.0.0.1:8514 local0

       defaults
         log global
         mode    tcp
         option  tcplog

log 127.0.0.1:8514 local0 - 8514 is the rsyslog listening port in Rancher’s HAProxy container. I think this should be added to the documentation.

Hi @kinarashah Thanks for your feedback. I tried to send some requests through the load balancer (a few tens of them) but get very little information from stderr and stdout:

2/15/2018 9:25:24 AMtime="2018-02-15T08:25:24Z" level=info msg=" -- starting haproxy\n * Starting haproxy haproxy\n[WARNING] 045/082523 (41) : config : 'option forwardfor' ignored for proxy 'default' as it requires HTTP mode.\n[WARNING] 045/082523 (42) : config : 'option forwardfor' ignored for proxy 'default' as it requires HTTP mode.\n   ...done.\n"
2/15/2018 9:25:37 AMFeb 15 08:25:37 localhost haproxy[43]: Connect from 10.134.45.21:8750 to 10.42.75.129:443 (443/HTTP)
2/15/2018 9:25:37 AMFeb 15 08:25:37 localhost haproxy[43]: Connect from 10.134.45.21:8752 to 10.42.75.129:443 (443/HTTP)
2/15/2018 9:25:37 AMFeb 15 08:25:37 localhost haproxy[43]: Connect from 10.134.45.21:8747 to 10.42.75.129:443 (443/HTTP)
2/15/2018 9:25:37 AMFeb 15 08:25:37 localhost haproxy[43]: Connect from 10.134.45.21:8748 to 10.42.75.129:443 (443/HTTP)
2/15/2018 9:25:37 AMFeb 15 08:25:37 localhost haproxy[43]: Connect from 10.134.45.21:8751 to 10.42.75.129:443 (443/HTTP)
2/15/2018 9:25:37 AMFeb 15 08:25:37 localhost haproxy[43]: Connect from 10.134.45.21:8749 to 10.42.75.129:443 (443/HTTP)
2/15/2018 9:27:44 AMFeb 15 08:27:44 localhost haproxy[43]: Connect from 10.134.45.21:8765 to 10.42.75.129:443 (443/HTTP)
2/15/2018 9:27:44 AMFeb 15 08:27:44 localhost haproxy[43]: Connect from 10.134.45.21:8766 to 10.42.75.129:443 (443/HTTP)
2/15/2018 9:27:44 AMFeb 15 08:27:44 localhost haproxy[43]: Connect from 10.134.45.21:8767 to 10.42.75.129:443 (443/HTTP)
2/15/2018 9:27:44 AMFeb 15 08:27:44 localhost haproxy[43]: Connect from 10.134.45.21:8768 to 10.42.75.129:443 (443/HTTP)
2/15/2018 9:27:44 AMFeb 15 08:27:44 localhost haproxy[43]: Connect from 10.134.45.21:8769 to 10.42.75.129:443 (443/HTTP)
2/15/2018 9:27:44 AMFeb 15 08:27:44 localhost haproxy[43]: Connect from 10.134.45.21:8770 to 10.42.75.129:443 (443/HTTP)

All these messages are reporting is the IP of the machine where the requests are coming from (10.134.45.21) and the IP of the load balancer container (10.42.75.129):

Connect from 10.134.45.21:8750 to 10.42.75.129:443 (443/HTTP)

But there is no information about connection to the backend containers and status of the requests going through

Tested with rancher/server: Build from V1.6 branch Lb image - rancher/lb-service-haproxy:v0.7.6

Scenario 1: Create a service Add a load balancer and configure a port with its target pointing to this service (without enabling logging). Make some requests to the load balancer’s port No logs are generated for haproxy traffic - /var/log/haproxy/events|errors|traffic are empty. /var/log/messages does not show any error messages relating to parsing file /etc/haproxy/rsyslogd.conf

Scenario 2: Create a service Add a load balancer and configure a port with its target pointing to this service with following custom config for enabling logs:

defaults
log 127.0.0.1:8514 local0 debug

Make some requests to the load balancer’s port

Haproxy events and traffic related messages get logged in both - /var/log/haproxy/traffic and /var/log/haproxy/events . https://github.com/rancher/rancher/issues/9369 - logged for redundant logging.

Awesome. Is this documented somewhere?

Solved this by explicitly setting the log format for the frontends in HAProxy with option httplog

defaults
log 127.0.0.1:8514 local0 debug

frontend 8080
option httplog

frontend 443
option httplog

Entries in the log file now look like this:

5/25/2018 12:20:46 PMMay 25 10:20:46 localhost haproxy[351]: 10.134.45.21:15810 [25/May/2018:10:20:35.465] 443~ 443_application_cluster/123436406cec0b9217672c7306ba9b6f2 9496/0/1/1827/11327 200 1022 - - ---- 6/6/0/1/0 0/0 "GET /application/test? HTTP/1.1"

With HTTP status, backend name, server name, ecc…

@randomorder sometime after this, https://github.com/rancher/rancher/issues/9616, logs from those files were redirected to UI. are you able to see events and traffic getting logged in Standard Out and Errors in Standard Error under “View Logs”?

Any updates on this? I am seeing Apr 7 19:05:13 dfde93741290 rsyslogd-2207:error during parsing file /etc/haproxy/rsyslogd.conf, on or before line 16: invalid character ')' - is there an invalid escape sequence somewhere? [v8.16.0 try http://www.rsyslog.com/e/2207 ] in my rancher/lb-service-haproxy:v0.5.9 containers. All of my logs are empty in /var/log/haproxy/*

I also have the same config as others above, such as @VAdamec

$ModLoad imudp.so
$UDPServerRun 8514
$template CustomFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg%\n"
$ActionFileDefaultTemplate CustomFormat

/* info */
if $programname contains 'haproxy' and $syslogseverity == 6 then (
    action(type="omfile" file="/var/log/haproxy/traffic")
)
if $programname contains 'haproxy' and $syslogseverity-text == 'err' then (
    action(type="omfile" file="/var/log/haproxy/errors")
)
/* notice */
if $programname contains 'haproxy' and $syslogseverity <= 5 then (
    action(type="omfile" file="/var/log/haproxy/events")
)

*.* /var/log/messages