logspout: Not sending logs to papertrail as of a few days ago

I’ve been running latest in my dev environments for a while now successfully and as of 3/3/2016 I noticed that we weren’t getting any logs in papertrail. The container is started and I when I do a docker logs logspout I see only the below (port changed for posting here). I see nothing in docker.log or anywhere else. I tried pulling/running the v2 tag which works for a time and then ultimately crashes when another container is updated and possibly in other scenarios as well (haven’t spent time troubleshooting v2…would rather get v3 working). Using docker inspect logspout I didn’t see any obvious config differences between the running v2 and the nonfunctional v3. Any ideas or how I can further troubleshoot?

# logspout v3 by gliderlabs
# adapters: syslog raw tls udp tcp
# options : persist:/mnt/routes
# jobs    : pump http[logs,routes]:80
# routes  :
#   ADAPTER ADDRESS             CONTAINERS  SOURCES OPTIONS
#   syslog  logs2.papertrailapp.com:12345               map[]

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 30 (6 by maintainers)

Most upvoted comments

We are having problems where logspout occasionally stops sending logs to papertrail. Interestingly, logspout continues to send logs for some containers on a host, even after logs for one container stops showing up in papertrail. This is for a low volume of log lines, using UDP logging on v3. It usually happens after a uptime of > 1 week (for both the logspout container and the application container).

I can force logspout to stop sending by running load tests that cause a high volume of log lines (>1000/sec). Nothing shows up in the logspout container’s log when this happens. I asked papertrail, and they say that their service can handle the volume of logs we are sending and that we are not hitting a limit on their side.

raj-room on Apr 15, 2016

I’m with Papertrail. While this isn’t Papertrail-specific[1] and I don’t know the logspout codebase well, I skimmed the reconnect logic in master and the recent commits. A few things stood out:

It looks like this Write call may be blocking. It may need a call to SetWriteDeadline, similar to how Papertrail’s own remote_syslog2 calls it
the calls to Dial (here, less so here) may need a DialTimeout call to make the Dial return

Again, I don’t know the logspout codebase well, so these may not be problems or may be taken care of somewhere else. I’m trying to do what I can in hopes that it inspires someone who does know the codebase better.

[1]: Syslog clients/senders control reconnect behavior (and some people pasted output from UDP, where the client doesn’t even know about reachability). Papertrail is widely used enough that it’s often the service which uncovers unexpected behavior in syslog clients.

troy on May 4, 2016

Same issue here, though not with papertrail. One of the containers is sending the logs correctly but the other isnt. Error message in faulty container is:

# logspout v3 by gliderlabs
# adapters: raw tls udp tcp syslog
# options : persist:/mnt/routes
# jobs    : http[routes,logs]:80 pump
# routes  :
#   ADAPTER ADDRESS     CONTAINERS  SOURCES OPTIONS
#   syslog  10.4.1.142:5555             map[]
2016/04/19 17:29:06 syslog: write udp 10.4.1.142:5555: connection refused
2016/04/19 17:29:10 syslog: write udp 10.4.1.142:5555: connection refused
2016/04/19 17:29:10 syslog: write udp 10.4.1.142:5555: connection refused
2016/04/19 17:29:10 syslog: write udp 10.4.1.142:5555: connection refused
2016/04/19 17:29:10 syslog: write udp 10.4.1.142:5555: connection refused
2016/04/19 17:29:11 syslog: write udp 10.4.1.142:5555: connection refused
2016/04/19 17:29:11 syslog: write udp 10.4.1.142:5555: connection refused

version is gliderlabs/logspout:latest

theikkila on Apr 19, 2016

Here to confirm it’s working with v2.

ghost on Mar 18, 2016