logstash-logger: TCP logging breaks rails when connection breaks

If we stop our ELK server while our app is running, our app will eventually stop working altogether after about 5 minutes. We are using Passenger Standalone and it doesn’t even seem to notice that anything is wrong, don’t see any helpful errors anywhere.

If we use UDP, an ELK outage is handled gracefully.

Here is our configuration for our QA environment:

  config.logger = LogStashLogger.new(
  type: :multi_delegator,
  outputs: [
    { type: :file, path: "log/#{Rails.env}.log" },
    { type: :tcp, host: ENV["ELK_LOG_HOST"], port: ENV["ELK_LOG_PORT"]}
  ])

  LogStashLogger.configure do |config|
    config.customize_event do |event|
      event["application_name"] = ENV["APPLICATION_TYPE"].present? ? ENV["APPLICATION_TYPE"] : "SentryLinkwebserver"
      event["environment_name"] = Rails.env
    end
  end