passenger: Passenger process crashing on 5.0.27

Since version 5.0.27 we’re getting random crashes of a passenger process which results in 502 errors in Nginx for all the requests being handled by the crashing process. We’ve rolled back to version 5.0.26, which doesn’t have this issue. We’re running this version on two servers which both crash the same way since the update to 5.0.27. The webservers are loadbalanced by Haproxy.

We’re also using 5.0.27 successfully on identical servers but without the ‘direct’ spawn method required for our application using Ruby 1.8.7. So my guess would be it’s related to this option. I’ve attached the crash log and additional info below.

Passenger version: open source 5.0.27 with nginx (Phusion APT repo) (multi-app mode) OS: Ubuntu 14.04 LTS Ruby versions running: Ruby 1.8.7-p370, Ruby 1.9.3-p374, Ruby 2.1.1, Ruby 2.3.0

Passenger Nginx config:

server {
  listen   80 default_server; ## listen for ipv4
  listen   443 default_server ssl

  server_name  .piedpiper.com;

  root /srv/piedpiper/current/public;

  passenger_ruby /home/piedpiper/.rvm/gems/ruby-1.8.7-p370/wrappers/ruby;
  passenger_enabled on;
  passenger_spawn_method direct;
  passenger_min_instances 5;
  passenger_max_requests 50000;
}

Nginx log message:

Assertion failed! src/cxx_supportlib/MemoryKit/mbuf.cpp:218: void Passenger::MemoryKit::mbuf_block_put(Passenger::MemoryKit::mbuf_block*): ((mbuf_block)->ne
xt.stqe_next) == __null
[ pid=31259, timestamp=1460034815 ] Process aborted! signo=SIGABRT(6), reason=SI_TKILL, signal sent by PID 31259 with UID 0, si_addr=0x7a1b, randomSeed=1460
034763

Passenger crash log:

[ pid=28580, timestamp=1460034509 ] Process aborted! signo=SIGABRT(6), reason=SI_TKILL, signal sent by PID 28580 with UID 0, si_addr=0x6fa4, randomSeed=1460034161
[ pid=28580 ] Crash log dumped to /var/tmp/passenger-crash-log.1460034509
[ pid=28580 ] Date, uname and ulimits:
Thu Apr  7 13:08:29 UTC 2016
Linux 3.13.0-53-generic #89-Ubuntu SMP Wed May 20 10:34:39 UTC 2015 x86_64 x86_64
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     0
memory(kbytes)       unlimited
locked memory(kbytes) 64
process              256132
nofiles              1024
vmemory(kbytes)      unlimited
locks                unlimited
[ pid=28580 ] Phusion Passenger version: 5.0.27
[ pid=28580 ] Last assertion failure: (((mbuf_block)->next.stqe_next) == __null), function void Passenger::MemoryKit::mbuf_block_put(Passenger::MemoryKit::mbuf_block*), file src/cxx_supportlib/MemoryKit/mbuf.cpp, line 218.
[ pid=28580 ] libc backtrace available!
--------------------------------------
[ pid=28580 ] Backtrace with 35 frames:
Using backtrace sanitizer.
Passenger core[0x481390]
Passenger core[0x480c3c]
Passenger core[0x481f40]
Passenger core[0x482bfa]
/lib/x86_64-linux-gnu/libpthread.so.0(?? at ??:0; +0x10340) [0x7f00c36b5340]
/lib/x86_64-linux-gnu/libc.so.6(?? at ??:0; gsignal+0x39) [0x7f00c2676cc9]
/lib/x86_64-linux-gnu/libc.so.6(?? at ??:0; abort+0x148) [0x7f00c267a0d8]
Passenger core[0x482e8c]
Passenger core[0x47bedb]
Passenger core[0x4a0781]
Passenger core[0x5e6615]
Passenger core[0x5e62fe]
Passenger core[0x5f71cb]
Passenger core[0x5f0411]
Passenger core[0x5dcbb4]
Passenger core[0x5daf05]
Passenger core[0x4a20df]
Passenger core[0x4a357c]
Passenger core[0x4a4ced]
Passenger core[0x4a49e7]
Passenger core[0x4a49a1]
Passenger core[0x6d1d8e]
Passenger core[0x6d2c68]
Passenger core[0x4f3c5c]
Passenger core[0x559981]
Passenger core[0x553451]
Passenger core[0x548d08]
Passenger core[0x45e7be]
Passenger core[0x6c6cc9]
Passenger core[0x4f3a20]
Passenger core[0x4f35db]
Passenger core[0x4f019a]
Passenger core[0x6bbae9]
/lib/x86_64-linux-gnu/libpthread.so.0(?? at ??:0; +0x8182) [0x7f00c36ad182]
/lib/x86_64-linux-gnu/libc.so.6(?? at ??:0; clone+0x6d) [0x7f00c273a47d]
--------------------------------------
[ pid=28580 ] Dumping additional diagnostical information...
--------------------------------------
### Backtraces
Thread 'Main thread' (0x7f00c3cc97c0, LWP 28580):
     in 'void waitForExitEvent()' (CoreMain.cpp:875)
     in 'void mainLoop()' (CoreMain.cpp:768)
     in 'int runCore()' (CoreMain.cpp:995)

Thread 'Pool garbage collector' (0x7f00c3c86700, LWP 28580):
     in 'static void Passenger::ApplicationPool2::Pool::garbageCollect(Passenger::ApplicationPool2::PoolPtr)' (GarbageCollection.cpp:62)

Thread 'Pool analytics collector' (0x7f00c3cc7700, LWP 28580):
     in 'static void Passenger::ApplicationPool2::Pool::collectAnalytics(Passenger::ApplicationPool2::PoolPtr)' (AnalyticsCollection.cpp:64)

Thread 'Main event loop: thread 1' (0x7f00bd3d5700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 1: libuv poller' (0x7f00bcbd4700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 2' (0x7f00bcb53700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 2: libuv poller' (0x7f00bc352700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 3' (0x7f00affff700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 3: libuv poller' (0x7f00bc2d1700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 4' (0x7f00af7fe700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 4: libuv poller' (0x7f00bc250700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 5' (0x7f00aeffd700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 5: libuv poller' (0x7f00bc1cf700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 6' (0x7f00ae7fc700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 6: libuv poller' (0x7f00bc14e700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 7' (0x7f00adffb700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 7: libuv poller' (0x7f00bc0cd700, LWP 28580):
     (empty)

Thread 'Main event loop: thread 8' (0x7f00ad7fa700, LWP 28580):
     in 'Passenger::ServerKit::Channel::Result Passenger::Core::Controller::onAppSourceData(Passenger::Core::Client*, Passenger::ServerKit::HttpServer<Passenger::Core::Controller, Passenger::Core::Client>::Request*, const Passenger::MemoryKit::mbuf&, int)' (ForwardResponse.cpp:160) -- Client 8-1054

Thread 'Main event loop: thread 8: libuv poller' (0x7f00acff9700, LWP 28580):
     (empty)

Thread 'API event loop' (0x7f00acf78700, LWP 28580):
     (empty)

Thread 'API event loop: libuv poller' (0x7f00ac777700, LWP 28580):
     (empty)

Thread 'Load balancer' (0x7f0067fff700, LWP 28580):
     (empty)

Thread 'PipeWatcher: PID 28611 stdout, fd 77' (0x7f00ac06d700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28611 stderr, fd 78' (0x7f00675f2700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28616 stdout, fd 81' (0x7f00675b1700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28616 stderr, fd 82' (0x7f0067570700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28801 stdout, fd 178' (0x7f0065513700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28801 stderr, fd 179' (0x7f00654d2700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28882 stdout, fd 189' (0x7f0065479700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28882 stderr, fd 190' (0x7f0065438700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28982 stdout, fd 137' (0x7f00653df700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 28982 stderr, fd 150' (0x7f006539e700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29163 stdout, fd 195' (0x7f0065345700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29163 stderr, fd 196' (0x7f0065304700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29080 stdout, fd 191' (0x7f00652ab700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29080 stderr, fd 192' (0x7f006526a700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29285 stdout, fd 168' (0x7f0065211700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29285 stderr, fd 170' (0x7f00651d0700, LWP 28606):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29380 stdout, fd 86' (0x7f006518f700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29380 stderr, fd 87' (0x7f006514e700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29649 stderr, fd 124' (0x7f006508b700, LWP 29648):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29794 stdout, fd 132' (0x7f0064fde700, LWP 29648):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29804 stdout, fd 93' (0x7f0064f9d700, LWP 29648):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29541 stdout, fd 159' (0x7f0064f5c700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29541 stderr, fd 161' (0x7f0064f1b700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29841 stdout, fd 121' (0x7f0064eda700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 29841 stderr, fd 122' (0x7f0064e99700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30053 stdout, fd 96' (0x7f0064e40700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30053 stderr, fd 97' (0x7f0064dff700, LWP 28615):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30341 stdout, fd 101' (0x7f0064da6700, LWP 30340):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30341 stderr, fd 102' (0x7f0064d65700, LWP 30340):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30649 stdout, fd 104' (0x7f0064d0c700, LWP 30648):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30649 stderr, fd 105' (0x7f0064ccb700, LWP 30648):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30831 stdout, fd 90' (0x7f0064c72700, LWP 30830):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 30831 stderr, fd 91' (0x7f0064c31700, LWP 30830):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 31272 stdout, fd 100' (0x7f0064bd8700, LWP 31271):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 31272 stderr, fd 103' (0x7f0064b97700, LWP 31271):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 31740 stdout, fd 113' (0x7f0064b3e700, LWP 31739):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 31740 stderr, fd 114' (0x7f0064afd700, LWP 31739):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 32491 stdout, fd 107' (0x7f0064aa4700, LWP 32490):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 32491 stderr, fd 108' (0x7f0064a63700, LWP 32490):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 626 stdout, fd 109' (0x7f0064a0a700, LWP 625):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 626 stderr, fd 110' (0x7f00649c9700, LWP 625):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 827 stdout, fd 129' (0x7f0064970700, LWP 826):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 827 stderr, fd 130' (0x7f006492f700, LWP 826):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 2042 stdout, fd 134' (0x7f00648ee700, LWP 2041):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

Thread 'PipeWatcher: PID 2042 stderr, fd 135' (0x7f00648ad700, LWP 2041):
     in 'void Passenger::SpawningKit::PipeWatcher::threadMain()' (PipeWatcher.h:84)
     in 'static void Passenger::SpawningKit::PipeWatcher::threadMain(boost::shared_ptr<Passenger::SpawningKit::PipeWatcher>)' (PipeWatcher.h:66)

### Request handler state (thread 1)

{
    "active_client_count" : 1,
    "active_clients" :
    {
        "1-1055" :
        {
            "connected_at" :
            {
                "local" : "Thu Apr  7 13:08:29 2016",
                "relative" : "0s ago",
                "relative_timestamp" : -0.0004122257232666016,
                "timestamp" : 1460034509.332909
            },
            "connection_state" : "ACTIVE",
            "current_request" :
            {
                "app_response_http_state" : "PARSING_HEADERS",
                "app_sink_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : false
                },
                "app_source_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : true
                },
                "content_length" : 35956,
                "flags" :
                {
                    "dechunk_response" : true,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "www.piedpiper.com",
                "http_major" : 1,
                "http_minor" : 1,
                "http_state" : "PARSING_BODY",
                "last_data_receive_time" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -6.914138793945312e-06,
                    "timestamp" : 1460034509.333058
                },
                "last_data_send_time" : null,
                "method" : "POST",
                "path" : "/projects/E3350212/update.xml",
                "refcount" : 1,
                "request_body_already_read" : 35956,
                "request_body_fully_read" : true,
                "request_body_type" : "CONTENT_LENGTH",
                "response_begun" : false,
                "session" :
                {
                    "gupid" : "1734e4f-WHCU6B27UH",
                    "pid" : 29380
                },
                "session_checkout_try" : 1,
                "started_at" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -0.0003762245178222656,
                    "timestamp" : 1460034509.332804
                },
                "state" : "WAITING_FOR_APP_OUTPUT",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "1-1055",
            "number" : 1055,
            "output_channel_state" :
            {
                "bytes_buffered" :
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        }
    },
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 153.480
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.790
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 127,
    "free_request_count" : 13,
    "mbuf_pool" :
    {
        "active_blocks" : 3,
        "active_memory" :
        {
            "bytes" : 1536,
            "human_readable" : "1.5 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 313,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 160256,
            "human_readable" : "156.5 KB"
        }
    },
    "peak_active_client_count" : 14,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 153.480
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.790
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2648712,
    "total_clients_accepted" : 1055,
    "total_requests_begun" : 1055,
    "turbocaching" :
    {
        "fetches" : 4,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 4
    }
}

### Request handler state (thread 2)

{
    "active_client_count" : 1,
    "active_clients" :
    {
        "2-1055" :
        {
            "connected_at" :
            {
                "local" : "Thu Apr  7 13:08:29 2016",
                "relative" : "0s ago",
                "relative_timestamp" : -0.0002169609069824219,
                "timestamp" : 1460034509.33530
            },
            "connection_state" : "ACTIVE",
            "current_request" :
            {
                "app_response_http_state" : "PARSING_HEADERS",
                "app_sink_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : false
                },
                "app_source_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : true
                },
                "flags" :
                {
                    "dechunk_response" : true,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "piedpiper.com",
                "http_major" : 1,
                "http_minor" : 1,
                "http_state" : "COMPLETE",
                "last_data_receive_time" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -0.0001230239868164062,
                    "timestamp" : 1460034509.335277
                },
                "last_data_send_time" : null,
                "method" : "GET",
                "path" : "/dashboard",
                "refcount" : 1,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session" :
                {
                    "gupid" : "1734e4e-sT7bi5Yd7P",
                    "pid" : 28616
                },
                "session_checkout_try" : 1,
                "started_at" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -0.0001230239868164062,
                    "timestamp" : 1460034509.335322
                },
                "state" : "WAITING_FOR_APP_OUTPUT",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "2-1055",
            "number" : 1055,
            "output_channel_state" :
            {
                "bytes_buffered" :
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        }
    },
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 153.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.930
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 127,
    "free_request_count" : 14,
    "mbuf_pool" :
    {
        "active_blocks" : 4,
        "active_memory" :
        {
            "bytes" : 2048,
            "human_readable" : "2.0 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 291,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 148992,
            "human_readable" : "145.5 KB"
        }
    },
    "peak_active_client_count" : 15,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 153.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.930
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2741175,
    "total_clients_accepted" : 1055,
    "total_requests_begun" : 1055,
    "turbocaching" :
    {
        "fetches" : 4,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 4
    }
}

### Request handler state (thread 3)

{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.040
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.960
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 17,
    "mbuf_pool" :
    {
        "active_blocks" : 0,
        "active_memory" :
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 512,
        "free_blocks" : 294,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 150528,
            "human_readable" : "147.0 KB"
        }
    },
    "peak_active_client_count" : 17,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.040
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.020
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2418239,
    "total_clients_accepted" : 1055,
    "total_requests_begun" : 1055,
    "turbocaching" :
    {
        "fetches" : 1,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 1
    }
}

### Request handler state (thread 4)

{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.50
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.660
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 15,
    "mbuf_pool" :
    {
        "active_blocks" : 0,
        "active_memory" :
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 512,
        "free_blocks" : 329,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 168448,
            "human_readable" : "164.5 KB"
        }
    },
    "peak_active_client_count" : 15,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.50
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.660
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2578363,
    "total_clients_accepted" : 1055,
    "total_requests_begun" : 1055,
    "turbocaching" :
    {
        "fetches" : 2,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 2
    }
}

### Request handler state (thread 5)

{
    "active_client_count" : 1,
    "active_clients" :
    {
        "5-1055" :
        {
            "connected_at" :
            {
                "local" : "Thu Apr  7 13:08:29 2016",
                "relative" : "0s ago",
                "relative_timestamp" : -0.0002715587615966797,
                "timestamp" : 1460034509.339006
            },
            "connection_state" : "ACTIVE",
            "current_request" :
            {
                "app_response_http_state" : "PARSING_HEADERS",
                "app_sink_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : false
                },
                "app_source_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : true,
                    "io_watcher_active" : true
                },
                "flags" :
                {
                    "dechunk_response" : true,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "piedpiper.com",
                "http_major" : 1,
                "http_minor" : 1,
                "http_state" : "COMPLETE",
                "last_data_receive_time" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -0.0001716613769531250,
                    "timestamp" : 1460034509.338994
                },
                "last_data_send_time" : null,
                "method" : "GET",
                "path" : "/projects/22896014-javastraat-112-2/embed2",
                "refcount" : 1,
                "request_body_already_read" : 0,
                "request_body_fully_read" : true,
                "request_body_type" : "NO_BODY",
                "response_begun" : false,
                "session" :
                {
                    "gupid" : "1734e4e-o5wXNU2aWE",
                    "pid" : 28882
                },
                "session_checkout_try" : 1,
                "started_at" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : -0.0001716613769531250,
                    "timestamp" : 1460034509.339036
                },
                "state" : "WAITING_FOR_APP_OUTPUT",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "5-1055",
            "number" : 1055,
            "output_channel_state" :
            {
                "bytes_buffered" :
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        }
    },
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.730
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 127,
    "free_request_count" : 15,
    "mbuf_pool" :
    {
        "active_blocks" : 4,
        "active_memory" :
        {
            "bytes" : 2048,
            "human_readable" : "2.0 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 305,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 156160,
            "human_readable" : "152.5 KB"
        }
    },
    "peak_active_client_count" : 16,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.730
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2614924,
    "total_clients_accepted" : 1055,
    "total_requests_begun" : 1055,
    "turbocaching" :
    {
        "fetches" : 4,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 4
    }
}

### Request handler state (thread 6)

{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.850
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 15,
    "mbuf_pool" :
    {
        "active_blocks" : 0,
        "active_memory" :
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 512,
        "free_blocks" : 292,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 149504,
            "human_readable" : "146.0 KB"
        }
    },
    "peak_active_client_count" : 15,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 183.850
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 3280567,
    "total_clients_accepted" : 1054,
    "total_requests_begun" : 1054,
    "turbocaching" :
    {
        "fetches" : 5,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 5
    }
}

### Request handler state (thread 7)

{
    "active_client_count" : 0,
    "active_clients" : {},
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 184.60
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 128,
    "free_request_count" : 16,
    "mbuf_pool" :
    {
        "active_blocks" : 0,
        "active_memory" :
        {
            "bytes" : 0,
            "human_readable" : "0 bytes"
        },
        "chunk_size" : 512,
        "free_blocks" : 343,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 175616,
            "human_readable" : "171.5 KB"
        }
    },
    "peak_active_client_count" : 16,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.490
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 184.60
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2633528,
    "total_clients_accepted" : 1054,
    "total_requests_begun" : 1054,
    "turbocaching" :
    {
        "fetches" : 2,
        "hit_ratio" : 0.0,
        "hits" : 0,
        "store_success_ratio" : 0.0,
        "store_successes" : 0,
        "stores" : 2
    }
}

### Request handler state (thread 8)

{
    "active_client_count" : 1,
    "active_clients" :
    {
        "8-1054" :
        {
            "connected_at" :
            {
                "local" : "Thu Apr  7 13:08:29 2016",
                "relative" : "1s ago",
                "relative_timestamp" : -0.2005658149719238,
                "timestamp" : 1460034509.142548
            },
            "connection_state" : "ACTIVE",
            "current_request" :
            {
                "app_sink_state" :
                {
                    "callback_in_progress" : false,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "app_source_state" :
                {
                    "callback_in_progress" : true,
                    "initialized" : false,
                    "io_watcher_active" : false
                },
                "content_length" : 128,
                "flags" :
                {
                    "dechunk_response" : true,
                    "https" : false,
                    "request_body_buffering" : false
                },
                "host" : "www.piedpiper.com",
                "http_major" : 1,
                "http_minor" : 1,
                "http_state" : "WAITING_FOR_REFERENCES",
                "last_data_receive_time" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "1s ago",
                    "relative_timestamp" : -0.2005050182342529,
                    "timestamp" : 1460034509.142490
                },
                "last_data_send_time" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "0s ago",
                    "relative_timestamp" : 0.0,
                    "timestamp" : 1460034509.342995
                },
                "method" : "POST",
                "path" : "/login",
                "refcount" : 2,
                "request_body_already_read" : 128,
                "request_body_fully_read" : true,
                "request_body_type" : "CONTENT_LENGTH",
                "response_begun" : true,
                "session_checkout_try" : 1,
                "started_at" :
                {
                    "local" : "Thu Apr  7 13:08:29 2016",
                    "relative" : "1s ago",
                    "relative_timestamp" : -0.2005050182342529,
                    "timestamp" : 1460034509.142545
                },
                "state" : "WAITING_FOR_APP_OUTPUT",
                "sticky_session" : false,
                "want_keep_alive" : false
            },
            "lingering_request_count" : 0,
            "name" : "8-1054",
            "number" : 1054,
            "output_channel_state" :
            {
                "bytes_buffered" :
                {
                    "bytes" : 0,
                    "human_readable" : "0 bytes"
                },
                "callback_in_progress" : false,
                "mode" : "IN_MEMORY_MODE",
                "nbuffers" : 0,
                "reader_state" : "RS_INACTIVE"
            },
            "refcount" : 2,
            "requests_begun" : 1
        }
    },
    "client_accept_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.440
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.170
        }
    },
    "disconnected_client_count" : 0,
    "disconnected_clients" : {},
    "free_client_count" : 127,
    "free_request_count" : 13,
    "mbuf_pool" :
    {
        "active_blocks" : 5,
        "active_memory" :
        {
            "bytes" : 2560,
            "human_readable" : "2.5 KB"
        },
        "chunk_size" : 512,
        "free_blocks" : 316,
        "offset" : 464,
        "spare_memory" :
        {
            "bytes" : 161792,
            "human_readable" : "158.0 KB"
        }
    },
    "peak_active_client_count" : 14,
    "pid" : 28580,
    "request_begin_speed" :
    {
        "1h" :
        {
            "averaged_over" : "1 hour",
            "per" : "minute",
            "value" : 144.440
        },
        "1m" :
        {
            "averaged_over" : "1 minute",
            "per" : "minute",
            "value" : 182.170
        }
    },
    "server_state" : "ACTIVE",
    "total_bytes_consumed" : 2467229,
    "total_clients_accepted" : 1054,
    "total_requests_begun" : 1054,
    "turbocaching" :
    {
        "fetches" : 0,
        "hit_ratio" : null,
        "hits" : 0,
        "store_success_ratio" : null,
        "store_successes" : 0,
        "stores" : 0
    }
}

### Request handler config (thread 1)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 2)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 3)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 4)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 5)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 6)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 7)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Request handler config (thread 8)

{
    "accept_burst_count" : 32,
    "client_freelist_limit" : 1024,
    "data_buffer_dir" : "/tmp",
    "min_spare_clients" : 128,
    "request_freelist_limit" : 1024,
    "show_version_in_header" : true,
    "single_app_mode" : false,
    "start_reading_after_accept" : true,
    "stat_throttle_rate" : 10
}

### Pool state (simple)
----------- General information -----------
Max pool size : 50
App groups    : 3
Processes     : 23
Requests in top-level queue : 0

----------- Application groups -----------
/srv/portals/current/public:
  App root: /srv/portals/current
  Requests in queue: 0
  * PID: 29794   Sessions: 0       Processed: 31      Uptime: 5m 17s
    CPU: 1%      Memory  : 140M    Last used: 6s ago
    URL     : http://127.0.0.1:47878
    Password: Ol8OGwXCU5TgwLbo
  * PID: 29804   Sessions: 0       Processed: 0       Uptime: 5m 17s
    CPU: 0%      Memory  : 72M     Last used: 5m 17s ago
    URL     : http://127.0.0.1:42470
    Password: Ol8OGwXCU5TgwLbo

/srv/piedpiper/current/public:
  App root: /srv/piedpiper/current
  Requests in queue: 0
  * PID: 28616   Sessions: 1       Processed: 1144    Uptime: 5m 41s
    CPU: 22%     Memory  : 241M    Last used: 0s ago
    URL     : http://127.0.0.1:33582
    Password: alzftuPyrEUTXgL2
  * PID: 28882   Sessions: 1       Processed: 1098    Uptime: 5m 34s
    CPU: 21%     Memory  : 55M     Last used: 0s ago
    URL     : http://127.0.0.1:51254
    Password: alzftuPyrEUTXgL2
  * PID: 29080   Sessions: 0       Processed: 863     Uptime: 5m 28s
    CPU: 16%     Memory  : 240M    Last used: 1s ago
    URL     : http://127.0.0.1:55297
    Password: alzftuPyrEUTXgL2
  * PID: 29380   Sessions: 1       Processed: 599     Uptime: 5m 22s
    CPU: 12%     Memory  : 242M    Last used: 1s ago
    URL     : http://127.0.0.1:36184
    Password: alzftuPyrEUTXgL2
  * PID: 29541   Sessions: 0       Processed: 389     Uptime: 5m 17s
    CPU: 9%      Memory  : 238M    Last used: 1s ago
    URL     : http://127.0.0.1:49947
    Password: alzftuPyrEUTXgL2
  * PID: 29841   Sessions: 0       Processed: 225     Uptime: 5m 10s
    CPU: 5%      Memory  : 241M    Last used: 1s ago
    URL     : http://127.0.0.1:51225
    Password: alzftuPyrEUTXgL2
  * PID: 30053   Sessions: 0       Processed: 86      Uptime: 5m 5s
    CPU: 3%      Memory  : 238M    Last used: 1s ago
    URL     : http://127.0.0.1:43115
    Password: alzftuPyrEUTXgL2
  * PID: 30341   Sessions: 0       Processed: 29      Uptime: 4m 45s
    CPU: 2%      Memory  : 238M    Last used: 2s ago
    URL     : http://127.0.0.1:42400
    Password: alzftuPyrEUTXgL2
  * PID: 30649   Sessions: 0       Processed: 10      Uptime: 4m 27s
    CPU: 2%      Memory  : 237M    Last used: 13s ago
    URL     : http://127.0.0.1:34719
    Password: alzftuPyrEUTXgL2
  * PID: 31740   Sessions: 0       Processed: 4       Uptime: 2m 56s
    CPU: 3%      Memory  : 237M    Last used: 13s ago
    URL     : http://127.0.0.1:57264
    Password: alzftuPyrEUTXgL2
  * PID: 827     Sessions: 0       Processed: 1       Uptime: 1m 43s
    CPU: 5%      Memory  : 240M    Last used: 13s ago
    URL     : http://127.0.0.1:38925
    Password: alzftuPyrEUTXgL2
  * PID: 2042    Sessions: 0       Processed: 0       Uptime: 7s
    CPU: 62%     Memory  : 236M    Last used: 7s ago
    URL     : http://127.0.0.1:56086
    Password: alzftuPyrEUTXgL2

/srv/assets/current/public:
  App root: /srv/assets/current
  Requests in queue: 0
  * PID: 28611   Sessions: 0       Processed: 2402    Uptime: 5m 42s
    CPU: 16%     Memory  : 97M     Last used: 0s ago
    URL     : http://127.0.0.1:53306
    Password: Dsim6dBNi9wdsdKN
  * PID: 28801   Sessions: 0       Processed: 1007    Uptime: 5m 38s
    CPU: 8%      Memory  : 98M     Last used: 1s ago
    URL     : http://127.0.0.1:56174
    Password: Dsim6dBNi9wdsdKN
  * PID: 28982   Sessions: 0       Processed: 377     Uptime: 5m 34s
    CPU: 3%      Memory  : 95M     Last used: 2s ago
    URL     : http://127.0.0.1:57738
    Password: Dsim6dBNi9wdsdKN
  * PID: 29163   Sessions: 0       Processed: 117     Uptime: 5m 30s
    CPU: 2%      Memory  : 94M     Last used: 7s ago
    URL     : http://127.0.0.1:46674
    Password: Dsim6dBNi9wdsdKN
  * PID: 29285   Sessions: 0       Processed: 36      Uptime: 5m 26s
    CPU: 1%      Memory  : 90M     Last used: 24s ago
    URL     : http://127.0.0.1:36250
    Password: Dsim6dBNi9wdsdKN
  * PID: 30831   Sessions: 0       Processed: 10      Uptime: 4m 20s
    CPU: 1%      Memory  : 84M     Last used: 1m 13s ago
    URL     : http://127.0.0.1:60219
    Password: Dsim6dBNi9wdsdKN
  * PID: 31272   Sessions: 0       Processed: 4       Uptime: 3m 34s
    CPU: 2%      Memory  : 83M     Last used: 1m 13s ago
    URL     : http://127.0.0.1:47879
    Password: Dsim6dBNi9wdsdKN
  * PID: 32491   Sessions: 0       Processed: 2       Uptime: 2m 33s
    CPU: 2%      Memory  : 84M     Last used: 1m 13s ago
    URL     : http://127.0.0.1:50087
    Password: Dsim6dBNi9wdsdKN
  * PID: 626     Sessions: 0       Processed: 0       Uptime: 1m 54s
    CPU: 3%      Memory  : 77M     Last used: 1m 54s ago
    URL     : http://127.0.0.1:50511
    Password: Dsim6dBNi9wdsdKN


### mbuf stats

nfree_mbuf_blockq  : 313
nactive_mbuf_blockq: 3
mbuf_block_chunk_size: 512

### Pool state (XML)
<?xml version="1.0" encoding="iso8859-1" ?>
<info version="3"><passenger_version>5.0.27</passenger_version><group_count>3</group_count><process_count>23</process_count><max>50</max><capacity_used>23</capacity_used><get_wait_list_size>0</get_wait_list_size><get_wait_list></get_wait_list><supergroups><supergroup><name>/srv/portals/current/public</name><state>READY</state><get_wait_list_size>0</get_wait_list_size><capacity_used>2</capacity_used><secret>Ol8OGwXCU5TgwLbo</secret><group default="true"><name>/srv/portals/current/public</name><component_name>/srv/portals/current/public</component_name><app_root>/srv/portals/current</app_root><app_type>rack</app_type><environment>production</environment><uuid>1Bg9SS9kFBt4QYySMtYt</uuid><enabled_process_count>2</enabled_process_count><disabling_process_count>0</disabling_process_count><disabled_process_count>0</disabled_process_count><capacity_used>2</capacity_used><get_wait_list_size>0</get_wait_list_size><disable_wait_list_size>0</disable_wait_list_size><processes_being_spawned>0</processes_being_spawned><secret>Ol8OGwXCU5TgwLbo</secret><api_key>Ol8OGwXCU5TgwLbo</api_key><life_status>ALIVE</life_status><user>piedpiper</user><uid>1000</uid><group>piedpiper</group><gid>1000</gid><options><app_root>/srv/portals/current</app_root><app_group_name>/srv/portals/current/public</app_group_name><app_type>rack</app_type><start_command>/home/piedpiper/.rvm/gems/ruby-2.1.1/wrappers/ruby&#9;/usr/share/passenger/helper-scripts/rack-loader.rb</start_command><startup_file>/srv/portals/current/config.ru</startup_file><process_title>Passenger RubyApp</process_title><log_level>3</log_level><start_timeout>90000</start_timeout><environment>production</environment><base_uri>/</base_uri><spawn_method>smart</spawn_method><default_user>nobody</default_user><default_group>nogroup</default_group><integration_mode>nginx</integration_mode><ruby>/home/piedpiper/.rvm/gems/ruby-2.1.1/wrappers/ruby</ruby><python>python</python><nodejs>node</nodejs><ust_router_address>unix:/tmp/passenger.1uP1fgj/agents.s/ust_router</ust_router_address><ust_router_username>logging</ust_router_username><ust_router_password>t2GJBdr5nWbnQMk8GRzoFiOc</ust_router_password><debugger>false</debugger><analytics>false</analytics><api_key>Ol8OGwXCU5TgwLbo</api_key><min_processes>2</min_processes><max_processes>0</max_processes><max_preloader_idle_time>300</max_preloader_idle_time><max_out_of_band_work_instances>1</max_out_of_band_work_instances></options><processes><process><pid>29794</pid><sticky_session_id>1907340957</sticky_session_id><gupid>1734e4f-wlt7AiL450</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>31</processed><spawner_creation_time>1460034189502621</spawner_creation_time><spawn_start_time>1460034192071354</spawn_start_time><spawn_end_time>1460034192090197</spawn_end_time><last_used>1460034503644880</last_used><last_used_desc>6s ago</last_used_desc><uptime>5m 17s</uptime><code_revision>4cbd782</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>1</cpu><rss>218296</rss><pss>180165</pss><private_dirty>143992</private_dirty><swap>0</swap><real_memory>143992</real_memory><vmsize>762872</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/portals/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.nqFZo5dwp3mjo6vakiXGhfAQQyab88sanPdvSAv3y3u3OdvfP2yjkkC6iCWKKuP</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:47878</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29804</pid><sticky_session_id>387385906</sticky_session_id><gupid>1734e4f-MyXJSHIac2</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>0</processed><spawner_creation_time>1460034189502621</spawner_creation_time><spawn_start_time>1460034192148573</spawn_start_time><spawn_end_time>1460034192166233</spawn_end_time><last_used>1460034192166233</last_used><last_used_desc>5m 17s ago</last_used_desc><uptime>5m 17s</uptime><code_revision>4cbd782</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>0</cpu><rss>147208</rss><pss>109623</pss><private_dirty>74460</private_dirty><swap>0</swap><real_memory>74460</real_memory><vmsize>627680</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/portals/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.U8aIYvLCK6wzwYj1AxehFk1dE4rnQFYKm2jvLmgK0z9WeohP0DBhHlvgfJFfFSi</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:42470</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process></processes></group></supergroup><supergroup><name>/srv/piedpiper/current/public</name><state>READY</state><get_wait_list_size>0</get_wait_list_size><capacity_used>12</capacity_used><secret>alzftuPyrEUTXgL2</secret><group default="true"><name>/srv/piedpiper/current/public</name><component_name>/srv/piedpiper/current/public</component_name><app_root>/srv/piedpiper/current</app_root><app_type>rack</app_type><environment>production</environment><uuid>boYegTE5EWAwu1ZEAkfm</uuid><enabled_process_count>12</enabled_process_count><disabling_process_count>0</disabling_process_count><disabled_process_count>0</disabled_process_count><capacity_used>12</capacity_used><get_wait_list_size>0</get_wait_list_size><disable_wait_list_size>0</disable_wait_list_size><processes_being_spawned>0</processes_being_spawned><secret>alzftuPyrEUTXgL2</secret><api_key>alzftuPyrEUTXgL2</api_key><life_status>ALIVE</life_status><user>piedpiper</user><uid>1000</uid><group>piedpiper</group><gid>1000</gid><options><app_root>/srv/piedpiper/current</app_root><app_group_name>/srv/piedpiper/current/public</app_group_name><app_type>rack</app_type><start_command>/home/piedpiper/.rvm/gems/ruby-1.8.7-p370/wrappers/ruby&#9;/usr/share/passenger/helper-scripts/rack-loader.rb</start_command><startup_file>/srv/piedpiper/current/config.ru</startup_file><process_title>Passenger RubyApp</process_title><log_level>3</log_level><start_timeout>90000</start_timeout><environment>production</environment><base_uri>/</base_uri><spawn_method>direct</spawn_method><default_user>nobody</default_user><default_group>nogroup</default_group><integration_mode>nginx</integration_mode><ruby>/home/piedpiper/.rvm/gems/ruby-1.8.7-p370/wrappers/ruby</ruby><python>python</python><nodejs>node</nodejs><ust_router_address>unix:/tmp/passenger.1uP1fgj/agents.s/ust_router</ust_router_address><ust_router_username>logging</ust_router_username><ust_router_password>t2GJBdr5nWbnQMk8GRzoFiOc</ust_router_password><debugger>false</debugger><analytics>false</analytics><api_key>alzftuPyrEUTXgL2</api_key><min_processes>5</min_processes><max_processes>0</max_processes><max_preloader_idle_time>300</max_preloader_idle_time><max_out_of_band_work_instances>1</max_out_of_band_work_instances></options><processes><process><pid>28616</pid><sticky_session_id>1147769522</sticky_session_id><gupid>1734e4e-sT7bi5Yd7P</gupid><concurrency>1</concurrency><sessions>1</sessions><busyness>2147483647</busyness><processed>1144</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034161338182</spawn_start_time><spawn_end_time>1460034168987485</spawn_end_time><last_used>1460034509075818</last_used><last_used_desc>0s ago</last_used_desc><uptime>5m 41s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>22</cpu><rss>255544</rss><pss>248167</pss><private_dirty>247704</private_dirty><swap>0</swap><real_memory>247704</real_memory><vmsize>516120</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.VlIACqmLAzBr6iONPdeZRdnuqJjuR8SXDLt6DnslzwUHIH9pkc0X9SGOUQcnJL4</address><protocol>session</protocol><concurrency>1</concurrency><sessions>1</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:33582</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>28882</pid><sticky_session_id>205145316</sticky_session_id><gupid>1734e4e-o5wXNU2aWE</gupid><concurrency>1</concurrency><sessions>1</sessions><busyness>2147483647</busyness><processed>1098</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034168988314</spawn_start_time><spawn_end_time>1460034175307485</spawn_end_time><last_used>1460034509137226</last_used><last_used_desc>0s ago</last_used_desc><uptime>5m 34s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>21</cpu><rss>255776</rss><pss>153099</pss><private_dirty>57316</private_dirty><swap>0</swap><real_memory>57316</real_memory><vmsize>516624</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.tBqrKQeaDuDflbo6xdxPjveLFCX43GAcJMfzxupSkS3DdU2QdtuXjoZLuvPsC9D</address><protocol>session</protocol><concurrency>1</concurrency><sessions>1</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:51254</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29080</pid><sticky_session_id>836435173</sticky_session_id><gupid>1734e4e-BOaR5jZbb8</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>863</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034175308397</spawn_start_time><spawn_end_time>1460034181226241</spawn_end_time><last_used>1460034508950817</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 28s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>16</cpu><rss>254472</rss><pss>247099</pss><private_dirty>246628</private_dirty><swap>0</swap><real_memory>246628</real_memory><vmsize>514372</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.UCJb7sjToo4PTlT87GqXtVREluejAs2dDsm2mIRGipzGSaKFSkqbBg1BupLubOO</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:55297</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29380</pid><sticky_session_id>1511737890</sticky_session_id><gupid>1734e4f-WHCU6B27UH</gupid><concurrency>1</concurrency><sessions>1</sessions><busyness>2147483647</busyness><processed>599</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034181227100</spawn_start_time><spawn_end_time>1460034187076457</spawn_end_time><last_used>1460034508984267</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 22s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>12</cpu><rss>255768</rss><pss>248470</pss><private_dirty>248048</private_dirty><swap>0</swap><real_memory>248048</real_memory><vmsize>515672</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.6xfFrPQhV58iiQ2ahncFuyYzQwscLtHjRzoOqGueK2GDQ1qa3tG76NT7asjQbMW</address><protocol>session</protocol><concurrency>1</concurrency><sessions>1</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:36184</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29541</pid><sticky_session_id>1699461801</sticky_session_id><gupid>1734e4f-hFmx64IYrS</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>389</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034187077109</spawn_start_time><spawn_end_time>1460034192997591</spawn_end_time><last_used>1460034508611582</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 17s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>9</cpu><rss>252224</rss><pss>244920</pss><private_dirty>244496</private_dirty><swap>0</swap><real_memory>244496</real_memory><vmsize>512236</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.979yazxkCYFnBzklWCap9TLd1OTm2acokv53f1vJ35yChLUqIzPQ7SBMORYazvE</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:49947</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29841</pid><sticky_session_id>93839976</sticky_session_id><gupid>1734e4f-5WGeBqwh31</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>225</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034192998388</spawn_start_time><spawn_end_time>1460034199125300</spawn_end_time><last_used>1460034508638243</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 10s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>5</cpu><rss>255276</rss><pss>248154</pss><private_dirty>247768</private_dirty><swap>0</swap><real_memory>247768</real_memory><vmsize>515336</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.Zl5H8LyP3oe02CmRr8OxwzvhvuFqNqZu4wYh0Bd7JeRKBT6iUrx5bHQykikLqQ1</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:51225</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>30053</pid><sticky_session_id>1195154836</sticky_session_id><gupid>1734e4f-WAu7t8RAGG</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>86</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034199126245</spawn_start_time><spawn_end_time>1460034204880009</spawn_end_time><last_used>1460034508649844</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 5s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>3</cpu><rss>251456</rss><pss>244592</pss><private_dirty>244204</private_dirty><swap>0</swap><real_memory>244204</real_memory><vmsize>511876</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.vJyVaVZvdaDD3wCKhEiimu2Q08GRRXQp6gnjKhH49dGRXaskkAsp1yas522uN1P</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:43115</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>30341</pid><sticky_session_id>1542455619</sticky_session_id><gupid>1734e4f-hfNr8temR5</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>29</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034218292966</spawn_start_time><spawn_end_time>1460034224136214</spawn_end_time><last_used>1460034507702825</last_used><last_used_desc>2s ago</last_used_desc><uptime>4m 45s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>2</cpu><rss>251028</rss><pss>244114</pss><private_dirty>243756</private_dirty><swap>0</swap><real_memory>243756</real_memory><vmsize>511512</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.dmpgmxpf0AsclxpztHSu1YFTqPc1hDtI038vB2AuFzSR8LcEmifj9KLmEuIshT8</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:42400</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>30649</pid><sticky_session_id>1970717670</sticky_session_id><gupid>1734e4f-kwkRCzZDBM</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>10</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034236483732</spawn_start_time><spawn_end_time>1460034242905155</spawn_end_time><last_used>1460034496314730</last_used><last_used_desc>13s ago</last_used_desc><uptime>4m 27s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>2</cpu><rss>250420</rss><pss>243778</pss><private_dirty>243432</private_dirty><swap>0</swap><real_memory>243432</real_memory><vmsize>508812</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.rsfSWOAhX745aaoInmLK1MzuYa78VkUq42qN3xjsAMerbI0dFVqQuxxR5RfFkEk</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:34719</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>31740</pid><sticky_session_id>319550609</sticky_session_id><gupid>1734e51-JfajNuzuwH</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>4</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034327140536</spawn_start_time><spawn_end_time>1460034333080673</spawn_end_time><last_used>1460034496324808</last_used><last_used_desc>13s ago</last_used_desc><uptime>2m 56s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>3</cpu><rss>249476</rss><pss>243018</pss><private_dirty>242696</private_dirty><swap>0</swap><real_memory>242696</real_memory><vmsize>508088</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.vOcXkYtmRMcbBEd0rT5pO1MltqNMovOaXBGrsJ1H3e4a0bKcinLAucv16ScbgqL</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:57264</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>827</pid><sticky_session_id>554078224</sticky_session_id><gupid>1734e52-1gkZfJ0Zu3</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>1</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034400055334</spawn_start_time><spawn_end_time>1460034406290173</spawn_end_time><last_used>1460034496368837</last_used><last_used_desc>13s ago</last_used_desc><uptime>1m 43s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>5</cpu><rss>253088</rss><pss>246906</pss><private_dirty>246612</private_dirty><swap>0</swap><real_memory>246612</real_memory><vmsize>512124</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.b9IDrQYx6DofvHH7Im6zNDO0JKIYgMNzNifqj1fWNmWkbEw8TJfICDkUT96Um6i</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:38925</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>2042</pid><sticky_session_id>2137707450</sticky_session_id><gupid>1734e54-HD27bHZrdu</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>0</processed><spawner_creation_time>1460034161325623</spawner_creation_time><spawn_start_time>1460034496387206</spawn_start_time><spawn_end_time>1460034502258439</spawn_end_time><last_used>1460034502258439</last_used><last_used_desc>7s ago</last_used_desc><uptime>7s</uptime><code_revision>e4c0861a3b91be6e0350579ef5a29ddc2f0f4d07</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>62</cpu><rss>248908</rss><pss>242784</pss><private_dirty>242496</private_dirty><swap>0</swap><real_memory>242496</real_memory><vmsize>507892</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/piedpiper/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.HRWUo7qw72cO8lgHyB9thG8NPhItlbP78ZFZwwRNSUlWRdUFe66oBMgBAlZ2Iqz</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:56086</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process></processes></group></supergroup><supergroup><name>/srv/assets/current/public</name><state>READY</state><get_wait_list_size>0</get_wait_list_size><capacity_used>9</capacity_used><secret>Dsim6dBNi9wdsdKN</secret><group default="true"><name>/srv/assets/current/public</name><component_name>/srv/assets/current/public</component_name><app_root>/srv/assets/current</app_root><app_type>rack</app_type><environment>production</environment><uuid>jPYGpGi4vaz1nTBaMPuR</uuid><enabled_process_count>9</enabled_process_count><disabling_process_count>0</disabling_process_count><disabled_process_count>0</disabled_process_count><capacity_used>9</capacity_used><get_wait_list_size>0</get_wait_list_size><disable_wait_list_size>0</disable_wait_list_size><processes_being_spawned>0</processes_being_spawned><secret>Dsim6dBNi9wdsdKN</secret><api_key>Dsim6dBNi9wdsdKN</api_key><life_status>ALIVE</life_status><user>piedpiper</user><uid>1000</uid><group>piedpiper</group><gid>1000</gid><options><app_root>/srv/assets/current</app_root><app_group_name>/srv/assets/current/public</app_group_name><app_type>rack</app_type><start_command>/home/piedpiper/.rvm/gems/ruby-1.9.3-p551/wrappers/ruby&#9;/usr/share/passenger/helper-scripts/rack-loader.rb</start_command><startup_file>/srv/assets/current/config.ru</startup_file><process_title>Passenger RubyApp</process_title><log_level>3</log_level><start_timeout>90000</start_timeout><environment>production</environment><base_uri>/</base_uri><spawn_method>direct</spawn_method><default_user>nobody</default_user><default_group>nogroup</default_group><integration_mode>nginx</integration_mode><ruby>/home/piedpiper/.rvm/gems/ruby-1.9.3-p551/wrappers/ruby</ruby><python>python</python><nodejs>node</nodejs><ust_router_address>unix:/tmp/passenger.1uP1fgj/agents.s/ust_router</ust_router_address><ust_router_username>logging</ust_router_username><ust_router_password>t2GJBdr5nWbnQMk8GRzoFiOc</ust_router_password><debugger>false</debugger><analytics>false</analytics><api_key>Dsim6dBNi9wdsdKN</api_key><min_processes>5</min_processes><max_processes>0</max_processes><max_preloader_idle_time>300</max_preloader_idle_time><max_out_of_band_work_instances>1</max_out_of_band_work_instances></options><processes><process><pid>28611</pid><sticky_session_id>2074224666</sticky_session_id><gupid>1734e4e-QTafa2yv3Q</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>2402</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034161309905</spawn_start_time><spawn_end_time>1460034167194438</spawn_end_time><last_used>1460034509117154</last_used><last_used_desc>0s ago</last_used_desc><uptime>5m 42s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>16</cpu><rss>108848</rss><pss>100652</pss><private_dirty>100032</private_dirty><swap>0</swap><real_memory>100032</real_memory><vmsize>668844</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.Ezcbpj598kBYY1Nx23d6YbTu9DYbNb0ggSkP6PnZOkPCbC1eopEaWG3GL8CIzSV</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:53306</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>28801</pid><sticky_session_id>805413509</sticky_session_id><gupid>1734e4e-UX4AfGVK3G</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>1007</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034167195192</spawn_start_time><spawn_end_time>1460034171473155</spawn_end_time><last_used>1460034508833315</last_used><last_used_desc>1s ago</last_used_desc><uptime>5m 38s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>8</cpu><rss>109172</rss><pss>101108</pss><private_dirty>100488</private_dirty><swap>0</swap><real_memory>100488</real_memory><vmsize>668596</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.P3yDxLQa3FPOgtnT6DdkddSyIATn74vty7jbA79iO3hFGZohrOqNGolw1PRkiX6</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:56174</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>28982</pid><sticky_session_id>1886360986</sticky_session_id><gupid>1734e4e-8fVvFNqm1K</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>377</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034171474013</spawn_start_time><spawn_end_time>1460034175529790</spawn_end_time><last_used>1460034507518051</last_used><last_used_desc>2s ago</last_used_desc><uptime>5m 34s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>3</cpu><rss>106564</rss><pss>98382</pss><private_dirty>97768</private_dirty><swap>0</swap><real_memory>97768</real_memory><vmsize>668596</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.vHm6lYRZbaNLC5T9028I4E40pxzTiGBUrwTtXFVVHMW2KScSbCzlTO9P93qxZyX</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:57738</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29163</pid><sticky_session_id>887155047</sticky_session_id><gupid>1734e4e-qwyqMZyWDX</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>117</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034175530525</spawn_start_time><spawn_end_time>1460034179709612</spawn_end_time><last_used>1460034502552717</last_used><last_used_desc>7s ago</last_used_desc><uptime>5m 30s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>2</cpu><rss>105116</rss><pss>96870</pss><private_dirty>96256</private_dirty><swap>0</swap><real_memory>96256</real_memory><vmsize>668600</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.3iKloSxHg3o1zsXxHDNmfO7R4zsiJIW5ofJqh2WxL3atRqAZXMqeqY7X1s5QxD6</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:46674</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>29285</pid><sticky_session_id>1408309555</sticky_session_id><gupid>1734e4e-zZzmQ2Yto6</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>36</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034179710273</spawn_start_time><spawn_end_time>1460034183913598</spawn_end_time><last_used>1460034485535565</last_used><last_used_desc>24s ago</last_used_desc><uptime>5m 26s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>1</cpu><rss>101504</rss><pss>93503</pss><private_dirty>92892</private_dirty><swap>0</swap><real_memory>92892</real_memory><vmsize>668600</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.srSFVjllvEcAqCt0azwfy5WHszhZW5hFt58HMyY1X2Fxwt9VGjEl64b7XZ4JgVx</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:36250</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>30831</pid><sticky_session_id>174120964</sticky_session_id><gupid>1734e50-GI0n5QzHcj</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>10</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034245237412</spawn_start_time><spawn_end_time>1460034249406292</spawn_end_time><last_used>1460034436293771</last_used><last_used_desc>1m 13s ago</last_used_desc><uptime>4m 20s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>1</cpu><rss>95348</rss><pss>87415</pss><private_dirty>86804</private_dirty><swap>0</swap><real_memory>86804</real_memory><vmsize>668600</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.3LYoSNTEa00I9urLaSdkRBPfbkcPjcRKbcOMqTABwvzc6iZwfzMC5QuBGkex1S4</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:60219</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>31272</pid><sticky_session_id>85928402</sticky_session_id><gupid>1734e50-vwc0v78uqi</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>4</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034290917686</spawn_start_time><spawn_end_time>1460034295205240</spawn_end_time><last_used>1460034436338969</last_used><last_used_desc>1m 13s ago</last_used_desc><uptime>3m 34s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>2</cpu><rss>94564</rss><pss>86506</pss><private_dirty>85896</private_dirty><swap>0</swap><real_memory>85896</real_memory><vmsize>668600</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.dsl7lI9t1BsD690Dx75XRmhL59RdlJd1SoFUroEI7lFm1aNfy642f9SnXuIVUVG</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:47879</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>32491</pid><sticky_session_id>1699195757</sticky_session_id><gupid>1734e51-Jmn8diebz9</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>2</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034351156946</spawn_start_time><spawn_end_time>1460034356056661</spawn_end_time><last_used>1460034436340955</last_used><last_used_desc>1m 13s ago</last_used_desc><uptime>2m 33s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>2</cpu><rss>94872</rss><pss>86667</pss><private_dirty>86052</private_dirty><swap>0</swap><real_memory>86052</real_memory><vmsize>668608</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.qOzNO5gAUoj362CkPmVZxSE2hTOIW7a7RQpyeMMQi0FuoIkn5bf2oTr0n7srjfN</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:50087</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>626</pid><sticky_session_id>943456820</sticky_session_id><gupid>1734e52-1tbfWs7zwi</gupid><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>0</processed><spawner_creation_time>1460034161308980</spawner_creation_time><spawn_start_time>1460034391026230</spawn_start_time><spawn_end_time>1460034395124936</spawn_end_time><last_used>1460034395124936</last_used><last_used_desc>1m 54s ago</last_used_desc><uptime>1m 54s</uptime><code_revision>48e0ec0629482e8365256f126e72a816db29b9b9</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>3</cpu><rss>88492</rss><pss>80350</pss><private_dirty>79740</private_dirty><swap>0</swap><real_memory>79740</real_memory><vmsize>668600</vmsize><process_group_id>28580</process_group_id><command>Passenger RubyApp: /srv/assets/current/public</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1uP1fgj/apps.s/ruby.gQE0TfjVlWu1ixS7ZzlKiNb5mC6KuH5MiA78V7Cc4j2APW5CyDSFgplUeSb4jjO</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:50511</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process></processes></group></supergroup></supergroups></info>

--------------------------------------
[ pid=28580 ] Open files and file descriptors:
COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF       NODE NAME
Passenger 28580 root  cwd    DIR                9,2     4096          2 /
Passenger 28580 root  rtd    DIR                9,2     4096          2 /
Passenger 28580 root  txt    REG                9,2  4358856  103024733 /usr/lib/passenger/support-binaries/PassengerAgent (deleted)
Passenger 28580 root  mem    REG                9,2    47712   89657060 /lib/x86_64-linux-gnu/libnss_files-2.19.so
Passenger 28580 root  mem    REG                9,2    47760   89656978 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
Passenger 28580 root  mem    REG                9,2    97296   89656837 /lib/x86_64-linux-gnu/libnsl-2.19.so
Passenger 28580 root  mem    REG                9,2    39824   89656836 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
Passenger 28580 root  mem    REG                9,2    43368   89656834 /lib/x86_64-linux-gnu/libcrypt-2.19.so
Passenger 28580 root  mem    REG                9,2   754880  101588317 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
Passenger 28580 root  mem    REG                9,2   295816  101581226 /usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
Passenger 28580 root  mem    REG                9,2    56768  101581339 /usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
Passenger 28580 root  mem    REG                9,2   166040  101581247 /usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
Passenger 28580 root  mem    REG                9,2    30944  101581413 /usr/lib/x86_64-linux-gnu/libffi.so.6.0.1
Passenger 28580 root  mem    REG                9,2    85680  101581370 /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
Passenger 28580 root  mem    REG                9,2   204104  101581316 /usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
Passenger 28580 root  mem    REG                9,2   657712  101581350 /usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
Passenger 28580 root  mem    REG                9,2   552584  101581262 /usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
Passenger 28580 root  mem    REG                9,2    35856  101581256 /usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
Passenger 28580 root  mem    REG                9,2    14256   89656478 /lib/x86_64-linux-gnu/libkeyutils.so.1.4
Passenger 28580 root  mem    REG                9,2    18416   89656530 /lib/x86_64-linux-gnu/libgpg-error.so.0.10.0
Passenger 28580 root  mem    REG                9,2   268368  101581328 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.0.0
Passenger 28580 root  mem    REG                9,2    80256  101581381 /usr/lib/x86_64-linux-gnu/libtasn1.so.6.2.0
Passenger 28580 root  mem    REG                9,2   251872  101581223 /usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
Passenger 28580 root  mem    REG                9,2   109296  101581202 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
Passenger 28580 root  mem    REG                9,2   101240   89656610 /lib/x86_64-linux-gnu/libresolv-2.19.so
Passenger 28580 root  mem    REG                9,2    43672  101581248 /usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
Passenger 28580 root  mem    REG                9,2    14592   89656723 /lib/x86_64-linux-gnu/libcom_err.so.2.1
Passenger 28580 root  mem    REG                9,2   186824  101584726 /usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
Passenger 28580 root  mem    REG                9,2   831616  101581346 /usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
Passenger 28580 root  mem    REG                9,2    14664   89656831 /lib/x86_64-linux-gnu/libdl-2.19.so
Passenger 28580 root  mem    REG                9,2   524704   89656514 /lib/x86_64-linux-gnu/libgcrypt.so.11.8.2
Passenger 28580 root  mem    REG                9,2   775112  101580953 /usr/lib/x86_64-linux-gnu/libgnutls.so.26.22.6
Passenger 28580 root  mem    REG                9,2   323056  101581024 /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.8.3
Passenger 28580 root  mem    REG                9,2    59640  101581026 /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.8.3
Passenger 28580 root  mem    REG                9,2   290520  101581220 /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
Passenger 28580 root  mem    REG                9,2  1934624   89657205 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Passenger 28580 root  mem    REG                9,2   387256   89657204 /lib/x86_64-linux-gnu/libssl.so.1.0.0
Passenger 28580 root  mem    REG                9,2   105696  101581349 /usr/lib/x86_64-linux-gnu/librtmp.so.0
Passenger 28580 root  mem    REG                9,2   207128  101581246 /usr/lib/x86_64-linux-gnu/libidn.so.11.6.11
Passenger 28580 root  mem    REG                9,2  1840928   89657054 /lib/x86_64-linux-gnu/libc-2.19.so
Passenger 28580 root  mem    REG                9,2    90160   89656458 /lib/x86_64-linux-gnu/libgcc_s.so.1
Passenger 28580 root  mem    REG                9,2  1071552   89656829 /lib/x86_64-linux-gnu/libm-2.19.so
Passenger 28580 root  mem    REG                9,2   979056  101585650 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19
Passenger 28580 root  mem    REG                9,2   100728   89656575 /lib/x86_64-linux-gnu/libz.so.1.2.8
Passenger 28580 root  mem    REG                9,2   422696  101584867 /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
Passenger 28580 root  mem    REG                9,2   141574   89656975 /lib/x86_64-linux-gnu/libpthread-2.19.so
Passenger 28580 root  mem    REG                9,2    31792   89656984 /lib/x86_64-linux-gnu/librt-2.19.so
Passenger 28580 root  mem    REG                9,2   149120   89656976 /lib/x86_64-linux-gnu/ld-2.19.so
Passenger 28580 root    0u   CHR              136,2      0t0          5 /dev/pts/2 (deleted)
Passenger 28580 root    1w  FIFO                0,8      0t0 3456735105 pipe
Passenger 28580 root    2w  FIFO                0,8      0t0 3456735105 pipe
Passenger 28580 root    3u  unix 0xffff8807ee4d3800      0t0 3456622232 socket
Passenger 28580 root    5w  FIFO                0,8      0t0 3456735105 pipe
Passenger 28580 root    8r  FIFO                0,8      0t0 3456625344 pipe
Passenger 28580 root    9w  FIFO                0,8      0t0 3456625344 pipe
Passenger 28580 root   10r  FIFO                0,8      0t0 3456625345 pipe
Passenger 28580 root   11w  FIFO                0,8      0t0 3456625345 pipe
Passenger 28580 root   12r  FIFO                0,8      0t0 3456625346 pipe
Passenger 28580 root   13w  FIFO                0,8      0t0 3456625346 pipe
Passenger 28580 root   14u  unix 0xffff8800d309ad80      0t0 3456625347 /tmp/passenger.1uP1fgj/agents.s/core
Passenger 28580 root   15u  unix 0xffff8800d309fa80      0t0 3456625348 /tmp/passenger.1uP1fgj/agents.s/core_api
Passenger 28580 root   16r   CHR                1,9      0t0       1034 /dev/urandom
Passenger 28580 root   17u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   18u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   19r  FIFO                0,8      0t0 3456618077 pipe
Passenger 28580 root   20w  FIFO                0,8      0t0 3456618077 pipe
Passenger 28580 root   21u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   22r  FIFO                0,8      0t0 3456618078 pipe
Passenger 28580 root   23w  FIFO                0,8      0t0 3456618078 pipe
Passenger 28580 root   24u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   25u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   26u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   27u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   28u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   29u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   30r  FIFO                0,8      0t0 3456618079 pipe
Passenger 28580 root   31w  FIFO                0,8      0t0 3456618079 pipe
Passenger 28580 root   32u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   33u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   34u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   35u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   36r  FIFO                0,8      0t0 3456618080 pipe
Passenger 28580 root   37w  FIFO                0,8      0t0 3456618080 pipe
Passenger 28580 root   38u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   39u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   40u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   41u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   42r  FIFO                0,8      0t0 3456618081 pipe
Passenger 28580 root   43w  FIFO                0,8      0t0 3456618081 pipe
Passenger 28580 root   44u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   45u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   46u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   47u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   48r  FIFO                0,8      0t0 3456618082 pipe
Passenger 28580 root   49w  FIFO                0,8      0t0 3456618082 pipe
Passenger 28580 root   50u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   51u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   52u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   53u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   54r  FIFO                0,8      0t0 3456618083 pipe
Passenger 28580 root   55w  FIFO                0,8      0t0 3456618083 pipe
Passenger 28580 root   56u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   57u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   58u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   59u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   60r  FIFO                0,8      0t0 3456618084 pipe
Passenger 28580 root   61w  FIFO                0,8      0t0 3456618084 pipe
Passenger 28580 root   62u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   63u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   64u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   65u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   66r  FIFO                0,8      0t0 3456618085 pipe
Passenger 28580 root   67w  FIFO                0,8      0t0 3456618085 pipe
Passenger 28580 root   68u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   69u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   70u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   71u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   72r  FIFO                0,8      0t0 3456618086 pipe
Passenger 28580 root   73w  FIFO                0,8      0t0 3456618086 pipe
Passenger 28580 root   74u  0000                0,9        0       7345 anon_inode
Passenger 28580 root   75u  unix 0xffff880306b0e900      0t0 3456742629 /tmp/passenger.1uP1fgj/agents.s/core
Passenger 28580 root   76u  unix 0xffff880306b0bb80      0t0 3456743665 socket
Passenger 28580 root   77u  unix 0xffff8800d309d780      0t0 3456625356 socket
Passenger 28580 root   78r  FIFO                0,8      0t0 3456625357 pipe
Passenger 28580 root   79u  unix 0xffff8801dda2c280      0t0 3456742625 /tmp/passenger.1uP1fgj/agents.s/core
Passenger 28580 root   81u  unix 0xffff880789b8fa80      0t0 3456625733 socket
Passenger 28580 root   82r  FIFO                0,8      0t0 3456625734 pipe
Passenger 28580 root   84u  unix 0xffff8800d3098380      0t0 3456672132 socket
Passenger 28580 root   85u  unix 0xffff8801dda2f700      0t0 3456741578 /tmp/passenger.1uP1fgj/agents.s/core
Passenger 28580 root   86u  unix 0xffff8800d309f380      0t0 3456629440 socket
Passenger 28580 root   87r  FIFO                0,8      0t0 3456629441 pipe
Passenger 28580 root   88u  unix 0xffff88077dcb4980      0t0 3456682879 socket
Passenger 28580 root   89u  unix 0xffff8802efc0d780      0t0 3456742627 /tmp/passenger.1uP1fgj/agents.s/core
Passenger 28580 root   90u  unix 0xffff88077dcb2300      0t0 3456653127 socket
Passenger 28580 root   91r  FIFO                0,8      0t0 3456653128 pipe
Passenger 28580 root   92u  unix 0xffff8807ee4d3f00      0t0 3456741571 socket
Passenger 28580 root   93u  unix 0xffff88077c7bb480      0t0 3456631060 socket
Passenger 28580 root   94u  unix 0xffff8807f0bd8700      0t0 3456740782 socket
Passenger 28580 root   96u  unix 0xffff88077dcb6580      0t0 3456642019 socket
Passenger 28580 root   97r  FIFO                0,8      0t0 3456642020 pipe
Passenger 28580 root   98u  unix 0xffff8801dda2ad80      0t0 3456704674 socket
Passenger 28580 root  100u  unix 0xffff880789841180      0t0 3456668328 socket
Passenger 28580 root  101u  unix 0xffff8800d3099f80      0t0 3456646476 socket
Passenger 28580 root  102r  FIFO                0,8      0t0 3456646477 pipe
Passenger 28580 root  103r  FIFO                0,8      0t0 3456668329 pipe
Passenger 28580 root  104u  unix 0xffff8800d3098e00      0t0 3456656667 socket
Passenger 28580 root  105r  FIFO                0,8      0t0 3456656668 pipe
Passenger 28580 root  107u  unix 0xffff8807f0bdd780      0t0 3456687062 socket
Passenger 28580 root  108r  FIFO                0,8      0t0 3456687063 pipe
Passenger 28580 root  109u  unix 0xffff8807ed458a80      0t0 3456703668 socket
Passenger 28580 root  110r  FIFO                0,8      0t0 3456703669 pipe
Passenger 28580 root  113u  unix 0xffff8800d309e580      0t0 3456681461 socket
Passenger 28580 root  114r  FIFO                0,8      0t0 3456681462 pipe
Passenger 28580 root  118u  unix 0xffff88077dcb3f00      0t0 3456641498 socket
Passenger 28580 root  121u  unix 0xffff8801dda2cd00      0t0 3456636240 socket
Passenger 28580 root  122r  FIFO                0,8      0t0 3456636241 pipe
Passenger 28580 root  123u  unix 0xffff8807f0bdd080      0t0 3456640044 socket
Passenger 28580 root  124r  FIFO                0,8      0t0 3456640045 pipe
Passenger 28580 root  129u  unix 0xffff88077c7b9880      0t0 3456706106 socket
Passenger 28580 root  130r  FIFO                0,8      0t0 3456706107 pipe
Passenger 28580 root  132u  unix 0xffff8802efc09c00      0t0 3456639198 socket
Passenger 28580 root  134u  unix 0xffff8807f0bd8000      0t0 3456737094 socket
Passenger 28580 root  135r  FIFO                0,8      0t0 3456737095 pipe
Passenger 28580 root  136u  unix 0xffff8800d3303480      0t0 3456627825 socket
Passenger 28580 root  137u  unix 0xffff8801dda2de80      0t0 3456618329 socket
Passenger 28580 root  150r  FIFO                0,8      0t0 3456618330 pipe
Passenger 28580 root  152u  unix 0xffff88077dcb3800      0t0 3456633285 socket
Passenger 28580 root  159u  unix 0xffff8800d309c980      0t0 3456629737 socket
Passenger 28580 root  161r  FIFO                0,8      0t0 3456629738 pipe
Passenger 28580 root  168u  unix 0xffff8801dda2f000      0t0 3456618369 socket
Passenger 28580 root  170r  FIFO                0,8      0t0 3456618370 pipe
Passenger 28580 root  176u  unix 0xffff8807899c1f80      0t0 3456626199 socket
Passenger 28580 root  178u  unix 0xffff880789078a80      0t0 3456618109 socket
Passenger 28580 root  179r  FIFO                0,8      0t0 3456618110 pipe
Passenger 28580 root  184u  unix 0xffff88077dcb7a80      0t0 3456633126 socket
Passenger 28580 root  189u  unix 0xffff8807899c4980      0t0 3456626201 socket
Passenger 28580 root  190r  FIFO                0,8      0t0 3456626202 pipe
Passenger 28580 root  191u  unix 0xffff8807899c6200      0t0 3456626489 socket
Passenger 28580 root  192r  FIFO                0,8      0t0 3456626490 pipe
Passenger 28580 root  195u  unix 0xffff8802efc0ec80      0t0 3456628108 socket
Passenger 28580 root  196r  FIFO                0,8      0t0 3456628109 pipe
Passenger 28580 root  197u  unix 0xffff8807899c4d00      0t0 3456632050 socket
--------------------------------------
[ pid=28580 ] Dumping a more detailed backtrace with crash-watch...
*** ERROR ***: 'gdb' isn't installed. Please install it first.
       Debian/Ubuntu: sudo apt-get install gdb
RedHat/CentOS/Fedora: sudo yum install gdb
            Mac OS X: please install the Developer Tools or XCode
             FreeBSD: use the devel/gdb port

or passenger.crash.log.zip

Please let me know if additional information is required.

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 30 (14 by maintainers)

Most upvoted comments

@OnixGH It’s not crashing anymore! Thanks for the support. I leave it running through the night and it will do about a million requests before the morning. If it doesn’t crash, we can consider it fixed.

@OnixGH Thanks. It will take me some time to apply the patch and test it. I’m currently using the APT repo version of Passenger and need to install the gem version or build the source I think. So I will need to reconfigure nginx to test the patch. I will try tomorrow.

@remomueller I think your issue is completely different and not related to Passenger, but instead to Xcode, see: #1780.