moped: Writes fail with ConnectionPool::PoolShuttingDownError

Something happens in our MongoDB configuration (likely a step down of a master or some kind of failure) that causes all servers to start reporting the following error: ConnectionPool::PoolShuttingDownError on database writes to primary.

This is recent, we’ve seen it take out all our writes twice.

Moped 2.0.3 Mongoid 4.0.0

…nnection_pool-2.1.1/lib/connection_pool/timed_stack.rb:  78:in `block (2 levels) in pop'
…nnection_pool-2.1.1/lib/connection_pool/timed_stack.rb:  77:in `loop'
…nnection_pool-2.1.1/lib/connection_pool/timed_stack.rb:  77:in `block in pop'
…nnection_pool-2.1.1/lib/connection_pool/timed_stack.rb:  76:in `synchronize'
…nnection_pool-2.1.1/lib/connection_pool/timed_stack.rb:  76:in `pop'
….0.0/gems/connection_pool-2.1.1/lib/connection_pool.rb:  78:in `checkout'
….0.0/gems/connection_pool-2.1.1/lib/connection_pool.rb:  60:in `with'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 114:in `connection'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 140:in `disconnect'
….0.0/gems/moped-2.0.3/lib/moped/failover/disconnect.rb:  26:in `execute'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 185:in `rescue in ensure_connected'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 188:in `ensure_connected'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 589:in `block in flush'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 617:in `block in logging'
…ctivesupport-4.1.8/lib/active_support/notifications.rb: 159:in `block in instrument'
…4.1.8/lib/active_support/notifications/instrumenter.rb:  20:in `instrument'
…ctivesupport-4.1.8/lib/active_support/notifications.rb: 159:in `instrument'
…uby/2.0.0/gems/moped-2.0.3/lib/moped/instrumentable.rb:  31:in `instrument'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 616:in `logging'
…lic_moped-0.0.12/lib/newrelic_moped/instrumentation.rb:  40:in `block in logging_with_newrelic_trace'
…lic_moped-0.0.12/lib/newrelic_moped/instrumentation.rb:  49:in `call'
…lic_moped-0.0.12/lib/newrelic_moped/instrumentation.rb:  49:in `block in logging_with_newrelic_trace'
…lic_moped-0.0.12/lib/newrelic_moped/instrumentation.rb:  46:in `logging_with_newrelic_trace'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 587:in `flush'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 358:in `pipeline'
…by/2.0.0/gems/moped-2.0.3/lib/moped/operation/write.rb:  47:in `execute'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 665:in `write'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 502:in `update'
…gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/query.rb: 427:in `block in update'
…avity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/cluster.rb: 249:in `block in with_primary'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 204:in `block in ensure_primary'
…ty/ruby/2.0.0/gems/moped-2.0.3/lib/moped/executable.rb:  25:in `execute'
…/gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/node.rb: 203:in `ensure_primary'
…avity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/cluster.rb: 248:in `with_primary'
…gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/query.rb: 426:in `update'
…by/2.0.0/gems/mongoid-4.0.0/lib/mongoid/query_cache.rb: 117:in `update_with_clear_cache'
…gravity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/query.rb: 449:in `update_all'
…by/2.0.0/gems/mongoid-4.0.0/lib/mongoid/query_cache.rb: 117:in `update_all_with_clear_cache'
….0/gems/mongoid-4.0.0/lib/mongoid/contextual/atomic.rb: 152:in `set'
…uby/2.0.0/gems/mongoid-4.0.0/lib/mongoid/contextual.rb:  16:in `set'

About this issue

Original URL
State: closed
Created 9 years ago
Comments: 46 (14 by maintainers)

Commits related to this issue

Fix #345 — committed to fedenusy/moped by fedenusy 9 years ago
Spec for #345. — committed to dblock/moped by dblock 9 years ago
Merge pull request #353 from dblock/fix-345 Fix + spec for #345. — committed to mongoid/moped by arthurnn 9 years ago
Fix #345 — committed to jonhyman/moped by fedenusy 9 years ago
Spec for #345. — committed to jonhyman/moped by dblock 9 years ago

Most upvoted comments

I have broken ReplicaSet were slave is DOWN and primary is OK. So got tons of ConnectionPool::PoolShuttingDownError.

I downgraded from moped 2.0.7 to gem 'mongoid', '4.0.0' gem 'moped', '2.0.1' And now no any errors ConnectionPool::PoolShuttingDownError.

nodkz on Oct 7, 2015

This issue is happening to us as well.

Mongoid (4.0.2) Moped (2.0.4)

Is there a particular old version that is not experiencing this error? I am thinking of just using an old version, since it seems to be (as per thread) regression.

ardeearam on Mar 3, 2015

Got this issue too since my last update of moped and mongoid. I haven’t changed my setup so this is probably a regression. I have a sidekiq process with 10 threads and a long running daemon with 10 threads My mongoid is configured with a pool size of 15.

jarthod on Feb 9, 2015