moped: Stuck in Moped::Errors::ConnectionFailure: Could not connect to a primary node for replica set

Moped::Errors::ConnectionFailure: Could not connect to a primary node for replica set #<Moped::Cluster:128953180 @seeds=[<Moped::Node resolved_address="10.95.128.244:27017">, <Moped::Node resolved_address="10.184.156.102:27017">]>
…avity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/cluster.rb: 254:in `with_primary'
…ty/ruby/2.0.0/gems/moped-2.0.3/lib/moped/collection.rb: 124:in `insert'
…by/2.0.0/gems/mongoid-4.0.0/lib/mongoid/query_cache.rb: 117:in `insert_with_clear_cache'
…ems/mongoid-4.0.0/lib/mongoid/persistable/creatable.rb:  79:in `insert_as_root'

Occasionally we see a machine or two stuck in this. I am not sure when this happens, but about 10% of nodes end up in this state every 24 hours. The MongoDB cluster is doing fine.

This issue could probably use more detail, please tell me what to look for next time I have a machine in this state.

About this issue

  • Original URL
  • State: open
  • Created 9 years ago
  • Comments: 29 (4 by maintainers)

Most upvoted comments

It could be that mongo is not running. Have you tried:

sudo rm /var/lib/mongodb/mongod.lock
sudo service mongodb start

@bastoune we used sidekiq for background jobs and puma. Both being multithreaded, supporting 25 and 16 threads by default. Now, mongoid by default has pool size as 5, evidently, there were situations wherein the poolsize got exhausted in this case resulting into
Moped::Errors::ConnectionFailure: Could not connect to a primary node for replica set #<Moped::Cluster:1223353180 @seeds=[<Moped::Node resolved_address="xx.xxx.xxx.xxx:27017">, <Moped::Node resolved_address="xx.xxx.xxx.xxx:27017">]>

Fixed it by tuning poolsize, sidekiq + puma threads. Here is an article for sql database though i suppose it clarifies the fundamentals