monstache: Frequently seeing "connection timed out" errors

Hey Ryan,

Seeing these error (below) messages every few minutes in monstache logs.
ERROR 2019/01/17 18:35:57 Unable to dial MongoDB: dial tcp <ip>:27017: connect: connection timed out ERROR 2019/01/17 18:35:57 Unable to dial MongoDB: dial tcp <ip>:27017: connect: connection timed out

My connection settings are: [mongo-dial-settings] ssl = true read-timeout = 300 write-timeout = 300

[mongo-session-settings] socket-timeout = 300 sync-timeout = 300

Ingestion in production is also relatively slower compared to our dev tests. In about ~14hrs, only about 1.2 million documents have been transferred from mongo to ES, whereas in dev it only took a few hours to transfer 2.5+ million documents. The only difference between the tests was adding a namespace-regex in prod. Is filtering performance expensive?

Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 35 (21 by maintainers)

Most upvoted comments

Definitely keep an eye on any errors in Elasticsearch. If they are in the set above then monstache will not drop them but instead keep them in the queue and since this presents back pressure to the ingestion from MongoDB, new events from MongoDB will not be processed.