monstache: Sync MongoDB to Elasticsearch via docker way not work.

I run monstache in my k8s cluster to sync mongo(inside k8s) to elasticsearch(outside k8s), it’s not work without any error, nothing synced to ES.

monstache log:

INFO 2018/05/10 09:04:55 Successfully connected to MongoDB version 3.6.4
INFO 2018/05/10 09:04:55 Successfully connected to Elasticsearch version 5.5.0
INFO 2018/05/10 09:04:55 Entering event loop
...
...
STATS 2018/05/10 09:25:48 {"Flushed":181,"Committed":20,"Indexed":185,"Created":0,"Updated":0,"Deleted":27,"Succeeded":51,"Failed":161,"Workers":[{"Queued":0,"LastDuration":4000000},{"Queued":0,"LastDuration":40000000},{"Queued":0,"LastDuration":1000000},{"Queued":0,"LastDuration":3000000}]}
STATS 2018/05/10 09:26:18 {"Flushed":211,"Committed":20,"Indexed":185,"Created":0,"Updated":0,"Deleted":27,"Succeeded":51,"Failed":161,"Workers":[{"Queued":0,"LastDuration":4000000},{"Queued":0,"LastDuration":40000000},{"Queued":0,"LastDuration":1000000},{"Queued":0,"LastDuration":3000000}]}

My env:

Mongo: replicaset (v3.6) in k8s cluster. Elasticsearch: v5.5 in isolate machine. Monstache: docker image: rwynn/monstache:3.13.3.cgo

My config.toml:

mongo-url = "mongodb://{user}:{password}@mongo-0.mongo.db,mongo-1.mongo.db,mongo-2.mongo.db"
elasticsearch-urls = ["http://{es.ip}:9200"]

gzip = true
stats = true
index-stats = false
fail-fast = true
resume = true
resume-write-unsafe = true
replay = true
elasticsearch-retry = true

dropped-collections = true
dropped-databases = true
mongo-oplog-collection-name = "oplog.rs"
namespace-regex = '^web-db\.(articles|\$cmd)$'
# direct-read-namespaces = ["web-db.articles"]

exit-after-direct-reads = false
enable-http-server = true

Data in monstache.monstache collection

{ 
    "_id" : "default", 
    "ts" : Timestamp(1525943995, 1)
}

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 18 (11 by maintainers)

Most upvoted comments

I will create a separate issue to figure out what to do with those error messages since they are not truly errors but the versioning system at work.