elasticsearch-rails: Model's settings and mappings are ignored when the index is created through Model.import

Hi all, I’m trying to use mappings defined in an AR model backed by postgres (rails 4.0.4) to create an index and its corresponding mappings. My model has the following:

    mappings dynamic: false do
      indexes :text
      indexes :neighborhood, index: 'not_analyzed'
    end

Unfortunately, when I run curl -XGET 'http://localhost:9200/my_index/_mapping?pretty=true'

{
  "my_index" : {
    "mappings" : {
      "my_type" : {
        "properties" : {
          "index" : {
            "properties" : {
              "_id" : {
                "type" : "long"
              },
              "_score" : {
                "type" : "long"
              },
              "neighborhood" : {
                "type" : "string"
              },
              "text" : {
                "type" : "string"
              }
            }
          }
        }
      }
    }
  }
}

Notice the lack of "index": "not_analyzed". Using the curl commands against ES directly, I was able to create the index needed.

This is all invoked using ModelName.import.

About this issue

Original URL
State: closed
Created 10 years ago
Reactions: 2
Comments: 17 (7 by maintainers)

Most upvoted comments

Thank you!

I checked all ways of creating index I know:

Model.__elasticsearch__.create_index!
Model.__elasticsearch__.create_index! force: true
Model.import force: true

Every time the index has been created with properly set mappings. The exeption was raised for Model.import without force: true as expected.

So #214 probably may be closed too.

denispeplin on Oct 12, 2015

This can be VERY confusing. Spent few hours trying to figure out what’s going on on production after removing and re-creating wrongly mapped index. Filters stopped working. Warning will not help when import will start to put tons of logs into console.

denispeplin on Oct 12, 2015

Do we want to silently create an index with the wrong mappings though? Because that’s what we’re doing now if we naively call Article.import with mappings but without an index, which is what a lot of new users will probably do. My main goal here is to prevent introductory users from creating the wrong mappings without knowing it. Perhaps it’s as simple as raising an error if the index doesn’t exist.

RE: IndexManager, I completely agree that having the separate IndexManager is the right approach for your scenario: One index with two types, one for each model. elasticsearch-model doesn’t need to handle that out of the box, the developer should handle that explicitly and not expect import to handle it.

The change I mentioned doesn’t change that. Because invoking SecondModel.import would be harmless since the index would already exist after running FirstModel.import and create_index! force: true never gets invoked. But, now the SecondModel's mappings are busted and we have the same problem all over again…

So, now I am leaning towards raising an error if mappings exist but an index doesn’t. Your call @karmi! Let me know what you think.

dimroc on Jul 18, 2014

I think one of my comments got lost – was travelling the last days.

I’m proposing that Article.import explicitly check if the index exists, and invoke create_index! if it doesn’t.

This cannot work like this, in my mind: consider the case where you have two models, with different mappings, within one index. See https://github.com/elasticsearch/elasticsearch-rails/blob/master/elasticsearch-persistence/examples/music/index_manager.rb#L10-L16 for an example.

UPDATE: And before you mention updating the mapping for the second model, sorry, I simply won’t go there 😃 The linked IndexManager is a much more sane approach, in my opinion.

karmi on Jul 18, 2014

Hi @karmi, I mentioned that in my second comment: It will work if you pass force: true.

My point is that running Article.import without force:true while not having an existing index will erroneously create an index with the default values (type: “string”). create_index! is never actually invoked, the index is created lazily by client.bulk.

You can see the if check right here.

I’m proposing that Article.import explicitly check if the index exists, and invoke create_index! if it doesn’t.

Something like this:

if options.delete(:force) || index_does_not_exist(target_index)
  self.create_index! force: true, index: target_index
end

dimroc on Jul 18, 2014