sentinl: ES 6.x: Rejecting mapping update to [watcher] as the final mapping would have more than 1 type

  1. Issue Description

Receiving from Kibana

elks_kibana_1 | {“type”:“log”,“@timestamp”:“2017-11-24T11:01:30Z”,“tags”:[“status”,“error”,“Sentinl”],“pid”:1,“message”:“[illegal_argument_exception] Rejecting mapping update to [watcher] as the final mapping would have more than 1 type: [sentinl-script, sentinl-watcher]”}

  1. Reproducing the Issue step-by-step

Apologies, extremely new to ELK Stack, Sentinl, Docker, Docker-Compose, etc. Been asked to try and prototype a monitoring system and still in the basic configuration stage.

Running Ubuntu 16.04 ELK Stack 6.0.0 Sentinl 6.0.0 Latest versions of Docker and Docker-Compose

Not sure if this is the right approach.

Creating a separate container for each of Elasticsearch, Logstash and Kibana using the main ELK definitions.

As we don’t want the ELK X-Pack I uninstall it from the images and replace the base .yml files removing any reference to X-Pack.

(I understand that in theory I could just build them without X-Pack and will probably do so at sometime, but just adds to an already large learning curve! 😃 )

Using a very basic configuration in a docker-compose.yml file the three containers come up and are happily talking to each other and I can access a pretty much empty Kibana screen.

I add Sentinl into the Dockerfile for creating the Kibana container and it builds fine.

Now when I start with Docker-compose I get the error at the top.

It looks, perhaps, as if the default Sentinl configuration is causing this, but not sure where to look or what to try.

  1. Expected Results vs. Actual Results

No error vs Error!

  1. Additional Details
OK, here goes

Dockerfile for Elasticsearch
-------------------------------------------
FROM docker.elastic.co/elasticsearch/elasticsearch:6.0.0
# This is the Dockerfile for creating the Elasticsearch Container for the ELKS Monitor

# Copy the .yml file with X-Pack removed
ADD elasticsearch.yml ./config/elasticsearch.yml

# Uninstall X-Pack
RUN ./bin/elasticsearch-plugin remove x-pack --purge
-------------------------------------------
elasticsearch.yml file
-------------------------------------------
cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
# Details: https://github.com/elastic/elasticsearch/pull/17288
discovery.zen.minimum_master_nodes: 1
-------------------------------------------
-------------------------------------------
Dockerfile for Logstash
-------------------------------------------
FROM docker.elastic.co/logstash/logstash:6.0.0
# This is the Dockerfile for creating the Logstash Container for the ELKS Monitor

# Copy the .yml file with X-Pack removed
ADD logstash.yml ./config/logstash.yml

# Uninstall X-Pack
RUN ./bin/logstash-plugin remove x-pack
-------------------------------------------
logstash.yml
-------------------------------------------
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
-------------------------------------------
-------------------------------------------
Dockerfile for Kibana
-------------------------------------------
FROM docker.elastic.co/kibana/kibana:6.0.0
# This is the Dockerfile for creating the Kibana Container for the Desk ELKS Monitor
#
# For this to work it requires the Sentinl Plug-in to be installed
# See https://github.com/sirensolutions/sentinl or https://github.com/sirensolutions/sentinl/wiki/SENTINL-Installation
#

ADD kibana.yml ./config/kibana.yml

# Uninstall X-Pack
RUN ./bin/kibana-plugin remove x-pack

# Install potentially missing libraries that Sentinl requires
USER root
RUN yum install -y fontconfig freetype

# Install Filebeats so the example Kibana Dashboards can be deployed
RUN curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm
RUN rpm -vi filebeat-6.0.0-x86_64.rpm

# Install Sentinl
USER kibana
RUN ./bin/kibana-plugin install https://github.com/sirensolutions/sentinl/releases/download/tag-6.0.0/sentinl-v6.0.0.zip
-------------------------------------------
kibana.yml
-------------------------------------------
# Default Kibana configuration from kibana-docker.

server.name: kibana
server.host: "0.0.0.0"
elasticsearch.url: http://elasticsearch:9200
elasticsearch.username: elastic
elasticsearch.password: changeme
-------------------------------------------
-------------------------------------------
docker-compose.yml file
-------------------------------------------
# docker-compose.yml file for starting the ELKS Monitoring System

version: '3.4'
services:

# Elasticsearch configuration
  elks_elasticsearch:
    build:
      context: ./elks_elasticsearch
    image: elks_elasticsearch:6.0.0
    ports: 
      - 9200:9200
      - 9300:9300
    networks: 
      - elks_nw
    volumes:
      - /data/elks/elks_data/elks_e_data:/usr/share/elasticsearch/data

# Logstash configuration
  elks_logstash:
    build:
      context: ./elks_logstash
    image: elks_logstash:6.0.0
#   There appears to be a bug in environment so writing them this way
    environment:
      - ELASTICSEARCH_URL=http://elks_elasticsearch:9200
    ports: 
      - 5000:5000
      - 5001:5001
    networks: 
      - elks_nw
    volumes:
      - /data/elks/elks_data/elks_l_data:/usr/share/logstash/data
    depends_on:
      - elks_elasticsearch

# Kibana configuration
  elks_kibana:
    build:
      context: ./elks_kibana
    image: elks_kibana:6.0.0
#   There appears to be a bug in environment so writing them this way
    environment:
      - ELASTICSEARCH_URL=http://elks_elasticsearch:9200
    ports: 
      - 5601:5601
    networks: 
      - elks_nw
    volumes:
      - /data/elks/elks_data/elks_k_data:/usr/share/kibana/data
    depends_on:
      - elks_elasticsearch

networks: 
  elks_nw:
    driver: bridge  
-------------------------------------------
Hopefully as you can see a very basic setup. Hope it is nice and simple.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 21

Most upvoted comments

@EricFlis this won’t work. Seems that the problem is not with the index you define for input but with the watcher index. At least, that what the error says:

Rejecting mapping update to [watcher] as the final mapping would have more than 1 type: [sentinl-script, sentinl-watcher]

What I understood of that error message is that watcher index can’t have both: sentinl-script and sentinl-watcher types.

Edit note: I’m trying with ELK 6.0.0 and alpha sentinl 6.x release from here: https://github.com/sirensolutions/sentinl/releases/download/tag-6.0.0/sentinl-v6.0.0.zip and I can afirm it’s not working at all.

We found a solution which will work until the Sentinl code catches up with Elasticsearch changes. Unfortunately it requires a minor manual modification to the input code when you go to set a watcher. It involves removing the line which defines the index, Sentinl still seems to know where to pull the data from without this reference. Code sample below:

{
  "search": {
    "request": {
      "index": [],
      "body": {
        "version": true,
        "size": 500,
        "sort": [
          {
            "postdate": {
              "order": "desc",
              "unmapped_type": "boolean"
            }
          }
        ],
        "_source": {
          "excludes": []
        },
        "aggs": {
          "2": {
            "date_histogram": {
              "field": "postdate",
              "interval": "30s",
              "time_zone": "Australia/Sydney",
              "min_doc_count": 1
            }
          }
        },
        "stored_fields": [
          "*"
        ],
        "script_fields": {},
        "docvalue_fields": [
          "postdate"
        ],
        "query": {
          "bool": {
            "must": [
              {
                "query_string": {
                  "query": "\"australia\"",
                  "analyze_wildcard": true,
                  "default_field": "*"
                }
              },
              {
                "match_phrase": {
                  "entity": {
                    "query": "Cyber Security"
                  }
                }
              },
              {
                "range": {
                  "postdate": {
                    "gte": 1513053875621,
                    "lte": 1513054775621,
                    "format": "epoch_millis"
                  }
                }
              },
              {
                "range": {
                  "postdate": {
                    "from": "now-1h"
                  }
                }
              }
            ],
            "filter": [],
            "should": [],
            "must_not": []
          }
        },
        "highlight": {
          "pre_tags": [
            "@kibana-highlighted-field@"
          ],
          "post_tags": [
            "@/kibana-highlighted-field@"
          ],
          "fields": {
            "*": {}
          },
          "fragment_size": 2147483647
        }
      }
    }
  }
}