jaeger: query error after run es_indices_clean.sh

jaeger version: cba413ede300564e6f92a16f812604e4a829d0c7 storage: elasticsearch 6.1.1, Build: bd92e7f/2017-12-17T20:23:25.338Z, JVM: 9.0.1

It’s work before clean.

My operation process:

⋊> ~/opt ./es_indices_clean.sh 0 localhost:9200                                                                                                                                                             
Installing python dependencies required for curator...
Requirement already satisfied: elasticsearch in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied: elasticsearch-curator in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from elasticsearch)
Requirement already satisfied: click>=6.7 in /usr/local/lib/python2.7/dist-packages (from elasticsearch-curator)
Requirement already satisfied: pyyaml>=3.10 in /usr/local/lib/python2.7/dist-packages (from elasticsearch-curator)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from elasticsearch-curator)
Requirement already satisfied: voluptuous>=0.9.3 in /usr/local/lib/python2.7/dist-packages (from elasticsearch-curator)

Removing jaeger-service-2017-12-26
Removing jaeger-span-2017-12-26
⋊> ~/opt curl -XGET 'http://localhost:9200/_cat/indices'          # generate new data                                                                                                                                          
yellow open jaeger-service-2017-12-26 sMNJrlcYQGqMDp4AUrLlWA 5 1  2 0   9.9kb   9.9kb
yellow open jaeger-span-2017-12-26    wBc1qghHQ4uo0XobS7tKgA 5 1 27 0 263.7kb 263.7kb

Then visit http://local:16686/search, get error:

There was an error querying for traces:
HTTP Error: Search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]

Status | 500
Status text | Internal Server Error
URL | /api/services
Response body | {   "data": null,   "total": 0,   "limit": 0,   "offset": 0,   "errors": [     {       "code": 500,       "msg": "Search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]"     }   ] }

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 7
Comments: 16 (7 by maintainers)

Most upvoted comments

I have seen the same problem with bad mappings. You will need to either:

A. Shutdown all jaeger collectors, delete the bad indexes, then restart the collectors and lose the data or B. Create new ElasticSearch index templates and trigger a reindex to a new index and delete the bad ones after it completes. The index mappings for the template can be extracted from a working index.

mikelduke on Jan 9, 2019

I have fixed this by creating an index template

curl -ivX PUT -H "Content-Type: application/json" localhost:9200/_template/span  -d @./plugin/storage/es/mappings/jaeger-span.json
curl -ivX PUT -H "Content-Type: application/json" localhost:9200/_template/service  -d @./plugin/storage/es/mappings/jaeger-service.json

I think jaeger-collector should create the template on startup. The create index would just create an index and mapping would be derived from the template stored in ES. Maybe we could even omit creating the index and it would be created automatically once data is inserted.

pavolloffay on Jun 21, 2019

This issue is related to #374.

Steps to reproduce:

Start jaeger

docker run -it --rm -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=false" --name=elasticsearch  docker.elastic.co/elasticsearch/elasticsearch:5.6.10
SPAN_STORAGE_TYPE=elasticsearch go run -tags ui ./cmd/all-in-one/main.go

Generate spans via Jaeger UI or hotrod
Remove all indices python plugin/storage/es/esCleaner.py 0 localhost:9200
refresh Jaeger UI - it should show HTTP Error: Search service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception].

pavolloffay on Jun 21, 2019

thx @mikelduke ! , your post helped me solve this weird error that was, at least for me, likely caused by the ES being recreated but collectors werent reset/restarted. and me then spending few hours to debug until I stumbled over your precious hint ❤️ and noticed the index templates were missing !! I restored them based on https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/es/mappings

philicious on Nov 18, 2020

I have the same issue:

/api/services
 service failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]"

I just updated es from 6.2 to 6.3, and cleaned the old data.

oiooj on Jul 9, 2018