generator-jhipster: ElasticSearch Exception on Newly Generated Application

Overview of the issue

In a newly generated application, Elastic Search is causing an “Internal server error” by throwing an exception. Once an entity instance has been updated, it can be searched. On all other (not yet updated) entities of the same type, ElasticSearch doesn’t find anything. On other entity types, where no entity yet has been updated, the Internal Server Error/ElasticSearch Exception still occurs.

Stack Traces and Messages

Application Message when no entity has been updated:

2023-11-13T21:32:20.676+01:00 DEBUG 262279 --- [ XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource : Enter: searchAS() with argument[s] = [Kutch, Page request [number: 0, size 20, sort: id: ASC]] 2023-11-13T21:32:20.676+01:00 DEBUG 262279 --- [ XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource : REST request to search for a page of AS for query Kutch 2023-11-13T21:32:20.677+01:00 DEBUG 262279 --- [ XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl : Enter: search() with argument[s] = [Kutch, Page request [number: 0, size 20, sort: id: ASC]] 2023-11-13T21:32:20.677+01:00 DEBUG 262279 --- [ XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl : Request to search for a page of AS for query Kutch 2023-11-13T21:32:20.830+01:00 ERROR 262279 --- [ XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl : Exception in search() with cause = 'co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed' and exception = '[es/search] failed: [search_phase_execution_exception] all shards failed'

org.springframework.data.elasticsearch.UncategorizedElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed at org.springframework.data.elasticsearch.client.elc.ElasticsearchExceptionTranslator.translateExceptionIfPossible(ElasticsearchExceptionTranslator.java:102) at org.springframework.data.elasticsearch.client.elc.ElasticsearchExceptionTranslator.translateException(ElasticsearchExceptionTranslator.java:63) at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.execute(ElasticsearchTemplate.java:625) at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.doSearch(ElasticsearchTemplate.java:341) at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.search(ElasticsearchTemplate.java:334) at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.search(AbstractElasticsearchTemplate.java:492) at com.mycompany.myapp.repository.search.ASearchRepositoryInternalImpl.search(ASearchRepository.java:53) at com.mycompany.myapp.repository.search.ASearchRepositoryInternalImpl.search(ASearchRepository.java:48) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:237) at com.mycompany.myapp.repository.search.$Proxy212.search(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:288) … … Caused by: co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:334) at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:154) at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1882) at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.lambda$doSearch$14(ElasticsearchTemplate.java:341) at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.execute(ElasticsearchTemplate.java:623) … 246 common frames omitted

2023-11-13T21:32:20.879+01:00 WARN 262279 — [ XNIO-1 task-7] .m.m.a.ExceptionHandlerExceptionResolver : Resolved [org.springframework.data.elasticsearch.UncategorizedElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed]

Messages upon successful search on an previously updated entity:

2023-11-13T21:34:07.349+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : Enter: searchAS() with argument[s] = [Kutch, Page request [number: 0, size 20, sort: id: ASC]] 
2023-11-13T21:34:07.350+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : REST request to search for a page of AS for query Kutch 
2023-11-13T21:34:07.350+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Enter: search() with argument[s] = [Kutch, Page request [number: 0, size 20, sort: id: ASC]] 
2023-11-13T21:34:07.350+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Request to search for a page of AS for query Kutch 
2023-11-13T21:34:07.632+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Exit: search() with result = Page 1 of 1 containing com.mycompany.myapp.service.dto.ADTO instances 
2023-11-13T21:34:07.633+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : Exit: searchAS() with result = <200 OK OK,[ADTO{id=1, firstName='Kenya', lastName='Kutch'}],[X-Total-Count:"1", Link:"<http://localhost:8080/api/as/_search?query=Kutch&sort=id%2Casc&page=0&size=20>; rel="last",<http://localhost:8080/api/as/_search?query=Kutch&sort=id%2Casc&page=0&size=20>; rel="first""]>

Messages upon failed search on a not-yet updated entity:

2023-11-13T21:35:13.353+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : Enter: searchAS() with argument[s] = [Reta, Page request [number: 0, size 20, sort: id: ASC]] 
2023-11-13T21:35:13.353+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : REST request to search for a page of AS for query Reta 
2023-11-13T21:35:13.353+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Enter: search() with argument[s] = [Reta, Page request [number: 0, size 20, sort: id: ASC]] 
2023-11-13T21:35:13.353+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Request to search for a page of AS for query Reta 
2023-11-13T21:35:13.364+01:00 DEBUG 262279 --- [  XNIO-1 task-7] c.m.myapp.service.impl.AServiceImpl      : Exit: search() with result = Page 1 of 0 containing UNKNOWN instances 
2023-11-13T21:35:13.365+01:00 DEBUG 262279 --- [  XNIO-1 task-7] com.mycompany.myapp.web.rest.AResource   : Exit: searchAS() with result = <200 OK OK,[],[X-Total-Count:"0", Link:"<http://localhost:8080/api/as/_search?query=Reta&sort=id%2Casc&page=-1&size=20>; rel="last",<http://localhost:8080/api/as/_search?query=Reta&sort=id%2Casc&page=0&size=20>; rel="first""]>

Motivation for or Use Case

A newly generated application should work out of the box. Apparently ElasticSearch has to build datastructures that do not exist on a new application. Also, initial indexing does not happen on fake data. All of this creates much irritation because you are not told, what the problem is.

Reproduce the error

A sample JDL is given below. JDL is in the otherwise empty application folder.

jhipster jdl application.jdl --force
in a second terminal window: docker compose -f src/main/docker/elasticsearch.yml up
./mvnw clean && ./mvnw
browse to http://localhost:8080
sign in
navigate to Entities/A
put the last name of the first instance (in my case “Kutch”) into the search box and search: first sequence of messages appears
clear the search, select the edit button on that entity. Click save.
Again put the last name into the search box: search is successfull with the second sequence of messages
put the last name of the second instance into the search box and search: not successfull, third sequence of messages
navigate to Entities/B: put a search term. Again Internal Server Error happens, repeat 8. und 9.

Related issues

https://github.com/jhipster/generator-jhipster/issues/22603

Suggest a Fix

The ElasticSearch exception should not just lead to “Internal server error”, but be more specific.
When the app is initializing the database (mvn clean), it should also initialize ElasticSearch (important during development cycles).

JHipster Version(s)

JHipster configuration

.yo-rc.json file

{
  "generator-jhipster": {
    "applicationIndex": 0,
    "applicationType": "monolith",
    "authenticationType": "jwt",
    "baseName": "espaginationconflict",
    "buildTool": "maven",
    "cacheProvider": "ehcache",
    "clientFramework": "angular",
    "creationTimestamp": 1699883023373,
    "databaseType": "sql",
    "devDatabaseType": "h2Disk",
    "devServerPort": 4200,
    "enableHibernateCache": true,
    "enableSwaggerCodegen": true,
    "enableTranslation": true,
    "entities": [
      "A",
      "B",
      "C"
    ],
    "jhipsterVersion": "8.0.0",
    "languages": [
      "en",
      "de",
      "fr"
    ],
    "lastLiquibaseTimestamp": 1699883203000,
    "messageBroker": false,
    "microfrontends": [],
    "nativeLanguage": "en",
    "packageFolder": "com/mycompany/myapp",
    "packageName": "com.mycompany.myapp",
    "prodDatabaseType": "mariadb",
    "reactive": false,
    "searchEngine": "elasticsearch",
    "serviceDiscoveryType": false,
    "testFrameworks": [
      "gatling",
      "cucumber"
    ],
    "websocket": "spring-websocket",
    "withAdminUi": true
  }
}

JDL definitions

    application {
  config {
    jhipsterVersion "8.0.0"
    creationTimestamp 1699883023373
    baseName espaginationconflict
    packageName com.mycompany.myapp
    applicationType monolith
    buildTool maven
//    jhiPrefix jhi
//  entitySuffix false  // do not enable, will cause complation errors
//    dtoSuffix DTO
    authenticationType jwt
    withAdminUi true
    enableSwaggerCodegen true
    databaseType sql
    devDatabaseType h2Disk
    prodDatabaseType mariadb
    searchEngine elasticsearch
    websocket spring-websocket
    cacheProvider ehcache
    enableHibernateCache true
    clientFramework angular
    enableTranslation true
    nativeLanguage en
    languages [en, de, fr]
    testFrameworks [gatling, cucumber]
    messageBroker false
    serviceDiscoveryType false
    microfrontends []
    reactive false
  }
    entities *
}
entity A {
    firstName String
    lastName String
}
entity B {
    shortName String
    longName String
}
entity C {
    shortIdent String
    longIdent String
}
service  all with serviceImpl
dto      all with mapstruct
filter   all
paginate all with pagination
search   all with elasticsearch

Entity configuration(s) `entityName.json` files generated in the `.jhipster` directory

Browsers and Operating System

Ubuntu 22.04 Firefox: 119.0.1

Environment and Tools

openjdk version “17.0.8.1” 2023-08-24 OpenJDK Runtime Environment (build 17.0.8.1+1-Ubuntu-0ubuntu122.04) OpenJDK 64-Bit Server VM (build 17.0.8.1+1-Ubuntu-0ubuntu122.04, mixed mode, sharing) git version 2.34.1 node: v20.9.0 npm: 10.1.0 Docker version 24.0.7, build afdd53b

Checking this box is mandatory (this is just to show you read everything)

About this issue

Original URL
State: open
Created 8 months ago
Comments: 16 (3 by maintainers)

Most upvoted comments

@user-0209 I found this sample with elasticsearch on version 8. https://github.com/jhipster/jhipster-sample-app-elasticsearch

Maybe it is of any to you.

Thank you for responding. In fact, I find it amazing, how fast I’m getting responses in this Jhipster context. However, although the project has been upgraded to 8.0.0, it exhibits the same weired behaviour as my newly generated application. In fact, it is even more strange: The BankAccount and Label entities can be searched without throwing exception, but matching entities would not be found. The Operation entity is throwing an exception. In all cases, after just updating an entity (click edit, then save), that specific entity instance can be found, all others of the same type still not. I guess, there has been a difference, how these three entities have been generated. Altogether, this makes it very difficult to get into a strictly repeatable development/testing flow. That’s why I’m so insisting on that. I hope, I can achieve something with the blueprint link, which appears to open another box… Anyways, I’ll try to solve this and report back.

user-0209 on Nov 16, 2023

I think I know what happened here, give me some time I’ll get a reason and a way to solve it.

Nenzyz on Dec 11, 2023

@Nenzyz Thank you for your input! Unfortunately, when I simply change the version in the docker container from 8.7.1 to 7.17.4 – it does not solve the problem. Have you done more than this? Can you maybe share your relevant config snippets with us?

The only change from the freshly created project was changes in src/main/docker/elasticsearch.yml version, and for sure restart of the container. It would be nice to know if the error you are getting the same as reporter has or something else?

What doesn’t work still is reindexing of existing entities, but this is unrelated to this issue.

(I won’t post in the old issue, since this one is referenced in the old)

Nenzyz on Dec 11, 2023

The issue seems to be with new elasticsearch. As a workaround I’ve used elasticsearch version from Jhipster 7 and it works very good. Version 7.17.4 –> OK, version 8.7.1 -> NOT OK. I’m pretty sure protocol changed on ES side, since Jhipster 7 with ES v8.7.1 also gets an error.

Nenzyz on Dec 10, 2023

@aturyng: The problem is not ES as such. Once the index for the entity is in place, things work (for the entity instances that have been updated). The problem is with the preloaded fake-data. Not sure, if I’m old-fashioned, but I want defined, repeatable test data in the development cycle. Earlier (I think version 6), I had one of those re-indexing extensions in place and it worked. However, and here is the point: since few versions so many things have changed that all these blueprints don’t work and apparently, nobody is taking the effort, to update them (guess why). So I embarked on porting one of the reindexing modules to a proper blueprint, but I have to say, it is very time-consuming due to a complete lack of documentation of those jHipster base classes, but not only. The blueprints chapter can only serve as a teaser, re-eingineering the base classes is necessary. In summary, I’d highly appreciate JSDoc for the baseclasses and in the blueprints documentation experts should say, when (scenario) you want which base class. Amongst the core-team maintained official blueprints there should be an (almost) empty blueprint that is up to speed with the respective JHipster version.

Having said this, I’m still much in favour of getting the application generated, because otherwise you’d end in another tunnel. Thanks to the core team. Let me know, if I’m missing some documentation.

user-0209 on Nov 23, 2023

Our elasticsearch tests needs improvements to avoid regressions like this one.

mshima on Nov 16, 2023

@user-0209 I found this sample with elasticsearch on version 8. https://github.com/jhipster/jhipster-sample-app-elasticsearch

Maybe it is of any to you.

aturyng on Nov 15, 2023

@user-0209 I believe you can use https://github.com/jhipster/generator-jhipster-entity-audit as an example. Updated documentation is available here, any feedback is welcome

DanielFran on Nov 15, 2023

I don’t have a solution at this moment. I’m trying to make https://github.com/Ebsan/generator-jhipster-es-entity-reindexer work. I’m pretty sure its just about rebuilding the index.

user-0209 on Nov 13, 2023