OpenSearch: `_search` calls to OpenSearch with `terminate_after` throw `null_pointer_exception`. Among others, this makes same visualizations that worked on OpenSearch 2.13 to fail on 2.14
Visualization type: Controls
I had this visualization working in opensearch until I updated to 2.14. I tested it back in opensearch 2.13 and in this version it works properly
When I check opensearch logs I see the following error:
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed
at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:757) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:394) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:796) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:542) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:315) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:104) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:75) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:766) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.TransportService$9.handleException(TransportService.java:1729) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:410) [opensearch-security-2.14.0.0.jar:2.14.0.0]
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1515) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1629) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1603) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:81) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:75) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:70) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.action.ActionRunnable.onFailure(ActionRunnable.java:104) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:54) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) [opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.14.0.jar:2.14.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.OpenSearchException$3: Cannot invoke "org.opensearch.core.common.io.stream.NamedWriteable.getWriteableName()" because "namedWriteable" is null
at org.opensearch.OpenSearchException.guessRootCauses(OpenSearchException.java:710) ~[opensearch-core-2.14.0.jar:2.14.0]
at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:392) [opensearch-2.14.0.jar:2.14.0]
... 24 more
Caused by: java.lang.NullPointerException: Cannot invoke "org.opensearch.core.common.io.stream.NamedWriteable.getWriteableName()" because "namedWriteable" is null
at org.opensearch.core.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1134) ~[opensearch-core-2.14.0.jar:2.14.0]
at org.opensearch.core.common.io.stream.StreamOutput.writeNamedWriteableList(StreamOutput.java:1233) ~[opensearch-core-2.14.0.jar:2.14.0]
at org.opensearch.search.aggregations.InternalAggregations.writeTo(InternalAggregations.java:154) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.io.stream.DelayableWriteable$Referencing.writeToBuffer(DelayableWriteable.java:147) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.io.stream.DelayableWriteable$Referencing.writeTo(DelayableWriteable.java:120) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.search.query.QuerySearchResult.writeToNoId(QuerySearchResult.java:445) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.cache.policy.CachedQueryResult.writeToNoId(CachedQueryResult.java:60) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.indices.IndicesService.lambda$loadIntoContext$24(IndicesService.java:1707) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.indices.IndicesService.lambda$cacheShardLevelResult$25(IndicesService.java:1764) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:345) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:328) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.cache.store.OpenSearchOnHeapCache.lambda$computeIfAbsent$0(OpenSearchOnHeapCache.java:98) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.cache.Cache.compute(Cache.java:470) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.cache.Cache.computeIfAbsent(Cache.java:427) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.cache.store.OpenSearchOnHeapCache.computeIfAbsent(OpenSearchOnHeapCache.java:98) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:287) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1770) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.indices.IndicesService.loadIntoContext(IndicesService.java:1701) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:557) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:623) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:592) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-2.14.0.jar:2.14.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.14.0.jar:2.14.0]
... 8 more
OpenSearch Version 2.14
Dashboards Version 2.14
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (6 by maintainers)
Hum, after investigating a little more, maybee I found somethink.
If I Take the query send to Opensearch by dashboard on devtools, I get the same error
I get errors
Now, If I removed the `“terminate_after”: 1000000``from the query, no more error (Or If I put a very big value on it like “terminate_after”: 1000000000)
I get the expected result
Looked into this with @jed326 and it is related to https://github.com/opensearch-project/OpenSearch/pull/12697/ . He is creating a PR to fix it. The issue happens with combination of
terminate_afterand aggregation. So a workaround is to not haveterminate_afterin your request or have a higher value such that query is not actually force terminated early.