druid: AvaticaClientRuntimeException: Remote driver error: IllegalStateException: DefaultQueryMetrics must not be modified from multiple threads.

Hi, I am getting the error below. The same query runs without any problems when just a few records are returned by the where clause. I am currently running on a single node “quickstart” configuration.

Thank You

`2017-09-19 11:03:54 DEBUG DruidDao:414 - SQL: select A, B, C, D, E, G, H, I, L from dataSource where A in (‘aValue’ ) AND B in (‘SomeVALUE’ ) [WARNING] /dl AvaticaClientRuntimeException: Remote driver error: IllegalStateException: DefaultQueryMetrics must not be modified from multiple threads. If it is needed to gather dimension or metric information from multiple threads or from an async thread, this information should explicitly be passed between threads (e. g. using Futures), or this DefaultQueryMetrics’s ownerThread should be reassigned explicitly. Error -1 (00000) UNKNOWN

java.lang.IllegalStateException: DefaultQueryMetrics must not be modified from multiple threads. If it is needed to gather dimension or metric information from multiple threads or from an async thread, this information should explicitly be passed between threads (e. g. using Futures), or this DefaultQueryMetrics’s ownerThread should be reassigned explicitly at io.druid.query.DefaultQueryMetrics.checkModifiedFromOwnerThread(DefaultQueryMetrics.java:59) at io.druid.query.DefaultQueryMetrics.reportMetric(DefaultQueryMetrics.java:253) at io.druid.query.DefaultQueryMetrics.reportCpuTime(DefaultQueryMetrics.java:231) at io.druid.query.CPUTimeMetricQueryRunner$1.after(CPUTimeMetricQueryRunner.java:86) at io.druid.java.util.common.guava.WrappingYielder.close(WrappingYielder.java:96) at io.druid.java.util.common.guava.Yielders$2.close(Yielders.java:73) at io.druid.java.util.common.guava.ConcatSequence.wrapYielder(ConcatSequence.java:123) at io.druid.java.util.common.guava.ConcatSequence.access$000(ConcatSequence.java:29) at io.druid.java.util.common.guava.ConcatSequence$3.next(ConcatSequence.java:143) at io.druid.sql.avatica.DruidStatement.nextFrame(DruidStatement.java:277) at io.druid.sql.avatica.DruidMeta.fetch(DruidMeta.java:209) at org.apache.calcite.avatica.remote.LocalService.apply(LocalService.java:252) at org.apache.calcite.avatica.remote.Service$FetchRequest.accept(Service.java:1350) at org.apache.calcite.avatica.remote.Service$FetchRequest.accept(Service.java:1317) at org.apache.calcite.avatica.remote.AbstractHandler.apply(AbstractHandler.java:95) at org.apache.calcite.avatica.remote.JsonHandler.apply(JsonHandler.java:52) at org.apache.calcite.avatica.server.AvaticaJsonHandler.handle(AvaticaJsonHandler.java:129) at io.druid.sql.avatica.DruidAvaticaHandler.handle(DruidAvaticaHandler.java:63) at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Suppressed: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: DefaultQueryMetrics must not be modified from multiple threads. If it is needed to gather dimension or metric information from multiple threads or from an async thread, this information should explicitly be passed between threads (e. g. using Futures), or this DefaultQueryMetrics’s ownerThread should be reassigned explicitly at com.google.common.base.Throwables.propagate(Throwables.java:160) at io.druid.sql.avatica.DruidStatement.close(DruidStatement.java:335) at io.druid.sql.avatica.DruidStatement.nextFrame(DruidStatement.java:290) … 22 more Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: DefaultQueryMetrics must not be modified from multiple threads. If it is needed to gather dimension or metric information from multiple threads or from an async thread, this information should explicitly be passed between threads (e. g. using Futures), or this DefaultQueryMetrics’s ownerThread should be reassigned explicitly at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at io.druid.sql.avatica.DruidStatement.close(DruidStatement.java:319) … 23 more Caused by: java.lang.IllegalStateException: DefaultQueryMetrics must not be modified from multiple threads. If it is needed to gather dimension or metric information from multiple threads or from an async thread, this information should explicitly be passed between threads (e. g. using Futures), or this DefaultQueryMetrics’s ownerThread should be reassigned explicitly at io.druid.query.DefaultQueryMetrics.checkModifiedFromOwnerThread(DefaultQueryMetrics.java:59) at io.druid.query.DefaultQueryMetrics.reportMetric(DefaultQueryMetrics.java:253) at io.druid.query.DefaultQueryMetrics.reportCpuTime(DefaultQueryMetrics.java:231) at io.druid.query.CPUTimeMetricQueryRunner$1.after(CPUTimeMetricQueryRunner.java:86) at io.druid.java.util.common.guava.WrappingYielder.close(WrappingYielder.java:96) at io.druid.java.util.common.guava.ConcatSequence$3.close(ConcatSequence.java:157) at io.druid.java.util.common.guava.ConcatSequence$3.close(ConcatSequence.java:156) at io.druid.sql.avatica.DruidStatement.lambda$close$1(DruidStatement.java:315) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) … 1 more

at org.apache.calcite.avatica.remote.Service$ErrorResponse.toException(Service.java:2476) at org.apache.calcite.avatica.remote.JsonService.decode(JsonService.java:55) at org.apache.calcite.avatica.remote.JsonService.apply(JsonService.java:140) at org.apache.calcite.avatica.remote.RemoteMeta$14.call(RemoteMeta.java:309) at org.apache.calcite.avatica.remote.RemoteMeta$14.call(RemoteMeta.java:306) at org.apache.calcite.avatica.AvaticaConnection.invokeWithRetries(AvaticaConnection.java:756) at org.apache.calcite.avatica.remote.RemoteMeta.fetch(RemoteMeta.java:305) at org.apache.calcite.avatica.MetaImpl$FetchIterator.moveNext(MetaImpl.java:1606) at org.apache.calcite.avatica.MetaImpl$FetchIterator.next(MetaImpl.java:1591) at org.apache.calcite.avatica.util.IteratorCursor.next(IteratorCursor.java:46) at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:239) at … (DruidDao.java:1181) at … (Download.java:75) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:833) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:206) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:564) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128) at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591) at java.lang.Thread.run(Thread.java:748)`

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 18 (7 by maintainers)

Most upvoted comments

Hello there, we also run into this issue, any update for the fix? Thanks in advance!

I wrote a Presto connector for druid, and it is strait-forward to use JDBC, otherwise it would take a lot more effort with json-over-http.

I also raised an issue, but feel free to duplicate it - https://github.com/druid-io/druid/issues/5110

guangyy on Nov 21, 2017

@fmarelli apologies for the inconvenience, we’ll get it fixed as soon as we can.

Some notes on that: I think the same-thread check in QueryMetrics is counter-productive for the JDBC API. DruidStatement already synchronizes calls to the yielder, and I think that’s the only thing that might call QueryMetrics, so it should be safe as-is if the check were ignored. The issue is just that JDBC delivers query results not all in one http request, but spread out over multiple http requests, so even though the calls are synchronized they might not happen in the same thread.

Would it make sense to disable the check for JDBC? Or at least have some way to adjust the ownerThread? The QueryMetrics docs say “this DefaultQueryMetrics’s ownerThread should be reassigned explicitly” but it’s not clear how best to do that when it’s buried inside a Sequence.

@leventov or @jon-wei do you have any comment on this?

gianm on Sep 19, 2017

@Saurabh111191 the issue you reported is different from one from the one reported in this issue. You aren’t using Druid SQL (it’s the normal QueryResource). I think the issue you’re running into is with the chunkPeriod setting. For reasons mentioned in https://github.com/druid-io/druid/pull/4004#issuecomment-284171911, chunkPeriod is not typically useful anyway and I’ve been considering pushing for its removal. In light of what it actually does, do you think you really need it? If not then I’d suggest turning it off.

I raised https://github.com/druid-io/druid/issues/5128 for the bug you ran into.

gianm on Nov 30, 2017

Hello ,

Recently we have upgraded druid to latest version form 0.9.1.1 to druid-0.10.1. And we too are facing the same issue. Our time series query and topN query used to wok previously but after upgrading , same issue is coming on broker side. For workaround , we haven’t upgraded broker and everything is working as normal. But can you please tell us what the issue might be and how can we fix it. Exception from broker is { “error”: “Unknown exception”, “errorMessage”: “DefaultQueryMetrics must not be modified from multiple threads. If it is needed to gather dimension or metric information from multiple threads or from an async thread, this information should explicitly be passed between threads (e. g. using Futures), or this DefaultQueryMetrics’s ownerThread should be reassigned explicitly”, “errorClass”: “java.lang.IllegalStateException”, “host”: null } Whole exception on the broker can be found at : https://jpst.it/180vD

Saurabh111191 on Nov 27, 2017