drill: 1.20.0-SNAPSHOT Fails on mongodb cluster query

Describe the bug Drill is not able to query mongodb cluster, it throws a NPE

2021-10-28 16:21:52,535 [1e8531df-4e21-f062-707b-b9cbb72c619e:foreman] INFO  o.a.d.e.s.mongo.MongoStoragePlugin - Created srv protocol connection to [address=mongoclusterDNSisOK:27017, user=USERalsoOK].
2021-10-28 16:21:52,535 [1e8531df-4e21-f062-707b-b9cbb72c619e:foreman] INFO  o.a.d.e.s.mongo.MongoStoragePlugin - Number of open connections 1.
2021-10-28 16:21:53,165 [1e8531df-4e21-f062-707b-b9cbb72c619e:foreman] WARN  o.a.d.e.s.m.s.MongoSchemaFactory - Failure while getting collection names from 'config'. Command failed with error 13 (Unauthorized): 'not authorized on config to execute command ...
2021-10-28 16:21:53,208 [1e8531df-4e21-f062-707b-b9cbb72c619e:foreman] WARN  o.a.d.e.s.m.s.MongoSchemaFactory - Failure while getting collection names from 'local'. Command failed with error 13 (Unauthorized): 'not authorized on local to execute command ...
2021-10-28 16:21:58,711 [1e8531df-4e21-f062-707b-b9cbb72c619e:frag:1:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 1e8531df-4e21-f062-707b-b9cbb72c619e:1:0: State change requested AWAITING_ALLOCATION --> FINISHED
2021-10-28 16:21:58,711 [1e8531df-4e21-f062-707b-b9cbb72c619e:frag:1:0] INFO  o.a.d.e.w.f.FragmentStatusReporter - 1e8531df-4e21-f062-707b-b9cbb72c619e:1:0: State to report: FINISHED
2021-10-28 16:21:58,711 [drill-executor-6] ERROR o.a.d.exec.server.BootStrapContext - org.apache.drill.exec.work.WorkManager$WorkerBee$1.run() leaked an exception.
java.lang.NullPointerException: null
	at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:347)
	at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)

The DBs unauthorized are cluster-private stuff, these warning lines are also as DEBUG at 1.18.0 logs but after the query is starting on the requested DB without problems.

To Reproduce Steps to reproduce the behavior:

Create a mongodb cluster storage
Try a query
Check the logs

Expected behavior Running the query

Additional context [version] apache/drill:master-openjdk-14 [DIGEST] 55e6551a8523 (2021/10/28 5:23pm) [OS] docker

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (7 by maintainers)

Most upvoted comments

Hi @cgivre fully agree NPE is not a great hint, it took me some time to find a trail

we can note down this as a known issue but for me that’s not an 100% acceptable solution, how high should you go with the slice_target to cover your needs i.e.:1M,10M?

also I’m a bit concerned about possible the side effects as we don’t know the root cause of the issue.

javier-nieto-nexplore on Nov 3, 2021

Hi @luocooong

logs are a bit dry 😦 but I realized that this is related to a config param (planner.slice_target), looks like it has to be greater than collection doc count, not tested in dist mode though

2021-11-03 14:27:40,553 [1e7d63a2-9c2f-c629-d5f1-50c47d2ff3b1:foreman] INFO  o.a.drill.exec.work.foreman.Foreman - Query text for query with id 1e7d63a2-9c2f-c629-d5f1-50c47d2ff3b1 issued by anonymous: select STREET from mongo_local.fake_data_test.frisco
2021-11-03 14:27:41,848 [1e7d63a2-9c2f-c629-d5f1-50c47d2ff3b1:frag:1:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 1e7d63a2-9c2f-c629-d5f1-50c47d2ff3b1:1:0: State change requested AWAITING_ALLOCATION --> FINISHED
2021-11-03 14:27:41,848 [1e7d63a2-9c2f-c629-d5f1-50c47d2ff3b1:frag:1:0] INFO  o.a.d.e.w.f.FragmentStatusReporter - 1e7d63a2-9c2f-c629-d5f1-50c47d2ff3b1:1:0: State to report: FINISHED
2021-11-03 14:27:41,848 [drill-executor-7] ERROR o.a.d.exec.server.BootStrapContext - org.apache.drill.exec.work.WorkManager$WorkerBee$1.run() leaked an exception.
java.lang.NullPointerException: null
	at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:347)
	at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)

javier-nieto-nexplore on Nov 3, 2021

Hi @luocooong

Tested with admin user an same result, I would say it could be something about collection size. I’ve just done these tests:

silly collection with 1 field -> OK
complex collection (flat document) with 10K docs -> OK
complex collection (flat doc) with 160K docs -> NPE

Update: More test local mongodb with agnostic data from here and there and 100K seems to be the magic number, 99.999 documents collection works, when I add a new doc, it fails at the planning and not even querying the DB. Hope it helps.

javier-nieto-nexplore on Nov 2, 2021

@luocooong show databases it’s ok and show tables for the “public” DB is also working OK

At show databases I got the “unathorized” WARN for private DBs but no errors:

 WARN  o.a.d.e.s.m.s.MongoSchemaFactory - Failure while getting collection names from 'config'. Command failed with error 13 (Unauthorized):

javier-nieto-nexplore on Oct 29, 2021