trino: Query failure when IN compares indeterminate objects: VerifyException: result is null

CREATE TABLE memory.default.orders AS TABLE tpch.tiny.orders;

trino> SELECT * FROM memory.default.orders WHERE ARRAY[NULL,orderkey] IN ( ARRAY[0,0],ARRAY[0,1],ARRAY[0,2],ARRAY[0,3],ARRAY[0,4],ARRAY[0,5],ARRAY[0,6],ARRAY[0,7] );

Query 20220225_102124_00012_w9kv5, FAILED, 1 node
http://localhost:8080/ui/query.html?20220225_102124_00012_w9kv5
Splits: 24 total, 10 done (41.67%)
CPU Time: 0.0s total,  250K rows/s,   31MB/s, 55% active
Per Node: 0.1 parallelism, 28.2K rows/s,  3.5MB/s
Parallelism: 0.1
Peak Memory: 0B
0.27 [7.5K rows, 953KB] [28.2K rows/s, 3.5MB/s]

Query 20220225_102124_00012_w9kv5 failed: result is null
com.google.common.base.VerifyException: result is null
	at com.google.common.base.Verify.verifyNotNull(Verify.java:503)
	at io.trino.util.FastutilSetHelper$ObjectStrategy.equals(FastutilSetHelper.java:235)
	at it.unimi.dsi.fastutil.objects.ObjectOpenCustomHashSet.contains(ObjectOpenCustomHashSet.java:442)
	at io.trino.util.FastutilSetHelper.in(FastutilSetHelper.java:93)
	at io.trino.$gen.PageFilter_20220225_102124_25.filter(Unknown Source)
	at io.trino.$gen.PageFilter_20220225_102124_25.filter(Unknown Source)
	at io.trino.operator.project.DictionaryAwarePageFilter.filter(DictionaryAwarePageFilter.java:83)
	at io.trino.operator.project.PageProcessor.createWorkProcessor(PageProcessor.java:121)
	at io.trino.operator.ScanFilterAndProjectOperator$SplitToPages.lambda$processPageSource$1(ScanFilterAndProjectOperator.java:295)
	at io.trino.operator.WorkProcessorUtils.lambda$flatMap$5(WorkProcessorUtils.java:264)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:338)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils.lambda$flatten$7(WorkProcessorUtils.java:296)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:338)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:240)
	at io.trino.operator.WorkProcessorUtils.lambda$processStateMonitor$3(WorkProcessorUtils.java:219)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:240)
	at io.trino.operator.WorkProcessorUtils.lambda$finishWhen$4(WorkProcessorUtils.java:234)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorSourceOperatorAdapter.getOutput(WorkProcessorSourceOperatorAdapter.java:150)
	at io.trino.operator.Driver.processInternal(Driver.java:388)
	at io.trino.operator.Driver.lambda$processFor$9(Driver.java:292)
	at io.trino.operator.Driver.tryWithLock(Driver.java:693)
	at io.trino.operator.Driver.processFor(Driver.java:285)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1092)
	at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:488)
	at io.trino.$gen.Trino_dev____20220225_101503_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

trino> SELECT DISTINCT ARRAY[NULL,orderkey] IN ( ARRAY[0,0],ARRAY[0,1],ARRAY[0,2],ARRAY[0,3],ARRAY[0,4],ARRAY[0,5],ARRAY[0,6],ARRAY[0,7] ) FROM memory.default.orders ;

Query 20220225_102405_00019_w9kv5, FAILED, 1 node
http://localhost:8080/ui/query.html?20220225_102405_00019_w9kv5
Splits: 40 total, 0 done (0.00%)
CPU Time: 0.0s total, 3.75M rows/s, 32.2MB/s, 40% active
Per Node: 0.0 parallelism, 56.4K rows/s,  496KB/s
Parallelism: 0.0
Peak Memory: 125KB
0.13 [7.5K rows, 66KB] [56.4K rows/s, 496KB/s]

Query 20220225_102405_00019_w9kv5 failed: result is null
com.google.common.base.VerifyException: result is null
	at com.google.common.base.Verify.verifyNotNull(Verify.java:503)
	at io.trino.util.FastutilSetHelper$ObjectStrategy.equals(FastutilSetHelper.java:235)
	at it.unimi.dsi.fastutil.objects.ObjectOpenCustomHashSet.contains(ObjectOpenCustomHashSet.java:442)
	at io.trino.util.FastutilSetHelper.in(FastutilSetHelper.java:93)
	at io.trino.$gen.PageProjectionWork_20220225_102405_38.evaluate(Unknown Source)
	at io.trino.$gen.PageProjectionWork_20220225_102405_38.process(Unknown Source)
	at io.trino.operator.project.DictionaryAwarePageProjection$DictionaryAwarePageProjectionWork.processInternal(DictionaryAwarePageProjection.java:185)
	at io.trino.operator.project.DictionaryAwarePageProjection$DictionaryAwarePageProjectionWork.process(DictionaryAwarePageProjection.java:121)
	at io.trino.operator.project.PageProcessor$ProjectSelectedPositions.processBatch(PageProcessor.java:337)
	at io.trino.operator.project.PageProcessor$ProjectSelectedPositions.process(PageProcessor.java:209)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils.lambda$flatten$7(WorkProcessorUtils.java:296)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:338)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils.lambda$flatten$7(WorkProcessorUtils.java:296)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:338)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:240)
	at io.trino.operator.WorkProcessorUtils.lambda$processStateMonitor$3(WorkProcessorUtils.java:219)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:240)
	at io.trino.operator.WorkProcessorUtils.lambda$finishWhen$4(WorkProcessorUtils.java:234)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
	at io.trino.operator.WorkProcessorSourceOperatorAdapter.getOutput(WorkProcessorSourceOperatorAdapter.java:150)
	at io.trino.operator.Driver.processInternal(Driver.java:388)
	at io.trino.operator.Driver.lambda$processFor$9(Driver.java:292)
	at io.trino.operator.Driver.tryWithLock(Driver.java:693)
	at io.trino.operator.Driver.processFor(Driver.java:285)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1092)
	at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:488)
	at io.trino.$gen.Trino_dev____20220225_101503_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Triggering conditions

  • source is pages (so not tpch, but eg hive or memory connector)
  • filter uses IN with a list of 8+ values so that SwitchGenerationCase.SET_CONTAINS is used
  • compared values are indeterminate (eg ARRAY with NULL elements)
  • but they still have hash collisions, so that equals is invoked

See also https://github.com/trinodb/trino/issues/11193

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 17 (17 by maintainers)

Most upvoted comments

@findepi by projection case, do you mean inside conditional expressions?

projection:

select x IN (a, b, c)

filter:

select * FROM table WHERE x IN (a, b, c)
+1
sopel39 on Apr 21, 2022

Or just verify compared values aren’t indeterminate, to avoid running an algorithm with quadratic cost. Of course, the downside that an expression works in a Filter, but doesn’t work in a Projection, is a non-negligible one.

You mean at runtime or during planning?

+1
sopel39 on Apr 20, 2022

In any case, query should not fail because of IN predicate

+1
sopel39 on Apr 19, 2022

comparisons against NULL should always yield NULL results until you can for sure deduce that results is false. E.g:

(5, null) IN [(null, 5)] => null
(5, null) IN [(4, 5)] = > false
+1
sopel39 on Apr 19, 2022

I’d be happy to address this, if that’s okay?

sure, go ahead

+1
sopel39 on Mar 29, 2022

This is not related to those changes - yet before those you needed long IN list to reproduce (>32 constants). On Trino 356 it still reproduces but with:

SELECT * FROM tpch.sf1.orders WHERE ARRAY[NULL,orderkey] 
  IN ( 
  ARRAY[0,0],ARRAY[0,1],ARRAY[0,2],ARRAY[0,3],ARRAY[0,4],ARRAY[0,5],ARRAY[0,6],ARRAY[0,7],
  ARRAY[0,8],ARRAY[0,9],ARRAY[0,10],ARRAY[0,11],ARRAY[0,12],ARRAY[0,13],ARRAY[0,14], ARRAY[0,15],
  ARRAY[0,16],ARRAY[0,17],ARRAY[0,18],ARRAY[0,19],ARRAY[0,20],ARRAY[0,21],ARRAY[0,22],ARRAY[0,23], 
  ARRAY[0,24],ARRAY[0,25],ARRAY[0,26],ARRAY[0,27],ARRAY[0,28],ARRAY[0,29],ARRAY[0,30],ARRAY[0,31],
  ARRAY[0,32]
  );
+1
losipiuk on Mar 1, 2022
Read more comments on GitHub
← trino: Provide AT ZONE-equivalent functions where zone name is not constant
trino: Release notes for 320 →