trino: java.lang.IndexOutOfBoundsException: with Presto 331

2020-03-20T02:23:58.914Z        ERROR   SplitRunner-147-94      io.prestosql.execution.executor.TaskExecutor    Error processing Split 20200320_021857_00999_pg3ku.3.3-47  (start = 1.654072930317E7, wall = 301460 ms, cpu = 74010 ms, wait = 190255 ms, calls = 580)
java.lang.IndexOutOfBoundsException: Index (658) is greater than or equal to list size (0)
:
        at io.prestosql.operator.LookupJoinOperator$JoinProcessor.lambda$processProbe$3(LookupJoinOperator.java:477)
        at io.prestosql.operator.PartitionedLookupSourceFactory$SpillAwareLookupSourceProvider.withLease(PartitionedLookupSourceFactory.java:438)
        at io.prestosql.operator.LookupJoinOperator$JoinProcessor.processProbe(LookupJoinOperator.java:474)
        at io.prestosql.operator.LookupJoinOperator$JoinProcessor.getOutput(LookupJoinOperator.java:389)
        at io.prestosql.operator.LookupJoinOperator$JoinProcessor.process(LookupJoinOperator.java:686)
        at io.prestosql.operator.LookupJoinOperator$JoinProcessor.process(LookupJoinOperator.java:139)
        at io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:319)
        at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
        at io.prestosql.operator.WorkProcessorOperatorAdapter.getOutput(WorkProcessorOperatorAdapter.java:188)
        at io.prestosql.operator.Driver.processInternal(Driver.java:379)
        at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
        at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
        at io.prestosql.operator.Driver.processFor(Driver.java:276)
        at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
        at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
        at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
        at io.prestosql.$gen.Presto_331____20200319_214358_2.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)


2020-03-20T02:23:58.914Z        ERROR   SplitRunner-147-94      io.prestosql.execution.executor.TaskExecutor    Error processing Split 20200320_021857_00999_pg3ku.3.3-47  (start = 1.654072930317E7, wall = 301460 ms, cpu = 74010 ms, wait = 190255 ms, calls = 580)
java.lang.IndexOutOfBoundsException: Index (658) is greater than or equal to list size (0)
        at it.unimi.dsi.fastutil.objects.ObjectArrayList.get(ObjectArrayList.java:310)
        at io.prestosql.operator.LookupJoinOperator$JoinProcessor.processProbe(LookupJoinOperator.java:474)
        at io.prestosql.operator.LookupJoinOperator$JoinProcessor.getOutput(LookupJoinOperator.java:389)
        at io.prestosql.operator.LookupJoinOperator$JoinProcessor.process(LookupJoinOperator.java:686)
        at io.prestosql.operator.LookupJoinOperator$JoinProcessor.process(LookupJoinOperator.java:139)
        at io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:319)
        at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
        at io.prestosql.operator.WorkProcessorOperatorAdapter.getOutput(WorkProcessorOperatorAdapter.java:188)
        at io.prestosql.operator.Driver.processInternal(Driver.java:379)
        at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
        at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
        at io.prestosql.operator.Driver.processFor(Driver.java:276)
        at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
        at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
        at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
        at io.prestosql.$gen.Presto_331____20200319_214358_2.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 15 (8 by maintainers)

Most upvoted comments

I’m seeing a very similar error with presto-server-331 while attempting a query using the hive-hadoop2 connector:

2020-04-30T23:08:40.649Z	ERROR	SplitRunner-15-75	io.prestosql.execution.executor.TaskExecutor	Error processing Split 20200430_225617_00112_5yc46.2.0-74  (start = 3.820089125954287E10, wall = 724510 ms, cpu = 37699 ms, wait = 1254 ms, calls = 51)
java.lang.IndexOutOfBoundsException: Index (27703) is greater than or equal to list size (0)
	at it.unimi.dsi.fastutil.objects.ObjectArrayList.get(ObjectArrayList.java:310)
	at io.prestosql.$gen.PagesHashStrategy_20200430_230746_202.positionEqualsRowIgnoreNulls(Unknown Source)
	at io.prestosql.operator.PagesHash.positionEqualsCurrentRowIgnoreNulls(PagesHash.java:208)
	at io.prestosql.operator.PagesHash.getAddressIndex(PagesHash.java:162)
	at io.prestosql.operator.JoinHash.getJoinPosition(JoinHash.java:91)
	at io.prestosql.operator.PartitionedLookupSource.getJoinPosition(PartitionedLookupSource.java:143)
	at io.prestosql.operator.JoinProbe.getCurrentJoinPosition(JoinProbe.java:87)
	at io.prestosql.operator.LookupJoinOperator$JoinProcessor.advanceProbePosition(LookupJoinOperator.java:639)
	at io.prestosql.operator.LookupJoinOperator$JoinProcessor.processProbe(LookupJoinOperator.java:554)
	at io.prestosql.operator.LookupJoinOperator$JoinProcessor.lambda$processProbe$3(LookupJoinOperator.java:477)
	at io.prestosql.operator.PartitionedLookupSourceFactory$SpillAwareLookupSourceProvider.withLease(PartitionedLookupSourceFactory.java:438)
	at io.prestosql.operator.LookupJoinOperator$JoinProcessor.processProbe(LookupJoinOperator.java:474)
	at io.prestosql.operator.LookupJoinOperator$JoinProcessor.getOutput(LookupJoinOperator.java:389)
	at io.prestosql.operator.LookupJoinOperator$JoinProcessor.process(LookupJoinOperator.java:686)
	at io.prestosql.operator.LookupJoinOperator$JoinProcessor.process(LookupJoinOperator.java:139)
	at io.prestosql.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:319)
	at io.prestosql.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:372)
	at io.prestosql.operator.WorkProcessorOperatorAdapter.getOutput(WorkProcessorOperatorAdapter.java:188)
	at io.prestosql.operator.Driver.processInternal(Driver.java:379)
	at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
	at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
	at io.prestosql.operator.Driver.processFor(Driver.java:276)
	at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
	at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
	at io.prestosql.$gen.Presto_331____20200430_225457_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

This occurred during a very simple query, i.e. SELECT COUNT(1) FROM example_table.

I believe the cause is that the S3 API is returning 503 responses while attempting to rate-limit PrestoSQL’s requests when it attempts to fetch the data. While the query that was logging the above was executing, we tried another query, and got the following error:

io.prestosql.spi.PrestoException: Failed to list directory: s3://REDACTED/REDACTED
	at io.prestosql.plugin.hive.util.HiveFileIterator$FileStatusIterator.processException(HiveFileIterator.java:185)
	at io.prestosql.plugin.hive.util.HiveFileIterator$FileStatusIterator.hasNext(HiveFileIterator.java:164)
	at io.prestosql.plugin.hive.util.HiveFileIterator.computeNext(HiveFileIterator.java:80)
	at io.prestosql.plugin.hive.util.HiveFileIterator.computeNext(HiveFileIterator.java:38)
	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
	at java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1811)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
	at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:169)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)
	at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
	at io.prestosql.plugin.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:290)
	at io.prestosql.plugin.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:107)
	at io.prestosql.plugin.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:216)
	at io.prestosql.plugin.hive.util.ResumableTasks$1.run(ResumableTasks.java:38)
	at io.prestosql.$gen.Presto_331____20200430_225348_2.run(Unknown Source)
	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: ; S3 Extended Request ID: null), S3 Extended Request ID: null
	at io.prestosql.plugin.hive.s3.PrestoS3FileSystem$1.hasNext(PrestoS3FileSystem.java:324)
	at io.prestosql.plugin.hive.util.HiveFileIterator$FileStatusIterator.hasNext(HiveFileIterator.java:161)
	... 19 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: ; S3 Extended Request ID: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1726)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1381)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1127)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:784)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:745)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4992)
	at com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:938)
	at io.prestosql.plugin.hive.s3.PrestoS3FileSystem$2.computeNext(PrestoS3FileSystem.java:545)
	at io.prestosql.plugin.hive.s3.PrestoS3FileSystem$2.computeNext(PrestoS3FileSystem.java:535)
	at com.google.common.collect.AbstractSequentialIterator.next(AbstractSequentialIterator.java:75)
	at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
	at com.google.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1332)
	at io.prestosql.plugin.hive.s3.PrestoS3FileSystem$1.hasNext(PrestoS3FileSystem.java:321)
	... 20 more

When we get into this state, we can sometimes get simple queries to work, such as SELECT * FROM example_table LIMIT 10. So it seems to be related to PrestoSQL’s exception handling with regard to the rate limiting.

mpontillo on May 6, 2020