trino: Query fails with EXCEEDED_LOCAL_MEMORY_LIMIT on 474 when JVM set higher

SELECT kind, max(length(payload)) max_payload_len
FROM hive.default.test_payload_tbl
GROUP BY 1

fails with INSUFFICIENT_RESOURCES when JVM is set to 86GB on EC2 r6a.4xlarge

io.trino.ExceededMemoryLimitException: Query exceeded per-node memory limit of 25.80GB [Allocated: 25.79GB, Delta: 14.39MB, Top Consumers: {ScanFilterAndProjectOperator=25.78GB, HashAggregationOperator=7.92MB, PagePartitioner=4.08MB}]
	at io.trino.ExceededMemoryLimitException.exceededLocalUserMemoryLimit(ExceededMemoryLimitException.java:40)
	at io.trino.memory.QueryContext.enforceUserMemoryLimit(QueryContext.java:330)
	at io.trino.memory.QueryContext.updateUserMemory(QueryContext.java:165)
	at io.trino.memory.QueryContext.lambda$addTaskContext$0(QueryContext.java:250)
	at io.trino.memory.QueryContext$QueryMemoryReservationHandler.reserveMemory(QueryContext.java:311)
	at io.trino.memory.context.RootAggregatedMemoryContext.updateBytes(RootAggregatedMemoryContext.java:37)
	at io.trino.memory.context.ChildAggregatedMemoryContext.updateBytes(ChildAggregatedMemoryContext.java:38)
	at io.trino.memory.context.ChildAggregatedMemoryContext.updateBytes(ChildAggregatedMemoryContext.java:38)
	at io.trino.memory.context.ChildAggregatedMemoryContext.updateBytes(ChildAggregatedMemoryContext.java:38)
	at io.trino.memory.context.SimpleLocalMemoryContext.setBytes(SimpleLocalMemoryContext.java:65)
	at io.trino.operator.OperatorContext$InternalLocalMemoryContext.setBytes(OperatorContext.java:701)
	at io.trino.operator.ScanFilterAndProjectOperator$SplitToPages.lambda$processPageSource$4(ScanFilterAndProjectOperator.java:299)
	at io.trino.operator.WorkProcessorUtils$BlockingProcess.process(WorkProcessorUtils.java:210)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:423)
	at io.trino.operator.WorkProcessorUtils.lambda$flatten$6(WorkProcessorUtils.java:317)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:359)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:423)
	at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:346)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:423)
	at io.trino.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:261)
	at io.trino.operator.WorkProcessorUtils.lambda$processStateMonitor$2(WorkProcessorUtils.java:240)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:423)
	at io.trino.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:261)
	at io.trino.operator.WorkProcessorUtils.lambda$finishWhen$3(WorkProcessorUtils.java:255)
	at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:423)
	at io.trino.operator.WorkProcessorSourceOperatorAdapter.getOutput(WorkProcessorSourceOperatorAdapter.java:133)
	at io.trino.operator.Driver.processInternal(Driver.java:403)
	at io.trino.operator.Driver.lambda$process$8(Driver.java:306)
	at io.trino.operator.Driver.tryWithLock(Driver.java:709)
	at io.trino.operator.Driver.process(Driver.java:298)
	at io.trino.operator.Driver.processForDuration(Driver.java:269)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:890)
	at io.trino.execution.executor.dedicated.SplitProcessor.run(SplitProcessor.java:77)
	at io.trino.execution.executor.dedicated.TaskEntry$VersionEmbedderBridge.lambda$run$0(TaskEntry.java:201)
	at io.trino.$gen.Trino_474____20250331_090836_2.run(Unknown Source)
	at io.trino.execution.executor.dedicated.TaskEntry$VersionEmbedderBridge.run(TaskEntry.java:202)
	at io.trino.execution.executor.scheduler.FairScheduler.runTask(FairScheduler.java:177)
	at io.trino.execution.executor.scheduler.FairScheduler.lambda$submit$0(FairScheduler.java:164)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1575)

Same query finishes successfully when JVM is set to 9GB on EC2 r5.large

Both has spill-enabled=TRUE

show create table hive.default.test_payload_tbl;

CREATE TABLE hive.default.test_payload_tbl (
   kind varchar(5),
   payload varchar
)
WITH (
   format = 'ORC'
)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 20 (17 by maintainers)

Most upvoted comments

I can approve the the PR solves the issue. Thanks!

@guyco33 can you check whether the PR that was merged fixes the issue?

I can approve that problem starts after the change done in SourcePage PR

I’ve tested the following snapshots:

Snapshot 472-10-GD2C414A - OK

* d2c414a0e9 - (test) Limit max bigquery.metadata.parallelism to 32 (5 weeks ago) <Yuya Ebihara>
* 20f03d7d6a - Rename plugin folder for Ranger in binaries (5 weeks ago) <Manfred Moser>
* 496f020a72 - Add AI functions to plugin list (5 weeks ago) <Manfred Moser>
* d46bb32eb7 - Add new hudi session property: ignore_absent_partitions (10 months ago) <Will Zhang>
* 8ae85872e1 - Fix HUDI connector gets stuck when an exception occurs (10 months ago) <Will Zhang>
* 789eb1886b - Add download links for plugins (5 weeks ago) <Manfred Moser>
* 69ebd77656 - Remove Phoenix connector (5 weeks ago) <Manfred Moser>
* bc5d4fd6a4 - Remove Kudu connector (5 weeks ago) <Manfred Moser>
* 647d51105a - Fix case-sensitivity issue with views and column masks (5 months ago) <Krzysztof Sobolewski>
* 9823c63eb1 - [maven-release-plugin] prepare for next development iteration (5 weeks ago) <Martin Traverso>
* 5daadac2d6 - (tag: 472) [maven-release-plugin] prepare release 472 (5 weeks ago) <Martin Traverso>

Snapshopt 472-29-GE437B16 - Issue starts

* e437b16694 - (HEAD -> test-2) Mark ConnectorPageSource.getNextPage() deprecated for removal (5 months ago) <Dain Sundstrom>
* 7865b2dbf5 - Convert Hive, Hudi, and Delta to SourcePage and TransformCPS (5 months ago) <Dain Sundstrom>
* 20a0026d18 - Remove unused Hive ReaderColumns and ReaderPageSource (5 months ago) <Dain Sundstrom>
* 090548f2a5 - Convert Hive RcFilePageSourceFactory to projectColumnDereferences (5 months ago) <Dain Sundstrom>
* 7b00798faa - Convert Hive LinePageSourceFactory to projectColumnDereferences (5 months ago) <Dain Sundstrom>
* c33c39c5ab - Convert Hive AvroPageSourceFactory to projectColumnDereferences (5 months ago) <Dain Sundstrom>
* b2f6c7bbe3 - Handle column dereference directly in Hive OrcPageSourceFactory (5 months ago) <Dain Sundstrom>
* e193a5afcb - Cleanup Hive OrcPageSourceFactory (5 months ago) <Dain Sundstrom>
* af6a86c0dd - Handle column dereference directly in Hive ParquetPageSourceFactory (5 months ago) <Dain Sundstrom>
* 1ac306857c - Change Delta CheckpointEntryIterator ADD to use existing base column (5 months ago) <Dain Sundstrom>
* 42b5c97ef2 - Move constants and static classes out of HivePageSource (5 months ago) <Dain Sundstrom>
* 9d204462ae - Convert Iceberg to SourePage and TransformConnectorPageSource (5 months ago) <Dain Sundstrom>
* f126462f1a - Add TransformConnectorPageSource to Hive (5 months ago) <Dain Sundstrom>
* b773f3d885 - Simplify Hive and Iceberg projectSufficientColumns (5 months ago) <Dain Sundstrom>
* 91a84b309b - Convert ORC and Parquet to SourcePage (5 months ago) <Dain Sundstrom>
* caf7f19902 - Convert non-object store connectors to SourcePage (5 months ago) <Dain Sundstrom>
* 9be7098cad - Add SourcePage interface for ConnectorPageSource (5 months ago) <Dain Sundstrom>
* 36d81a0306 - Replace process bytes lazy load monitor with state monitor in SFP (5 months ago) <Dain Sundstrom>
* c92a2ba2f8 - Move Iceberg reader early exit checks to start of method (5 months ago) <Dain Sundstrom>
* d2c414a0e9 - (test) Limit max bigquery.metadata.parallelism to 32 (5 weeks ago) <Yuya Ebihara>
* 20f03d7d6a - Rename plugin folder for Ranger in binaries (5 weeks ago) <Manfred Moser>
* 496f020a72 - Add AI functions to plugin list (5 weeks ago) <Manfred Moser>
* d46bb32eb7 - Add new hudi session property: ignore_absent_partitions (10 months ago) <Will Zhang>
* 8ae85872e1 - Fix HUDI connector gets stuck when an exception occurs (10 months ago) <Will Zhang>
* 789eb1886b - Add download links for plugins (5 weeks ago) <Manfred Moser>
* 69ebd77656 - Remove Phoenix connector (5 weeks ago) <Manfred Moser>
* bc5d4fd6a4 - Remove Kudu connector (5 weeks ago) <Manfred Moser>
* 647d51105a - Fix case-sensitivity issue with views and column masks (5 months ago) <Krzysztof Sobolewski>
* 9823c63eb1 - [maven-release-plugin] prepare for next development iteration (5 weeks ago) <Martin Traverso>
* 5daadac2d6 - (tag: 472) [maven-release-plugin] prepare release 472 (5 weeks ago) <Martin Traverso>

@dain FYI