trino: Error "Source subPlans not found for exchange node"

I updated my Trino cluster from version 455 to 461, and I’m receiving the following error: ‘Source subPlans not found for exchange node.’ I’m using dbt, and in a scenario with more than 500 models, this error occurs only in a specific query that, as far as I can tell, doesn’t have anything unusual. It’s not intermittent; it happens in every run.

My Trino cluster is configured with fault tolerance."

query example

create or replace table "schema_placeholder"."db_placeholder"."target_table"
as (
WITH cte_target_table AS (
  SELECT
    main_table.id_col1,
    main_table.id_col2,
    main_table.id_col3,
    main_table.id_col4,
    main_table.description_col5,
    joined_table1.value_col6,
    joined_table2.datetime_col7
  FROM "schema1"."db1"."main_table" AS main_table
  INNER JOIN "schema2"."db2"."joined_table1" AS joined_table1
    ON joined_table1.id_col1 = main_table.id_col1
  INNER JOIN "schema3"."db3"."joined_table2" AS joined_table2
    ON joined_table2.id_col3 = main_table.id_col3
    AND joined_table2.status_col = 'closed'
  INNER JOIN "schema4"."db4"."joined_table3" AS joined_table3
    ON joined_table3.id_col4 = main_table.id_col4
    AND joined_table3.status_col = 'active'
)

SELECT *
FROM cte_target_table
)

Stacktrace

com.google.common.base.VerifyException: Source subPlans not found for exchange node
	at com.google.common.base.Verify.verify(Verify.java:126)
	at io.trino.sql.planner.AdaptivePlanner$ExchangeSourceIdToSubPlanCollector.visitExchange(AdaptivePlanner.java:396)
	at io.trino.sql.planner.AdaptivePlanner$ExchangeSourceIdToSubPlanCollector.visitExchange(AdaptivePlanner.java:375)
	at io.trino.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:278)
	at io.trino.sql.planner.SimplePlanVisitor.visitPlan(SimplePlanVisitor.java:26)
	at io.trino.sql.planner.AdaptivePlanner$ExchangeSourceIdToSubPlanCollector.visitExchange(AdaptivePlanner.java:384)
	at io.trino.sql.planner.AdaptivePlanner$ExchangeSourceIdToSubPlanCollector.visitExchange(AdaptivePlanner.java:375)
	at io.trino.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:278)
	at io.trino.sql.planner.SimplePlanVisitor.visitPlan(SimplePlanVisitor.java:26)
	at io.trino.sql.planner.SimplePlanVisitor.visitPlan(SimplePlanVisitor.java:19)
	at io.trino.sql.planner.plan.PlanVisitor.visitTableWriter(PlanVisitor.java:134)
	at io.trino.sql.planner.plan.TableWriterNode.accept(TableWriterNode.java:173)
	at io.trino.sql.planner.SimplePlanVisitor.visitPlan(SimplePlanVisitor.java:26)
	at io.trino.sql.planner.AdaptivePlanner$ExchangeSourceIdToSubPlanCollector.visitExchange(AdaptivePlanner.java:384)
	at io.trino.sql.planner.AdaptivePlanner$ExchangeSourceIdToSubPlanCollector.visitExchange(AdaptivePlanner.java:375)
	at io.trino.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:278)
	at io.trino.sql.planner.SimplePlanVisitor.visitPlan(SimplePlanVisitor.java:26)
	at io.trino.sql.planner.AdaptivePlanner$ExchangeSourceIdToSubPlanCollector.visitExchange(AdaptivePlanner.java:384)
	at io.trino.sql.planner.AdaptivePlanner$ExchangeSourceIdToSubPlanCollector.visitExchange(AdaptivePlanner.java:375)
	at io.trino.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:278)
	at io.trino.sql.planner.SimplePlanVisitor.visitPlan(SimplePlanVisitor.java:26)
	at io.trino.sql.planner.SimplePlanVisitor.visitPlan(SimplePlanVisitor.java:19)
	at io.trino.sql.planner.plan.PlanVisitor.visitTableFinish(PlanVisitor.java:169)
	at io.trino.sql.planner.plan.TableFinishNode.accept(TableFinishNode.java:105)
	at io.trino.sql.planner.SimplePlanVisitor.visitPlan(SimplePlanVisitor.java:26)
	at io.trino.sql.planner.SimplePlanVisitor.visitPlan(SimplePlanVisitor.java:19)
	at io.trino.sql.planner.plan.PlanVisitor.visitOutput(PlanVisitor.java:49)
	at io.trino.sql.planner.plan.OutputNode.accept(OutputNode.java:82)
	at io.trino.sql.planner.AdaptivePlanner.optimize(AdaptivePlanner.java:143)
	at io.trino.execution.scheduler.faulttolerant.EventDrivenFaultTolerantQueryScheduler$Scheduler.optimizePlan(EventDrivenFaultTolerantQueryScheduler.java:1130)
	at io.trino.execution.scheduler.faulttolerant.EventDrivenFaultTolerantQueryScheduler$Scheduler.optimize(EventDrivenFaultTolerantQueryScheduler.java:1089)
	at io.trino.execution.scheduler.faulttolerant.EventDrivenFaultTolerantQueryScheduler$Scheduler.schedule(EventDrivenFaultTolerantQueryScheduler.java:1021)
	at io.trino.execution.scheduler.faulttolerant.EventDrivenFaultTolerantQueryScheduler$Scheduler.run(EventDrivenFaultTolerantQueryScheduler.java:847)
	at io.trino.$gen.Trino_461_3_g6bad56f____20241021_141224_2.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1575)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (15 by maintainers)

Most upvoted comments

Ok - I have reproduction - working on a fix.

@losipiuk how did you reproduce this?

https://gist.github.com/losipiuk/ede59b1b120388d5bfad2906458804eb

Run above in IcebergQueryRunnerWithTaskRetries

Thanks @ndrluis . I will look into that.

As a workaround use

SET SESSION fault_tolerant_execution_adaptive_query_planning_enabled=false

or set fault-tolerant-execution-adaptive-query-planning-enabled config property to false.

cc: @gaurav8297