iceberg: spark sql MERGE INTO There is an error Error: Error running query: java.lang.IllegalArgumentException: Can't zip RDDs with unequal numbers of partitions: List(1, 0) (state=,code=0)
CREATE TABLE IF NOT EXISTS cdp.test_merge_001(offline_channel STRING COMMENT '_pk',unique_key STRING COMMENT '_ck',open_id STRING COMMENT '',mobile string COMMENT '_ck',hobby STRING COMMENT '',activity_time STRING COMMENT '' )
USING iceberg;
CREATE TABLE IF NOT EXISTS cdp.test_merge_002(offline_channel STRING COMMENT '_pk',unique_key STRING COMMENT '_ck',open_id STRING COMMENT '',mobile string COMMENT '_ck',hobby STRING COMMENT '',activity_time STRING COMMENT '' )
USING iceberg;
this is table test_merge_002:

this is table test_merge_001:

but run this sql,appear error

MERGE INTO cdp.test_merge_002
tt1 USING
(SELECT * FROM cdp.test_merge_001) tt2 ON ( tt1.unique_key = tt2.unique_key AND tt1.mobile = tt2.mobile) WHEN MATCHED THEN UPDATE SET tt1.offline_channel = tt2.offline_channel,
tt1.unique_key = tt2.unique_key,
tt1.open_id = tt2.open_id,
tt1.mobile = tt2.mobile,
tt1.hobby = tt2.hobby,
tt1.activity_time = tt2.activity_time WHEN NOT MATCHED THEN INSERT *
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 21 (15 by maintainers)
I think this rule is fixing it for Spark
https://github.com/apache/spark/blob/8f0fef18438aa8fb07f5ed885ffad1339992f102/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala#L128