datafusion: Regression: `Invalid comparison operation: Utf8 == Utf8View` error during LEFT ANTI JOIN
Describe the bug
Between 42.2.0 and 43.0.0 there appears to have been a regression that introduced an error:
External(ArrowError(InvalidArgumentError("Invalid comparison operation: Utf8 == Utf8View"), None))
Note that the error happens at the plan execution phase, i.e. plan validation passes successfully.
To Reproduce
Minimal repro is:
- Read data from CSV (
Utf8columns) - Read data from Parquet (
Utf8View) - Do a
LEFT ANTI JOIN
Including a test project with sample data: datafusion-13510.zip
Physical plan:
CoalesceBatchesExec: target_batch_size=8192
HashJoinExec: mode=Partitioned, join_type=LeftAnti, on=[(date@0, date@0), (city@1, city@1)]
CoalesceBatchesExec: target_batch_size=8192
RepartitionExec: partitioning=Hash([date@0, city@1], 16), input_partitions=16
RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1
CsvExec: file_groups={1 group: [[home/.../datafusion-13510/data/data2.csv]]}, projection=[date, city, population], has_header=true
CoalesceBatchesExec: target_batch_size=8192
RepartitionExec: partitioning=Hash([date@0, city@1], 16), input_partitions=1
ParquetExec: file_groups={1 group: [[home/.../datafusion-13510/data/data1.parquet]]}, projection=[date, city]
Expected behavior
No error / error during planning if some operation is invalid
Additional context
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (15 by maintainers)
Awesome, thanks @alamb! Your fix makes sense to me 👍🏾
Hello! I’ve dug into this issue a bit and it seems that the problem here arises from the fact that the table name for each table scan in the plan defaults to
UNNAMED_TABLE("?table?").Because of this, column references in the “on” portion of the join node are ambiguous. In fact, if we dedup the references by either changing column names or qualifying at least one table in the example with an alias, it works because type coercion is able to determine that a cast is necessary. Without this, the data type lookup for the columns during type coercion will yield that associated with the first matched field in the schema (the left-hand side). Thus, there appears to be no cast necessary.
Other join types (e.g. inner) fail earlier in planning with a
DuplicateQualifiedFielderror because they compute the joined schema, performing duplicate name checks as a result. I’m thinking a good path forward here may be to perform this validation for all join types? That way we fail earlier in planning.Would love some feedback/thoughts! Thanks!
(FYI @alamb I tested #13370 rebased off the latest main and it does not fix the issue. But this is expected based on the description above! 😅 )
I added this to the list of items that i think we should fix before releasing version 45.0.0:
Phew, finally got the minimal repro:
Utf8columns)Utf8View)LEFT ANTI JOINIncluding the test project with sample data: datafusion-13510.zip
Still not sure why the same exact scenario didn’t break when using
datafusion-cli.Will update the ticket description with the repro.
@findepi was slammed this week unfortunately. I will re-test with latest master and try to narrow the issue down this weekend.
Related to #13568? I filed that as a separate issue, but it seems like they might have the same root cause.