datafusion: Regression: `Invalid comparison operation: Utf8 == Utf8View` error during LEFT ANTI JOIN

Describe the bug

Between 42.2.0 and 43.0.0 there appears to have been a regression that introduced an error:

External(ArrowError(InvalidArgumentError("Invalid comparison operation: Utf8 == Utf8View"), None))

Note that the error happens at the plan execution phase, i.e. plan validation passes successfully.

To Reproduce

Minimal repro is:

  • Read data from CSV (Utf8 columns)
  • Read data from Parquet (Utf8View)
  • Do a LEFT ANTI JOIN

Including a test project with sample data: datafusion-13510.zip

Physical plan:

CoalesceBatchesExec: target_batch_size=8192
  HashJoinExec: mode=Partitioned, join_type=LeftAnti, on=[(date@0, date@0), (city@1, city@1)]
    CoalesceBatchesExec: target_batch_size=8192
      RepartitionExec: partitioning=Hash([date@0, city@1], 16), input_partitions=16
        RepartitionExec: partitioning=RoundRobinBatch(16), input_partitions=1
          CsvExec: file_groups={1 group: [[home/.../datafusion-13510/data/data2.csv]]}, projection=[date, city, population], has_header=true
    CoalesceBatchesExec: target_batch_size=8192
      RepartitionExec: partitioning=Hash([date@0, city@1], 16), input_partitions=1
        ParquetExec: file_groups={1 group: [[home/.../datafusion-13510/data/data1.parquet]]}, projection=[date, city]

Expected behavior

No error / error during planning if some operation is invalid

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (15 by maintainers)

Most upvoted comments

Awesome, thanks @alamb! Your fix makes sense to me 👍🏾

Hello! I’ve dug into this issue a bit and it seems that the problem here arises from the fact that the table name for each table scan in the plan defaults to UNNAMED_TABLE ("?table?").

Because of this, column references in the “on” portion of the join node are ambiguous. In fact, if we dedup the references by either changing column names or qualifying at least one table in the example with an alias, it works because type coercion is able to determine that a cast is necessary. Without this, the data type lookup for the columns during type coercion will yield that associated with the first matched field in the schema (the left-hand side). Thus, there appears to be no cast necessary.

Other join types (e.g. inner) fail earlier in planning with a DuplicateQualifiedField error because they compute the joined schema, performing duplicate name checks as a result. I’m thinking a good path forward here may be to perform this validation for all join types? That way we fail earlier in planning.

Would love some feedback/thoughts! Thanks!

(FYI @alamb I tested #13370 rebased off the latest main and it does not fix the issue. But this is expected based on the description above! 😅 )

I added this to the list of items that i think we should fix before releasing version 45.0.0:

Phew, finally got the minimal repro:

  • Read data from CSV (Utf8 columns)
  • Read data from Parquet (Utf8View)
  • Do a LEFT ANTI JOIN

Including the test project with sample data: datafusion-13510.zip

Still not sure why the same exact scenario didn’t break when using datafusion-cli.

Will update the ticket description with the repro.

@findepi was slammed this week unfortunately. I will re-test with latest master and try to narrow the issue down this weekend.

Related to #13568? I filed that as a separate issue, but it seems like they might have the same root cause.