datafusion: Comparing a `Timestamp` to a `Date32` fails

Describe the bug Comparing a Timestamp to a Date32 fails.

To Reproduce Add the following test to timestamps.slt:

# Test that we can compare a timestamp to a string casted to a date
query C rowsort
select * from foo where ts > '2000-01-01'::date;
----
2 2000-02-01T00:00:00
3 2000-03-01T00:00:00

This fails w/:

[timestamps.slt] Running query: "select * from foo where ts > '2000-01-01'::date;"
Error: SqlLogicTest(query failed: DataFusion error: Internal error: The type of Timestamp(Nanosecond, None) > Date32 of binary physical should be same. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

Expected behavior It works.

Additional context Tested on https://github.com/apache/arrow-datafusion/commit/5c558e9f6b3fe29ef0d5f78ec9465852d020d989 .

This is likely due to a like in temporal_coercion as well as this workaround:

https://github.com/apache/arrow-datafusion/blob/5c558e9f6b3fe29ef0d5f78ec9465852d020d989/datafusion/optimizer/src/type_coercion.rs#L232-L239

Also see #3419.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 20 (20 by maintainers)

Most upvoted comments

Waitin arrow-rs 32.0.0 to proceed

comphead on Jan 12, 2023

I found something weird. first logical plan failed silently on cast. second is arrow cast acts not consistently on the same inputs. so if I run through first test

[datafusion/expr/src/expr_schema.rs:268] &this_type = Date32
[datafusion/expr/src/expr_schema.rs:269] &cast_to_type = Timestamp(
    Nanosecond,
    None,
)
[datafusion/expr/src/expr_schema.rs:270] can_cast_types(&this_type, cast_to_type) = true

but if I run second test

[datafusion/expr/src/expr_schema.rs:268] &this_type = Date32
[datafusion/expr/src/expr_schema.rs:269] &cast_to_type = Timestamp(
    Nanosecond,
    None,
)
[datafusion/expr/src/expr_schema.rs:270] can_cast_types(&this_type, cast_to_type) = false

I’m looking what in arrow-rs can cause such non determined behaviuor @liukun4515 @crepererum @alamb

Edit: looks like cargo clean helped to make the second test finally work. Sorry, facepalm. Creating PR soon

comphead on Jan 10, 2023

@crepererum @waitingkuo I can take that as a next ticket related to timestamp issues

comphead on Dec 16, 2022