datafusion: SortQueryFuzzer found a failing case on main
Describe the bug
Fuzzer failed during an unrelated change - https://github.com/apache/datafusion/actions/runs/15741542523/job/44367876525?pr=16449.
Not sure how long GitHub retains logs, so I’m also adding the actual output:
---- fuzz_cases::sort_query_fuzz::sort_query_fuzzer_runner stdout ----
[SortQueryFuzzer] Round 0, Query 0 (Config 0)
Seeds:
init_seed = 10313160656544581998
query_seed = 9760675044397831413
config_seed = 2456252688354296803
Dataset schema:
[u8_low:UInt8;N, binary:Binary;N, timestamp_us:Timestamp(Microsecond, None);N, time64_ns:Time64(Nanosecond);N, u8:UInt8;N, i32:Int32;N, duration_nanosecond:Duration(Nanosecond);N, utf8_low:Utf8;N, duration_milliseconds:Duration(Millisecond);N]
Query:
SELECT * FROM sort_fuzz_table ORDER BY time64_ns ASC
Config:
Dataset size: 286.6 KB
Number of partitions: 3
Batch size: 6
Memory limit: Unbounded
Per partition memory limit: Unbounded
Sort spill reservation bytes: 48.4 KB
Sort in place threshold bytes: 2.3 KB
[SortQueryFuzzer] Round 0, Query 0 (Config 1)
Seeds:
init_seed = 10313160656544581998
query_seed = 9760675044397831413
config_seed = 13112719340419033750
Dataset schema:
[u8_low:UInt8;N, binary:Binary;N, timestamp_us:Timestamp(Microsecond, None);N, time64_ns:Time64(Nanosecond);N, u8:UInt8;N, i32:Int32;N, duration_nanosecond:Duration(Nanosecond);N, utf8_low:Utf8;N, duration_milliseconds:Duration(Millisecond);N]
Query:
SELECT * FROM sort_fuzz_table ORDER BY time64_ns ASC
Config:
Dataset size: 286.6 KB
Number of partitions: 3
Batch size: 6
Memory limit: Unbounded
Per partition memory limit: Unbounded
Sort spill reservation bytes: 42.3 KB
Sort in place threshold bytes: 3.7 KB
[SortQueryFuzzer] Round 0, Query 0 (Config 2)
Seeds:
init_seed = 10313160656544581998
query_seed = 9760675044397831413
config_seed = 11553131583763959573
Dataset schema:
[u8_low:UInt8;N, binary:Binary;N, timestamp_us:Timestamp(Microsecond, None);N, time64_ns:Time64(Nanosecond);N, u8:UInt8;N, i32:Int32;N, duration_nanosecond:Duration(Nanosecond);N, utf8_low:Utf8;N, duration_milliseconds:Duration(Millisecond);N]
Query:
SELECT * FROM sort_fuzz_table ORDER BY time64_ns ASC
Config:
Dataset size: 286.6 KB
Number of partitions: 3
Batch size: 6
Memory limit: Unbounded
Per partition memory limit: Unbounded
Sort spill reservation bytes: 21.4 KB
Sort in place threshold bytes: 486.0 B
[SortQueryFuzzer] Round 0, Query 0 (Config 3)
Seeds:
init_seed = 10313160656544581998
query_seed = 9760675044397831413
config_seed = 7181[2061](https://github.com/apache/datafusion/actions/runs/15741542523/job/44367876525?pr=16449#step:4:2062)06220866377
Dataset schema:
[u8_low:UInt8;N, binary:Binary;N, timestamp_us:Timestamp(Microsecond, None);N, time64_ns:Time64(Nanosecond);N, u8:UInt8;N, i32:Int32;N, duration_nanosecond:Duration(Nanosecond);N, utf8_low:Utf8;N, duration_milliseconds:Duration(Millisecond);N]
Query:
SELECT * FROM sort_fuzz_table ORDER BY time64_ns ASC
Config:
Dataset size: 286.6 KB
Number of partitions: 3
Batch size: 6
Memory limit: Unbounded
Per partition memory limit: Unbounded
Sort spill reservation bytes: 43.8 KB
Sort in place threshold bytes: 3.2 KB
[SortQueryFuzzer] Round 0, Query 0 (Config 4)
Seeds:
init_seed = 10313160656544581998
query_seed = 9760675044397831413
config_seed = 17269272874247070475
Dataset schema:
[u8_low:UInt8;N, binary:Binary;N, timestamp_us:Timestamp(Microsecond, None);N, time64_ns:Time64(Nanosecond);N, u8:UInt8;N, i32:Int32;N, duration_nanosecond:Duration(Nanosecond);N, utf8_low:Utf8;N, duration_milliseconds:Duration(Millisecond);N]
Query:
SELECT * FROM sort_fuzz_table ORDER BY time64_ns ASC
Config:
Dataset size: 286.6 KB
Number of partitions: 3
Batch size: 6
Memory limit: Unbounded
Per partition memory limit: Unbounded
Sort spill reservation bytes: 45.0 KB
Sort in place threshold bytes: 2.3 KB
[SortQueryFuzzer] Round 0, Query 1 (Config 0)
Seeds:
init_seed = 10313160656544581998
query_seed = 15004039071976572201
config_seed = 11807432710583113300
Dataset schema:
[u8_low:UInt8;N, binary:Binary;N, timestamp_us:Timestamp(Microsecond, None);N, time64_ns:Time64(Nanosecond);N, u8:UInt8;N, i32:Int32;N, duration_nanosecond:Duration(Nanosecond);N, utf8_low:Utf8;N, duration_milliseconds:Duration(Millisecond);N]
Query:
SELECT * FROM sort_fuzz_table ORDER BY timestamp_us DESC LIMIT 3
Config:
Dataset size: 286.6 KB
Number of partitions: 3
Batch size: 6
Memory limit: Unbounded
Per partition memory limit: Unbounded
Sort spill reservation bytes: 25.8 KB
Sort in place threshold bytes: 2.8 KB
[SortQueryFuzzer] Round 0, Query 1 (Config 1)
Seeds:
init_seed = 10313160656544581998
query_seed = 15004039071976572201
config_seed = 759937414670321802
Dataset schema:
[u8_low:UInt8;N, binary:Binary;N, timestamp_us:Timestamp(Microsecond, None);N, time64_ns:Time64(Nanosecond);N, u8:UInt8;N, i32:Int32;N, duration_nanosecond:Duration(Nanosecond);N, utf8_low:Utf8;N, duration_milliseconds:Duration(Millisecond);N]
Query:
SELECT * FROM sort_fuzz_table ORDER BY timestamp_us DESC LIMIT 3
Config:
Dataset size: 286.6 KB
Number of partitions: 3
Batch size: 6
Memory limit: Unbounded
Per partition memory limit: Unbounded
Sort spill reservation bytes: 47.4 KB
Sort in place threshold bytes: 2.5 KB
thread 'fuzz_cases::sort_query_fuzz::sort_query_fuzzer_runner' panicked at datafusion/core/tests/fuzz_cases/sort_query_fuzz.rs:232:71:
called `Result::unwrap()` on an `Err` value: InconsistentResult { row_idx: 0, lhs_row: "+--------+--------------------------------------+--------------+---------------------------------------------------------------------------------------------+-----+-----------+-------------------------+-------------+-------------------------+", rhs_row: "+--------+------------------------------------------------+--------------+---------------------------------------------------------------------------------------------+-----+-------------+--------------------------+-------------+--------------------------+" }
To Reproduce
No response
Expected behavior
No response
Additional context
No response
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 21 (21 by maintainers)
Commits related to this issue
- Fuzzer-based reproduction and a temporary fix for #16452 — committed to AdamGS/arrow-datafusion by AdamGS a year ago
For transparency, @alamb noticed another failure: https://github.com/apache/datafusion/actions/runs/15780623790/job/44485106315
https://github.com/apache/datafusion/pull/16491 now has the fix!
I’ve built a repro that fails consistently, and the most interesting part of it is that when using the default test runtime (which is a tokio current thread variant), the test passes. It only fails with the multithreaded runtime which makes me think this is indeed a tokio futures interleaving issue. I’ll give it some more time later tonight.
I think that part may have its own issues!
I can confirm it’s related to #15770. It seems like there may be multiple errors. This is one I found tonight:
But if I comment out
update_filters()there are no failures. I think theupdate_filters()function may be the culprit of all of the evils and it’s manifesting via some sort of unstable sort / mishandling of nulls and this other random bug with Display for Date64.Bizarre… will keep pushing to find a root cause and see if we can address it. If I can’t get it sorted over the weekend @alamb I’d propose we revert the TopK specific part of that PR to unblock the rest of the project.
Perhaps this is related to the recent work from @adriangb with topk limit pushdown (once we figure out how to reliably reproduce it, we can verify by turning off the feature)
I got a very similar failure in #16447 - https://github.com/apache/datafusion/actions/runs/15744174093/job/44376646866?pr=16447
Thanks for the repro! I’ll try and spend some time hacking on it to see if I can reproduce it, maybe trying out
loomwill helpThanks for reporting!
This failure should be reproducable byI didn’t check it carefully and this reproducer is incorrect…See https://github.com/apache/datafusion/issues/16452#issuecomment-2988170885 for the correct reproducer.However, I’ve run it several times and it always passes. This could be either: