datafusion: Stack overflow with LEAD and LAG functions
Describe the bug
The following query causes stack overflow in rust
SELECT seq
FROM (
SELECT seq,
LEAD(seq) OVER (ORDER BY seq) AS next_seq
FROM parquet_table
) AS subquery
WHERE next_seq - seq > 1
thread 'main' has overflowed its stack
error: process didn't exit successfully: `target\debug\seq.exe` (exit code: 0xc00000fd, STATUS_STACK_OVERFLOW)
To Reproduce
use datafusion::prelude::SessionContext;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let ctx = datafusion::prelude::SessionContext::new();
ctx
.register_parquet(
"parquet_table",
"batches.parquet",
datafusion::prelude::ParquetReadOptions::default(),
)
.await?;
let sql_query = "
SELECT seq
FROM (
SELECT seq,
LEAD(seq) OVER (ORDER BY seq) AS next_seq
FROM parquet_table
) AS subquery
WHERE next_seq - seq > 1
";
let df = ctx.sql(sql_query).await?;
df.show().await?;
Ok(())
}
Expected behavior
The query should detect gaps in the seq number.
Additional context
datafusion = { version = "41.0.0", features = ["parquet", "default"] }
arrow = "53.0.0"
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 24 (18 by maintainers)
https://github.com/apache/datafusion-sqlparser-rs/issues/1465
Given this is a stack overflow in sqlparser, perhaps we can move the ticket there and try to solve it in the sqlparser library?
Imagine a service which implements an application to analyze structured log events. Where log events are stored in parquet files. The application is a web server which accepts adhoc SQL queries and runs them to extract useful patterns from log events. In such scenario the malicious user can craft complex SQL query to cause stack overflow and mount remote code execution attack.
In this scenario no matter how big the stack size is. It will always be possible to craft more complex query to cause the problem.
I found stack overflow not happening if stack size is set to 16MiB(default is 1 MiB on Windows), so you can reproduce this issue on linux by making stack size smaller.
I don’t think this is an bug, but it could be better if we document it somewhere(document
stack size is very small by default on windows).Also of note, optimized build significantly reduce memory usage, 1 MiB may be enough.