datafusion: Stack overflow with LEAD and LAG functions

Describe the bug

The following query causes stack overflow in rust

SELECT seq
  FROM (
    SELECT seq,
      LEAD(seq) OVER (ORDER BY seq) AS next_seq
      FROM parquet_table
  ) AS subquery
  WHERE next_seq - seq > 1
thread 'main' has overflowed its stack
error: process didn't exit successfully: `target\debug\seq.exe` (exit code: 0xc00000fd, STATUS_STACK_OVERFLOW)

To Reproduce

use datafusion::prelude::SessionContext;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
  let ctx = datafusion::prelude::SessionContext::new();
  ctx
    .register_parquet(
      "parquet_table",
      "batches.parquet",
      datafusion::prelude::ParquetReadOptions::default(),
    )
    .await?;

  let sql_query = "
  SELECT seq
  FROM (
    SELECT seq,
      LEAD(seq) OVER (ORDER BY seq) AS next_seq
      FROM parquet_table
  ) AS subquery
  WHERE next_seq - seq > 1
  ";

  let df = ctx.sql(sql_query).await?;

  df.show().await?;
  Ok(())
}

Expected behavior

The query should detect gaps in the seq number.

Additional context

datafusion = { version = "41.0.0", features = ["parquet", "default"] }
arrow = "53.0.0"

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 24 (18 by maintainers)

Most upvoted comments

Given this is a stack overflow in sqlparser, perhaps we can move the ticket there and try to solve it in the sqlparser library?

I think such stack overflow won’t happen if one of those requirement is meet

Imagine a service which implements an application to analyze structured log events. Where log events are stored in parquet files. The application is a web server which accepts adhoc SQL queries and runs them to extract useful patterns from log events. In such scenario the malicious user can craft complex SQL query to cause stack overflow and mount remote code execution attack.

In this scenario no matter how big the stack size is. It will always be possible to craft more complex query to cause the problem.

I found stack overflow not happening if stack size is set to 16MiB(default is 1 MiB on Windows), so you can reproduce this issue on linux by making stack size smaller.

I don’t think this is an bug, but it could be better if we document it somewhere(document stack size is very small by default on windows).

Also of note, optimized build significantly reduce memory usage, 1 MiB may be enough.