prql: Stack overflow on Windows

See https://github.com/PRQL/prql/pull/2700#issuecomment-1575206792 and #2713 and https://github.com/PRQL/prql/issues/2781#issuecomment-1595272822

I tried a few simple examples with the latest prqlc.exe built from main (8f9b77802da49971b94ebeb4baee7ccc725675bf), and I suspect that this has nothing to do with count and everything to do with the number of filter and take and sort.

Working

from employees
filter gross_cost > 0
group {title} (
  aggregate {
    ct = count s"*",
  }
)
sort ct
filter ct > 200

from employees
filter gross_cost > 0
take 20
sort ct
filter ct > 200
sort ct
take 20

from employees
filter gross_cost > 0
take 20
sort ct
filter ct > 200
sort ct
take 20
filter ct > 200
sort ct
filter ct > 200
sort ct

Not working

from employees
filter gross_cost > 0
group {title} (
  aggregate {
    ct = count s"*",
  }
)
sort ct
filter ct > 200
take 20
sort ct
filter ct > 200
take 20
sort ct
filter ct > 200
take 20
sort ct
filter ct > 200
take 20

from employees
filter gross_cost > 0
sort ct
filter ct > 200
take 20
sort ct
filter ct > 200
take 20
sort ct
filter ct > 200
take 20
sort ct
filter ct > 200
take 20

from employees
filter gross_cost > 0
take 20
sort ct
filter ct > 200
sort ct
take 20
filter ct > 200
sort ct
filter ct > 200
sort ct
filter ct > 200

_Originally posted by @eitsupi in https://github.com/PRQL/prql/issues/2717#issuecomment-1596099365_

About this issue

Original URL
State: open
Created a year ago
Reactions: 1
Comments: 30 (30 by maintainers)

Commits related to this issue

tests: Demo of printing the stack A slicker version of https://github.com/PRQL/prql/issues/2857#issuecomment-1603190843 in case that's helpful. Run with: ``` RUST_BACKTRACE=1 RUST_LOG=trace cargo +... — committed to max-sixty/prql by max-sixty 5 months ago

Most upvoted comments

It depends on exactly when the stack is filled, so it will vary with every new variable in a function that requires stack memory.

aljazerzen on Jan 18, 2024

So stack memory usage of our compiler grows linearly with call depth of the PRQL program. Which is not ideal and but would be hard to change.

We can change how fast does it grow with call depth and debugging a little, it looks like we do have a bunch of variables on the stack that could be on the heap. Will optimize a little.

aljazerzen on Jan 17, 2024

I did some quick diagnostics. Using this diff from the current main (similar to #2908 but uses an unstable feature):

diff --git a/prql-compiler/prqlc/tests/project/Project.prql b/prql-compiler/prqlc/tests/project/Project.prql
index 1fc6a97f..1b44377f 100644
--- a/prql-compiler/prqlc/tests/project/Project.prql
+++ b/prql-compiler/prqlc/tests/project/Project.prql
@@ -6,3 +6,4 @@ let favorite_artists = [
 
 favorite_artists
 join artists.input side:left (==artist_id)
+join long_query.long_query side:left (input.title == long_query.title)
diff --git a/prql-compiler/prqlc/tests/project/long_query.prql b/prql-compiler/prqlc/tests/project/long_query.prql
new file mode 100644
index 00000000..9f64f091
--- /dev/null
+++ b/prql-compiler/prqlc/tests/project/long_query.prql
@@ -0,0 +1,25 @@
+# TODO: move this into a separate test; but putting here because of the
+# `insta_cmd` issues, and because we're in a rush to ensure Windows builds
+# aren't randomly breaking.
+
+let long_query = (
+    from employees
+    filter gross_cost > 0
+    group {title} (
+        aggregate {
+            ct = count s"*",
+        }
+    )
+    sort ct
+    filter ct > 200
+    take 20
+    sort ct
+    filter ct > 200
+    take 20
+    sort ct
+    filter ct > 200
+    take 20
+    sort ct
+    filter ct > 200
+    take 20
+)
diff --git a/prql-compiler/prqlc/tests/test.rs b/prql-compiler/prqlc/tests/test.rs
index d495b301..37c65cd1 100644
--- a/prql-compiler/prqlc/tests/test.rs
+++ b/prql-compiler/prqlc/tests/test.rs
@@ -78,77 +78,7 @@ fn test_compile_project() {
     cmd.arg(project_path());
     cmd.arg("-");
     cmd.arg("main");
-    assert_cmd_snapshot!(cmd, @r###"
-    success: true
-    exit_code: 0
-    ----- stdout -----
-    WITH table_1 AS (
-      SELECT
-        120 AS artist_id,
-        DATE '2023-05-18' AS last_listen
-      UNION
-      ALL
-      SELECT
-        7 AS artist_id,
-        DATE '2023-05-16' AS last_listen
-    ),
-    favorite_artists AS (
-      SELECT
-        artist_id,
-        last_listen
-      FROM
-        table_1
-    ),
-    table_0 AS (
-      SELECT
-        *
-      FROM
-        read_parquet('artists.parquet')
-    ),
-    input AS (
-      SELECT
-        *
-      FROM
-        table_0
-    )
-    SELECT
-      favorite_artists.artist_id,
-      favorite_artists.last_listen,
-      input.*
-    FROM
-      favorite_artists
-      LEFT JOIN input ON favorite_artists.artist_id = input.artist_id
-
-    ----- stderr -----
-    "###);
-
-    let mut cmd = prqlc_command();
-    cmd.args(["compile", "--hide-signature-comment"]);
-    cmd.arg(project_path());
-    cmd.arg("-");
-    cmd.arg("favorite_artists");
-    assert_cmd_snapshot!(cmd, @r###"
-    success: true
-    exit_code: 0
-    ----- stdout -----
-    WITH table_0 AS (
-      SELECT
-        120 AS artist_id,
-        DATE '2023-05-18' AS last_listen
-      UNION
-      ALL
-      SELECT
-        7 AS artist_id,
-        DATE '2023-05-16' AS last_listen
-    )
-    SELECT
-      artist_id,
-      last_listen
-    FROM
-      table_0
-
-    ----- stderr -----
-    "###);
+    assert_cmd_snapshot!(cmd);
 }
 
 #[test]
@@ -162,7 +92,7 @@ fn test_format() {
     ----- stdout -----
 
     ----- stderr -----
-    Currently `fmt` only works with a single source, but found multiple sources: "`Project.prql`, `artists.prql`"
+    Currently `fmt` only works with a single source, but found multiple sources: "`Project.prql`, `artists.prql`, `long_query.prql`"
     "###);
 }
 
@@ -189,8 +119,8 @@ fn prqlc_command() -> Command {
     // `--color=never` flag.
     cmd.env_remove("CLICOLOR_FORCE");
     // We don't want the tests to be affected by the user's `RUST_BACKTRACE` setting.
-    cmd.env_remove("RUST_BACKTRACE");
-    cmd.env_remove("RUST_LOG");
+    // cmd.env_remove("RUST_BACKTRACE");
+    // cmd.env_remove("RUST_LOG");
     cmd.args(["--color=never"]);
     cmd
 }
diff --git a/prql-compiler/src/lib.rs b/prql-compiler/src/lib.rs
index dacfd237..aeb270df 100644
--- a/prql-compiler/src/lib.rs
+++ b/prql-compiler/src/lib.rs
@@ -65,7 +65,7 @@
 //!     $ prqlc compile query.prql
 //!     ```
 //!
-
+#![feature(backtrace_frames)]
 #![forbid(unsafe_code)]
 // Our error type is 128 bytes, because it contains 5 strings & an Enum, which
 // is exactly the default warning level. Given we're not that performance
diff --git a/prql-compiler/src/semantic/module.rs b/prql-compiler/src/semantic/module.rs
index 97b08a17..b86c7693 100644
--- a/prql-compiler/src/semantic/module.rs
+++ b/prql-compiler/src/semantic/module.rs
@@ -1,4 +1,7 @@
-use std::collections::{HashMap, HashSet};
+use std::{
+    backtrace,
+    collections::{HashMap, HashSet},
+};
 
 use anyhow::{bail, Result};
 use itertools::Itertools;
@@ -201,6 +204,12 @@ fn lookup_in(module: &Module, ident: Ident) -> HashSet<Ident> {
         }
 
         log::trace!("lookup: {ident}");
+        let bt = backtrace::Backtrace::capture();
+        let len = bt.frames().len();
+        log::trace!("{}", len);
+        if len >= 120 {
+            log::trace!("{:?}", bt);
+        }
 
         let mut res = HashSet::new();

And then running with:

RUST_BACKTRACE=1 RUST_LOG=trace cargo +nightly insta test --accept -p prqlc  -- project

We can then look at how deep the stack traces are with:

 rg '\]\s(\d+)$' -o -r '$1' -N prql-compiler/prqlc/tests/snapshots/test__compile_project.snap | sort -n

I then set some bound at which to log the whole stack trace (currently 120 deep), and then looked at one:

[TRACE prql_compiler::semantic::module] Backtrace [{ fn: “std::backtrace_rs::backtrace::libunwind::trace”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/…/…/backtrace/src/backtrace/libunwind.rs”, line: 93 }, { fn: “std::backtrace_rs::backtrace::trace_unsynchronized”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/…/…/backtrace/src/backtrace/mod.rs”, line: 66 }, { fn: “std::backtrace::Backtrace::create”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/backtrace.rs”, line: 332 }, { fn: “prql_compiler::semantic::module::Module::lookup”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/module.rs”, line: 207 }, { fn: “prql_compiler::semantic::module::Module::lookup::lookup_in”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/module.rs”, line: 171 }, { fn: “prql_compiler::semantic::module::Module::lookup”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/module.rs”, line: 220 }, { fn: “prql_compiler::semantic::context::Context::resolve_ident_fallback”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/context.rs”, line: 214 }, { fn: “prql_compiler::semantic::context::Context::resolve_ident_fallback”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/context.rs”, line: 219 }, { fn: “prql_compiler::semantic::context::Context::resolve_ident”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/context.rs”, line: 195 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_ident”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 519 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 196 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_type_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 906 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_ty_or_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 920 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function_types::{{closure}}”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 687 }, { fn: “core::iter::adapters::map::map_try_fold::{{closure}}”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/adapters/map.rs”, line: 91 }, { fn: “core::iter::traits::iterator::Iterator::try_fold”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/traits/iterator.rs”, line: 2303 }, { fn: “<core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/adapters/map.rs”, line: 117 }, { fn: “<core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/adapters/mod.rs”, line: 195 }, { fn: “<I as alloc::vec::in_place_collect::SpecInPlaceCollect<T,I>>::collect_in_place”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/alloc/src/vec/in_place_collect.rs”, line: 258 }, { fn: “alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/alloc/src/vec/in_place_collect.rs”, line: 182 }, { fn: “<alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/alloc/src/vec/mod.rs”, line: 2711 }, { fn: “core::iter::traits::iterator::Iterator::collect”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/traits/iterator.rs”, line: 1895 }, { fn: “<core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/result.rs”, line: 1932 }, { fn: “core::iter::adapters::try_process”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/adapters/mod.rs”, line: 164 }, { fn: “<core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/result.rs”, line: 1932 }, { fn: “core::iter::traits::iterator::Iterator::collect”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/traits/iterator.rs”, line: 1895 }, { fn: “itertools::Itertools::try_collect”, file: “/Users/maximilian/.cargo/registry/src/index.crates.io-6f17d22bba15001f/itertools-0.10.5/src/lib.rs”, line: 2014 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function_types”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 682 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 229 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 267 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 306 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_type_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 906 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_ty_or_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 920 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function_types::{{closure}}”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 687 }, { fn: “core::iter::adapters::map::map_try_fold::{{closure}}”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/adapters/map.rs”, line: 91 }, { fn: “core::iter::traits::iterator::Iterator::try_fold”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/traits/iterator.rs”, line: 2303 }, { fn: “<core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/adapters/map.rs”, line: 117 }, { fn: “<core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/adapters/mod.rs”, line: 195 }, { fn: “<I as alloc::vec::in_place_collect::SpecInPlaceCollect<T,I>>::collect_in_place”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/alloc/src/vec/in_place_collect.rs”, line: 258 }, { fn: “alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/alloc/src/vec/in_place_collect.rs”, line: 182 }, { fn: “<alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/alloc/src/vec/mod.rs”, line: 2711 }, { fn: “core::iter::traits::iterator::Iterator::collect”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/traits/iterator.rs”, line: 1895 }, { fn: “<core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/result.rs”, line: 1932 }, { fn: “core::iter::adapters::try_process”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/adapters/mod.rs”, line: 164 }, { fn: “<core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/result.rs”, line: 1932 }, { fn: “core::iter::traits::iterator::Iterator::collect”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/iter/traits/iterator.rs”, line: 1895 }, { fn: “itertools::Itertools::try_collect”, file: “/Users/maximilian/.cargo/registry/src/index.crates.io-6f17d22bba15001f/itertools-0.10.5/src/lib.rs”, line: 2014 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function_types”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 682 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 229 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 267 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 799 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_within_namespace”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 847 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_and_type_check”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 818 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_function_args”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 760 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_function”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 584 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 274 }, { fn: “prql_compiler::semantic::resolver::Resolver::resolve_pipeline”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 510 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_expr”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 279 }, { fn: “<prql_compiler::semantic::resolver::Resolver as prql_compiler::ast::pl::fold::AstFold>::fold_var_def”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 168 }, { fn: “prql_compiler::semantic::resolver::Resolver::fold_statements”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/resolver.rs”, line: 78 }, { fn: “prql_compiler::semantic::resolve”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/mod.rs”, line: 62 }, { fn: “prql_compiler::semantic::resolve_and_lower”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/semantic/mod.rs”, line: 31 }, { fn: “prql_compiler::pl_to_rq_tree”, file: “/Users/maximilian/workspace/prql/prql-compiler/src/lib.rs”, line: 282 }, { fn: “prqlc::cli::Command::execute::{{closure}}”, file: “./src/cli.rs”, line: 350 }, { fn: “core::result::Result<T,E>::and_then”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/result.rs”, line: 1319 }, { fn: “prqlc::cli::Command::execute”, file: “./src/cli.rs”, line: 349 }, { fn: “prqlc::cli::Command::run_io_command”, file: “./src/cli.rs”, line: 238 }, { fn: “prqlc::cli::Command::run”, file: “./src/cli.rs”, line: 217 }, { fn: “prqlc::cli::main”, file: “./src/cli.rs”, line: 37 }, { fn: “prqlc::main”, file: “./src/main.rs”, line: 16 }, { fn: “core::ops::function::FnOnce::call_once”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/ops/function.rs”, line: 250 }, { fn: “std::sys_common::backtrace::__rust_begin_short_backtrace”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/sys_common/backtrace.rs”, line: 135 }, { fn: “std::rt::lang_start::{{closure}}”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/rt.rs”, line: 166 }, { fn: “core::ops::function::impls::<impl core::ops::function::FnOnce for &F>::call_once”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/core/src/ops/function.rs”, line: 284 }, { fn: “std::panicking::try::do_call”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/panicking.rs”, line: 500 }, { fn: “std::panicking::try”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/panicking.rs”, line: 464 }, { fn: “std::panic::catch_unwind”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/panic.rs”, line: 142 }, { fn: “std::rt::lang_start_internal::{{closure}}”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/rt.rs”, line: 148 }, { fn: “std::panicking::try::do_call”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/panicking.rs”, line: 500 }, { fn: “std::panicking::try”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/panicking.rs”, line: 464 }, { fn: “std::panic::catch_unwind”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/panic.rs”, line: 142 }, { fn: “std::rt::lang_start_internal”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/rt.rs”, line: 148 }, { fn: “std::rt::lang_start”, file: “/rustc/498553fc04f6a3fdc53412320f4e913bc53bc267/library/std/src/rt.rs”, line: 165 }, { fn: “_main” }]

It looks like the resolver goes very deep. Is that because it structures foo | bar | baz as baz(bar(foo)), such that a long query is a very deeply nested query?

…rather than working like an intepreter, evaluating r0=bar(foo) and then baz(r0)?

If so — that’s an elegant approach, but maybe it scales badly with program size?

I need to sign off for a bit, and probably won’t be doing much coding in the next couple of days. I can look at this more after that though!

max-sixty on Jun 22, 2023

Great find @eitsupi ! That seems likely. So I guess one of our structs is huge (and this isn’t fortran!).

So it’s about the size of objects on the stack, rather than having a very long call-chain. I’ll have a look at what this might be (unless @aljazerzen sees this and have a good idea what this is likely to be).

And then probably we can just put that thing in a Box — the performance hit would be trivial.

max-sixty on Jun 19, 2023

If you have the patience (no problem if you’re busy; I don’t have access to a Windows machine unfortunately), would you mind running with RUST_LOG=trace RUST_BACKTRACE=full env vars?

Sure.

❯ $env:RUST_LOG='trace'
❯ $env:RUST_BACKTRACE='full'
❯ ./prqlc.exe compile
Enter PRQL, then press ctrl-z to compile:

from employees
filter gross_cost > 0
take 20
sort ct
filter ct > 200
sort ct
take 20
filter ct > 200
sort ct
filter ct > 200
sort ct
filter ct > 200

log.txt

for example whether prqlc resolve works…

Also does not work.

eitsupi on Jun 18, 2023