datafusion: Build time regression
Is your feature request related to a problem or challenge?
We observed a huge increase after upgrading datafusion https://github.com/GreptimeTeam/greptimedb/pull/5417. I run a script to test the build time change day by day since v38.0, and here is the result:
The build time keeps increasing and is almost doubled since v38.0. Given the codebase keeps adding new code, it’s expected to see a trend of increasing, but there are still some obvious “platforms” and “jumps”, which might be abnormal.
This is the raw data:
Describe the solution you’d like
No response
Describe alternatives you’ve considered
No response
Additional context
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 17 (17 by maintainers)
I have another hunch, but still need to proof it: DataFusion uses a lot of
This is mostly so that the caller can stick a lot of parameters into methods w/o needing to convert it to the right type. Sometimes it’s also used for backwards compatible-ish changes[^1].
In Rust that
implconstructs expands roughly toSince Rust generics are implemented using monomorphization, this leads to the following issues:
T. And this also includes multiple copies of the LLVM IR and resulting machine code. Note that this includes the WHOLE body, not just thelet param: Type = param.into();part[^2].Tthat you eventually gonna use, the compilation is done on the call site (instead the definition site). This however also means that you get even more copies of the function, because if you have a crate that defines the method and two crates that use it, the two user crates are compiled independently.My guess is that while this leads to “nice APIs”, it leads to a lot of compilation overhead both within DataFusion and for its users. I would therefore suggest that we change methods to look like this:
An intermediate solution that requires more code, has a slightly higher compilation overhead (due to more generics) but isn’t as bad as the original version would be this:
Technically this also applies to
impl IntoIterator<Item = Type>andimpl Iterator<Item = Type>, but there it’s usually harder to fix. There we would have the following options, all of which are slower than the original one due to dynamic dispatch:So I would leave the iterator cases out for now.
[^1]: That’s strictly speaking NOT true, changing
param: Stringtoparam: impl Into<String>is NOT backwards compatible since it affects type inference on the call site. [^2]: This partial replication of code might one day be possible via polymorphization, but I don’t expect this to happen within the next 3 years.Thanks for those updates!
I was trying to replace type parameters (
FforFn) insideTreeNodewithBoxthis holiday. Let’s see if it works. (it should reduce some monomorphization overhead, theoreticallyI can make Expr less than half the size in this PR:
I ran out of time today to run build benchmarks on it, but I will try to do it tomorrow if no one else beats me to it
After removing the
WildcardOptions(by replacing it with an empty structure) I can see the build time drops. Removing the rule itself and the change incoredoesn’t help. It looks like the change toExpris the root cause, though I don’t know the reason…One guess is that both
ExprandLogicalPlanare key enums used in many places, especially many type parameters (like theTreeNode). Thus a slight change to them will be amplified by a lot.I have some thoughts on this specific case, like putting the expansion somewhere else, or changing some Fn type parameters to trait objects. But I can’t tell how good it will be, and the balance between runtime overhead and build time consumption. Or we can wait for those refactors until we figure out why
WildcardOptionscauses that.Looking forward to the ongoing refactors!
Here is the script, I removed the target after every build:
script
A few more details:
The build command I use is
cargo build --release --timings --lib --quietThe latest timing file is (rename it to HTML, restriction from GitHub): Cargo Timing Jan 23 2025.txt
And screenshot:
The
datafusioncrate (core) spends time for single thread compiling, which would be a major reason for slowing down the build time. I can observe this bottleneck in the downstream greptimedb project as well.