fd: Benchmark is unfair. find(1) should not be used for grepping.
Hi,
I believe the benchmarks you provide compared to find(1) are unfair. find(1) should not be used for grepping; following the Unix principles, find(1) should just find, and grep(1) should be responsible for filtering the output of find(1).
If we pipe find(1) to grep(1), the performance is significantly faster than just using find(1):
$ time find ~ -iregex '.*[0-9]\.c$' >/dev/null
real 0m1.103s
user 0m0.881s
sys 0m0.220s
$ time find ~ | grep -i '/[^/]*[0-9]\.c$' >/dev/null
real 0m0.346s
user 0m0.103s
sys 0m0.260s
find | grep seems to be faster than fdfind in my own simple test:
$ time fdfind -HI '.*[0-9]\.c$' ~ >/dev/null
real 0m0.383s
user 0m1.920s
sys 0m6.126s
Can you please provide benchmarks against this pipeline in your readme?
What version of fd are you using?
[paste the output of fd --version here]
$ dpkg -l | grep fd-find
ii fd-find 8.7.0-3+b1 amd64 Simple, fast and user-friendly alternative to find
$ fdfind --version
fdfind 8.7.0
Just for completeness, here’s my CPU:
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Vendor ID: GenuineIntel
Model name: 13th Gen Intel(R) Core(TM) i9-13900T
CPU family: 6
Model: 183
Thread(s) per core: 1
Core(s) per socket: 24
...
Caches (sum of all):
L1d: 896 KiB (24 instances)
L1i: 1.3 MiB (24 instances)
L2: 32 MiB (12 instances)
L3: 36 MiB (1 instance)
Thanks!
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 23 (3 by maintainers)
On the other hand, it might be interesting to fork find(1) into a new program f(1), which would get rid of all the crap from find(1). Just find stuff. It looks like a fun project for me.
@jgardona
That’s a small thing, actually. Compare to this:
😃
There are two important cases where this isn’t true:
Thanks for doing those. Here’s my attempt to replicate them, with
hyperfine. The working directory is a checkout of the Chromium source tree (2,119,292 files):True, but for me
fd -j1is even better thanfind | grep.That’s not surprising to me. With
fdyou get multi-threaded filtering, but withfd | grepyou get at most one core ofgrep.What does surprise me is that
fd -j1 | grepis slower thanfind | grep. Not sure why that is, I’ll look into it.Don’t forget about
fd -j1! But to be fair, there’s no reason afindimplementation couldn’t be just as fast.find -iregexis pretty slow becausegrepuses a much faster regex implementation. Theoretically, GNU find could use the same implementation and become more useful. (Thoughfind | grepdoes allow for more parallelism, assuming single-threadedfind.)Personally I use
fdmainly interactively, and I’m happy to burn CPU cores for lower latency. In other contexts, like for example a background task on a loaded machine, efficiency is more important.But
fd -j1is pretty efficient too! It has the lowest user+system CPU consumption of anything in the benchmark. The reasonfdappears CPU-inefficient is that crossbeam (used to communicate between threads) tends to prefer spinning over blocking. Spinning burns CPU time but has much lower latency than blocking.Sounds good.👍 Let’s open a patch to GNU Coreutils to delete that functionality from GNU
find. Once that lands we can update these benchmarks.Because its inventor was told to do it that way, it seems. Here’s the (funny) history of the design of find(1): https://doc.cat-v.org/unix/find-history
@jgardona
findcompleting in 2.9 ms means that’s probably a very small directory tree. I’d imagine that most of the time taken byfdis startup overhead, maybe due to #1203