eza: feat: `eza` is substantially slower than `ls` in large directories

As discussed here, eza is significantly slower than ls when listing files in large directories. This issue persists with the latest version (0.18.9).

Here’s a benchmark for a directory with ~50k files:

Benchmark 1: ls -1 dir | wc -l
  Time (mean ± σ):     807.9 ms ± 219.8 ms    [User: 102.2 ms, System: 20.4 ms]
  Range (min … max):   597.6 ms … 1036.0 ms    3 runs

Benchmark 2: eza -1 dir | wc -l
  Time (mean ± σ):      3.541 s ±  0.422 s    [User: 0.203 s, System: 0.654 s]
  Range (min … max):    3.074 s …  3.895 s    3 runs

Summary
  ls -1 dir | wc -l ran
    4.38 ± 1.30 times faster than eza -1 UHGV | wc -l

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 4
  • Comments: 15 (10 by maintainers)

Most upvoted comments

I noticed a substantial slowdown as well.

Example (single sample):

strace -Ttc /usr/bin/eza -lah

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ------------------
 53.09    0.010221       10221         1           futex
 22.38    0.004308         179        24           clone3
  6.92    0.001332          37        36           mprotect
  6.86    0.001320          26        49           rt_sigprocmask
  4.32    0.000831          11        70           mmap
  1.45    0.000279           9        29           read
  0.90    0.000174           6        27         2 statx
  0.82    0.000158           7        21         1 openat
  0.58    0.000112          28         4           getcwd
  0.50    0.000097          24         4         4 readlink
  0.36    0.000070           3        20           close
  0.32    0.000062          31         2           munmap
  0.26    0.000051           7         7           rt_sigaction
  0.26    0.000050          25         2           getdents64
  0.21    0.000040           8         5           brk
  0.21    0.000040           6         6           ioctl
  0.11    0.000022          11         2           prlimit64
  0.10    0.000019           1        13           fstat
  0.09    0.000017           3         5           lseek
  0.07    0.000014           4         3           sigaltstack
  0.06    0.000012          12         1           poll
  0.06    0.000012           6         2           getrandom
  0.06    0.000011           5         2           sched_getaffinity
  0.00    0.000000           0        17           write
  0.00    0.000000           0         2           pread64
  0.00    0.000000           0         1         1 access
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0         1           set_tid_address
  0.00    0.000000           0         1           set_robust_list
  0.00    0.000000           0         1           rseq
------ ----------- ----------- --------- --------- ------------------
100.00    0.019252          53       360         8 total
strace -Ttc /usr/bin/ls -lah

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 18.65    0.000406          15        27           mmap
 15.20    0.000331          11        30        12 openat
  7.90    0.000172           6        28           close
  6.61    0.000144           7        19           write
  5.33    0.000116           5        20           fstat
  5.10    0.000111           6        16           read
  4.59    0.000100           5        18           statx
  4.41    0.000096           4        22           epoll_ctl
  4.36    0.000095           5        18           listxattr
  3.58    0.000078          11         7           mprotect
  3.45    0.000075          12         6           getdents64
  2.99    0.000065           4        16           epoll_pwait2
  2.25    0.000049          12         4           connect
  2.20    0.000048          16         3           munmap
  1.98    0.000043          10         4           newfstatat
  1.75    0.000038           9         4           socket
  1.47    0.000032           8         4           sendto
  1.47    0.000032           6         5         1 recvfrom
  1.15    0.000025           6         4           lseek
  1.01    0.000022           4         5           brk
  0.69    0.000015           3         4           rt_sigprocmask
  0.51    0.000011          11         1           ioctl
  0.51    0.000011           5         2           epoll_create1
  0.46    0.000010           3         3           futex
  0.41    0.000009           1         6         4 prctl
  0.41    0.000009           4         2           timerfd_create
  0.37    0.000008           4         2           timerfd_settime
  0.37    0.000008           2         3           getrandom
  0.32    0.000007           7         1           epoll_wait
  0.14    0.000003           3         1           prlimit64
  0.09    0.000002           2         1           getpid
  0.09    0.000002           2         1           arch_prctl
  0.09    0.000002           2         1           set_robust_list
  0.05    0.000001           1         1           set_tid_address
  0.05    0.000001           1         1           rseq
  0.00    0.000000           0         2           pread64
  0.00    0.000000           0         1         1 access
  0.00    0.000000           0         1           execve
------ ----------- ----------- --------- --------- ----------------
100.00    0.002177           7       294        18 total

It seems like the futex / clone3 syscalls take the longest.

Oh, one of the reasons there’s so much lock traffic through futex calls with -l is that eza depends on uzers to get UID -> name conversion but uzers uses a cache based on a RefCell so it’s incompatible with RwLock. So the formatting code thrashes the Mutex that eza’s concurrency approach requires.

Not a bad idea, even as just a nice feature to replace eza -l | head, which removes all styling. However, in terms of performance, it’s just a workaround, because the difference between eza and the other utils show that eza could do better. The write calls should be easy to improve. The statx are necessary to determine whether a file is executable, so it’s more of a design decision for the maintainers. It can still be improved easily in cases where the color is turned off though. If there’s demand for that I can pick up my PR again.