zsh-bench: Prompt times don't match the real times in a screen recording, and zsh4humans seem to be getting quite a good advantage
It’s hard to be sure how exactly zsh-bench is measuring what it claims to be measuring, by looking at its code.
I’m going to trust a screen recording as the source of truth for the prompt times in my machine. I’ve recorded zim and zsh4humans, 3 times each. Each time the steps are:
- Sleep 1 second. In this time I type
exitand ENTER, so it’s used as input in the next steps. - Print the framework name. I consider the counting starts when the output of this print appears in the recording.
- Start a new shell with
HOMEas the specific framework installation dir. I consider the counting ends when the prompt appears.
This translates to: for f in zim zsh4humans; do repeat 3 do; sleep 1; print ${f}; HOME=${PWD:h}/${f} zsh -li; done; done
The steps above are executed in a dir with a git repo with 1,000 directories and 10,000 files, set up using the code here and here.
Each framework was installed with the setup script here and here, respectively for zim and zsh4humans.
This should be enough to guarantee that the recordings are using the same scenario used by zsh-bench.
This is the screen recording:

This is the times extracted from checking the recording frames:
| begin_frame | prompt_frame | frames | ms | |
|---|---|---|---|---|
| zim 1 | 161 | 169 | 9 | 300.000 |
| zim 2 | 200 | 207 | 8 | 266.667 |
| zim 3 | 238 | 246 | 9 | 300.000 |
| zsh4humans 1 | 276 | 282 | 7 | 233.333 |
| zsh4humans 2 | 316 | 321 | 6 | 200.000 |
| zsh4humans 3 | 354 | 359 | 6 | 200.000 |
The recording has 30fps, so each frame represents 33.333ms. We can consider the error of each measurement above to be +/- 33.333ms.
This is what I get when running zsh-bench on my same machine. Ran 3 times each of the mentioned frameworks:
❯ ./zsh-bench zim
==> setting up a container for benchmarking ...
==> benchmarking zim ...
creates_tty=0
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=115.790
first_command_lag_ms=149.910
command_lag_ms=62.535
input_lag_ms=28.830
exit_time_ms=47.530
❯ ./zsh-bench zim
==> setting up a container for benchmarking ...
==> benchmarking zim ...
creates_tty=0
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=196.126
first_command_lag_ms=229.765
command_lag_ms=142.110
input_lag_ms=29.402
exit_time_ms=48.146
❯ ./zsh-bench zim
==> setting up a container for benchmarking ...
==> benchmarking zim ...
creates_tty=0
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=196.878
first_command_lag_ms=231.291
command_lag_ms=140.756
input_lag_ms=27.556
exit_time_ms=48.560
❯ ./zsh-bench zsh4humans
==> setting up a container for benchmarking ...
==> benchmarking zsh4humans ...
creates_tty=1
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=30.361
first_command_lag_ms=100.323
command_lag_ms=4.997
input_lag_ms=14.536
exit_time_ms=10.895
❯ ./zsh-bench zsh4humans
==> setting up a container for benchmarking ...
==> benchmarking zsh4humans ...
creates_tty=1
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=30.458
first_command_lag_ms=101.612
command_lag_ms=5.198
input_lag_ms=12.339
exit_time_ms=11.051
❯ ./zsh-bench zsh4humans
==> setting up a container for benchmarking ...
==> benchmarking zsh4humans ...
creates_tty=1
has_compsys=1
has_syntax_highlighting=1
has_autosuggestions=1
has_git_prompt=1
first_prompt_lag_ms=29.819
first_command_lag_ms=108.122
command_lag_ms=5.123
input_lag_ms=13.146
exit_time_ms=11.05
What is odd
zim first_prompt_lag_ms times in zsh-bench fluctuate quite a lot. It’s an average of 169.598ms with stdev of 46.601! In the recordings the average was 288.889ms with a stdev of 19.245, which was actually a stdev of 0.577 in terms of frames (it’s just one frame more in one recordings). zsh-bench should be more precise than a 30fps recording.
Also from zsh-bench’s code looks like this output is the min value. It would be good to know what is the stdev of values inside a zsh-bench run, to make sure the values are not fluctuating.
I could compare the minimum recording time with the minimum zsh-bench time for each framework, but let’s use the averages since the stdev of zsh-bench for zim was so high. So for completeness of information, the first_prompt_lag_ms average for zsh4humans is 30.213ms, with a stdev of 0.344. And in recordings the average was 244.444ms (stdev was 50.918, but again it was actually just one frame more in one of the recordings).
Then, it’s odd that zim is 1.703x faster in zsh-bench than the recordings – I was expecting much closer values --, and zsh4humans is 8.091x faster in zsh-bench than the recordings!
If anything, zsh-bench if favoring zsh4humans by a lot, both in terms of more stable measurements, than giving it times way faster than the real ones.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 48 (33 by maintainers)
Commits related to this issue
- mention that the benchmark data was compiled on a machine with NVMe M.2 (#5) — committed to romkatv/zsh-bench by romkatv 5 years ago
Sorry, just catching up on all this and discussion in the other benchmarking repo.
@ericbn My memory is hazy and I don’t remember if there is any interaction with login shells when using
zpty, which my original script used, and if there is/was it may not be documented.Awesome that you were able to generate so many stats. Reading all of this was really the best kind of issue report 😃
@romkatv this is a really cool tool, although I’ll admit I haven’t looked deeply into your zsh4humans project. If the claim to fame is that it is fast, awesome! Fixing the horrendously slow prompt lag on slower devices (first generation raspberry pis, old old intel chips, etc) that came with other frameworks was one of my goals when I started zim.
Part of the original design of zim was also to allow the “heavier” tasks to run at the login shell, with the idea that it usually only happens once per interactive session. That is to say, when you sit down and login or connect, you’re only going to do it once per session/terminal. This included things like
zcompileoperations, initialization of the ssh/gpg agents, etc. My perception was, “if the user is forced to wait, make them wait one time and only one time”.In that same line referenced above, you can see that an explicit and seemingly superfluous
zshprocess is launched with theZDOTDIRof the to-be-benchmarked framework rc’s. This explicitly allowed first-time initialization of each framework (explicitly not benchmarking thefirst prompt lag). This isn’t to say that it doesn’t matter, it was just an intentional compromise.Design obviously wasn’t perfect, but was a good compromise vs trying to make things asynchronous until the cows come home. By happenstance, working on my original zsh framework benchmarking tool is where I learned precisely how painful that is 😉
My only question, do you run login shells frequently? If you open a new terminal is it always a login shell? I ask only because it doesn’t match my personal experience.
I’d be interested to see these benchmarks run on a variety of devices. There is a huge difference in speed based on IO devices. This is why the original benchmarks for zim ran on a variety of devices. Machines with an NVMe will perform better with a Celeron chip than a machine with an HDD and a Ryzen Threadripper in some circumstances. (Zim was knowingly designed with file IO in mind, hence the
zcompileof user RC files that you disagree with). You reference your CPU here, but not the disk: https://github.com/romkatv/zsh-bench#full-benchmark-datacheers! 🍻
To summarize what happened since this issue was opened:
Thanks to @romkatv’s input and to his thoughts in zsh-bench, this is what we’ve improved in Zim:
exit). We’re using a different tool (expect) than zsh-bench (script) to measure that time, and @romkatv helped us make sure the times between our different approaches match.compinit -Calone is also considered “cutting corners” in zsh-bench.ZSH_AUTOSUGGEST_MANUAL_REBIND=1and source the zsh-users/zsh-autosuggestions module last.2023-02-26 EDIT: We were still insisting in delaying the completion dumpfile check, but we’re finally doing the same improved dumpfile check that zsh4humans uses during startup.
@ericbn Both exceptions have disappeared with your fix: https://github.com/zimfw/zsh-framework-benchmark/pull/10
Now all zsh-framework-benchmark results match first_prompt_lag from
zsh-bench --standalone --git no.zsh-users/zsh-history-substring-search is enabled by default in zsh4humans.
Opened https://github.com/zimfw/zimfw/issues/441.
Out of curiosity I’ve benchmarked zim and zsh4humans on a Raspberry Pi 4 4G Model B with a micro SD card for storage.
Edit: See more accurate results below.
I’d rather have both of these automated and then we can also manually perform the same steps to double-check that the automation produces the same results.
(Frankly, I don’t think recording a video is a wise approach. Recording the raw writes to the TTY is both easier and more precise. But if you want to go with video, that’s fine with me. It’s more work but the numbers should match eventually after we iron our potential quirks with vsync, adaptive frame rate, codecs, buffering, etc.)
You can also run
zsh-benchwithout positional arguments to benchmark your own local login shell. You’ll have to installscriptfrom util-linux though and I don’t know how easy or hard that is on macOS. If you manage to instal it, create a local user, set up any kind of zsh environment you like and runzsh-benchto see how that shell performs on macOS. You can also use your regular user if you have a way to restore your rc files (e.g., from git).The config called “zim” is for the default experience. It measures the performance of zim that a potential user would see if they installed zim and made no modifications. Similarly with “zsh4humans”. The config called
zim+disables everything – including global rcs – and enables only the core functionality (what’s considered core is in the docs of zsh-bench) with all possible optimizations. It shows the baseline latency for a config that one could build on top of zim. If zsh-bench ran all configs withno_gobal_rcs, it wouldn’t be able to benchmark some configs that people actually use. E.g., it wouldn’t be able to tell us what kind of performance people should expect if they installed zim or zsh4humans and used it without modifications.It looks like the first run was slow and the second two runs had identical first prompt lag. Perhaps something was running on your machine at the same time? Try using
--iters 64, close all applications and don’t do anything while the benchmark is running. As I mentioned, this is the value I used for all benchmarks for all published results.Yes, that’s how zsh-bench measures first command lag. But you aren’t measuring first command lag, you are measuring first prompt lag. If you want to measure first command lag, you need to buffer
print first""-""commandand then track when “first-command” appears on the screen. Note that first command lag depends on the length of the command if syntax highlighting is enabled. Buffering “exit” doesn’t allow you to measure first command lag and doesn’t help with measuring first prompt lag.I forgot to mention that you can replay the screen of the TTY that zsh-bench is acting on. Let’s say you are in
playground zim. Run this:Edit: I’ve added the crucial
--keepto the command above.Once this command finishes, you can see what zsh-bench was doing:
This works only if your
TERMis compatible with xterm-256color. You can multiply and/or increase by a constant all numbers in the first column of/tmp/zsh-bench-*/timingfor slower replay.This method is very useful when debugging zsh configs. It allows you to see errors if there are any and to generally verify that things work as they are supposed to.
You’ll see that at first zsh-bench issues a short command that sources something. That’s the first command and it gets sent to the TTY right away without waiting for the first prompt. The sourced file prints, and the result of that printing signifies execution of the first command.
Then zsh-bench starts measuring input latency. It prints a fairly long command composed of a bunch of escaped
abctokens (they are escaped to avoid triggering alias expansion, which can significantly slow down syntax highlighting), types one more character and waits for that character to be processed by zle. This is performed several times and then the command line is cleared without execution.Then zsh-bench measures command lag by spamming ENTER and seeing how many prompts it would get.
Once you play around with
scriptandzsh-bencha bit and verify some numbers with video recording, it shouldn’t be hard to see how zsh-bench measures things and that the way it measures it corresponds to what we actually want to measure. You can even runclear; print start; script -fqet$HOME/timing ~/typescriptin the container and record a video. This will allow you to verify that latencies extracted from the logs ofscriptmatch latencies that you get from video.In order to replicate your results I need the full list of commands that I can run on a freshly installed OS.
If a benchmark doesn’t measure real life performance, it’s a problem with the benchmark. You’ve added
sleep 1to zim benchmark for the same reason. Starting zim in a tight loop has terrible performance but it doesn’t matter because in real life no one starts login shell in a tight loop. You don’t need to “fix” zim in this regard – it’s not broken. So fixing the benchmark was the right call. We want to measure real life performance, so we adapt the benchmark until it starts measuring real life performance. We don’t change real life to adapt to a benchmark.The closer you get to real life usage, the less likely that you are measuring wrong things. In real life people don’t override
HOME, so it’s better to not do that. If you can demonstrate that overridingHOMEnever affects benchmark results, then you can overrideHOMEbut it makes the benchmark more brittle. If my benchmark doesn’t overrideHOMEand yours does, and our results disagree, my results are more authoritative because my benchmark better models real life usage. Let’s stay as close as possible to real life and let’s specify benchmark conditions as fully and precisely as possible. It’s much easier to argue about code when we both can run it than it is to argue about words. Easier to argue implies easier to sort out differences and come to an understanding of where the discrepancies in the measurements lie.For the sake of efficiency, let’s stay on the subject matter and avoid speculations about each other’s motivation, character, attitude, etc.
I’m willing to replicate your results if you give me instructions as specific as the ones I gave you. It should be literally “start with a freshly installed OS and run this series of commands”. All commands must have their source available. I would greatly appreciate if the OS is also free and open source – Linux, FreeBSD, etc. If I get different results when I run the same commands as you, we’ll narrow it down to things we’ve failed to control (machine, cosmic rays, etc.).
Another way to do this is to start with the specific commands that we already have – the ones I posted in my previous comment. I can run them and you can run them. You don’t have to accept the numbers printed by zsh-bench as authoritative. Go ahead and record TTY on video, or via
script, or viaasciinema. If you find a mismatch in numbers by using a different methodology, post the instructions for me to replicate your measurements (ideally packaged as a script). It would save a lot of time if you start with the container I’ve created but if you want to create your own, that’s also fine. Just post the code so that I can have the same container on my machine.Let’s postpone this discussion. It’s significantly less important but it can derail us.
This is obviously true. I’m probably missing a point you are trying to make here.
To my knowledge, it does not.
The procedure you’ve used for benchmarking is underspecified, so I cannot replicate your results. It also doesn’t model real-life use of zsh. Starting login shell and immediately closing it is quite unusual, and so is overriding
HOME. It’s cool that you want to validate zsh-bench results, and I’d love to see that happening, so let’s figure out a way to do it right.It’s vital to perform all measurements in a container to allow everyone to create the same environment. I’ve created a script that spins up a docker container and sets up zsh in it. You can use it like so:
Instead of
zimyou can passzsh4humansor any other name of a predefined config.After a few minutes you should have interactive zsh with
~/repoas the current working directory. This is a git repository with 1k directories and 10k files.You can install additional software with
sudo apt install <pkg>.If you want to record a video of starting a login zsh, use this command:
Once “start” appears on the screen, start the timer. Once “repo” (the current directory) appears, stop it. That’s first prompt lag.
If you decide to base your benchmark results on a video recording, please specify the exact command that everyone can use to record video the same way. Any non-reproducible step in your methodology will void the results, so be precise.
Instead of video you can record raw output of
zshto the TTY complete with timing of all writes. This is what zsh-bench does. Here’s how you can start a login shell and capture its writes to the TTY:Run a few commands and exit zsh with
exitas usual. Afterwards you can look at the raw output to the TTY:And the timing data:
The format of the timing data is documented in
man script:You can replay a recorded session like this:
Don’t resize your TTY or it won’t replay properly.
I would very much prefer if you share
~/typescriptand~/timinginstead of a video. They are much easier to work with.You can also try using
asciinemainstead ofscriptto record and replay a session. I don’t know what kind of latency it adds on its own but in terms of capabilities it should work just as well asscript. The advantage ofasciinemais its output format – a single json file with data and timing. In addition,asciinemasessions can be easily uploaded and watched in a browser.You can run
zsh-benchin the container like this:All benchmark data that I’ve published was acquired with
--iters 64but it takes a long time to run. With--iters 16results are also decent.You can edit rc files if you need to test something (install
nanoorvimfor this). Restart zsh withexec zshafterwards.~/zsh-bench/zsh-benchwill be affected by these changes, too.If you find anything unexpected, or if your measurements match that of
zsh-bench, please share either way.Correct.
If you send a clean PR, I’ll merge it.
If you take a minimum of several runs, these fluctuations won’t matter.
Minimum is better in benchmarks of this kind, especially when ran on commodity hardware with CPU scaling, background services and whatnot. Minimum fails only when workloads are different on different iterations but in these benchmarks they are the same.