ttyplot: [1.6.1] Chart output corrupts after a few minutes on macOS?

@quinncomendant: Thanks for bisecting this! It is really useful.

Commit 61c2cf5ccc4a951d5b6bf4f9caf7ee5f75c64d3e is only about getting the -I compiler flags right in order for the program to build. The issue is thus almost certainly triggered by c73270ccfb74cf2dddfe9526f68645630b43f9f5. This is a quite disruptive commit: we move from the historical ASCII-only ncurses API to to newer (tough probably still ancient) “wide character” API.

I did a small test to capture the output of ttyplot:

script -c 'seq 1 10 | ./ttyplot'

and compared c73270c with its parent. The most striking difference is that the output was initially ASCII-only, with lines made out of DEC Special Graphics, which got replaced by UTF-8-encoded Box-drawing characters. For example, the sequence

ESC ( 0 ESC [ 0 m x ESC ( B

where the x encodes a vertical line, became just │ (U+2502 Box drawings light vertical, encoded in UTF-8 as 0xe2 0x94 0x82).

However, since the vertical lines are initially rendered correctly, this difference in output does not explain the problem.

Throughout my testing, $TERM has always been xterm-256color everywhere.

You may run

infocmp xterm-256color

on both your Mac and the Ubuntu server, and compare the outputs.

@hartwork wrote:

I’m surprised that ttyplot links to /usr/lib/libncurses.5.4.dylib without any “w” in it on your system.

We have seen that the ASCII-only and the wide-character-capable APIs are declared in the same header file… or in different header files, depending on the OS. There seems to be quite a bit of variability in how ncurses is deployed. Since the wide-character-capable API is a superset of the older one, I am not that much surprised that a system could provide only one dynamic library with the most complete API.

@quinncomendant:

If there any way I can get a copy of the source code before these pull requests and merge them myself?

The Web UI way:

• browse https://github.com/tenox7/ttyplot/tree/162fefe1ba7ca439128e7d4aadb07033e3f9a12a
• click on the green “Code” button near the top right of the page
• click on “Download ZIP”

The git way:

git clone https://github.com/tenox7/ttyplot.git
cd ttyplot
git checkout c150198^  # the “^” suffix means “parent of”

Note that git can be very helpful for bisecting the history and get the exact commit that broke something:

git bisect start
git bisect good 1.5.2
git bisect bad 1.6.1

At this point git checks out an intermediate revision and asks you to test it. You reply by typing either git bisect good or git bisect bad. Repeat. After O(log(n)) steps, git tells you the ID of the commit that caused the breakage.

It’s not environmental differences that cause the issue.

That’s not certain. Ttyplot relies on ncurses, and ncurses uses the TERM environment variable and the terminfo database to know what escape sequences it has to give to the terminal to get a specific result. Terminal.app uses a different terminfo entry from iTerm. When you ssh to Ubuntu, ttyplot uses the terminfo database installed on the Ubuntu server, instead of the one on your Mac. If the iTerm entries in these databases differ, that could lead to different behaviors.

@edgar-bonet Thanks for the tip to use git bisect, that’s super cool! The process of bisecting and testing went smoothly until a commit that failed to compile. I was unable to git bisect skip because it was the last commit that needed testing. Here’s the output at the end of the git bisect process:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
c73270ccfb74cf2dddfe9526f68645630b43f9f5
61c2cf5ccc4a951d5b6bf4f9caf7ee5f75c64d3e
We cannot bisect more!

I manually checkout’ed a series of commits in this range and tested them one-by-one, with the following results (in order from newest commits to older):

• 61c2cf5 compiles but runs with corrupt output 🐞
• c73270c fails to compile (screenshot with errors) 💣
• 07ce2f9 compiles and runs successfully ✅

This is confirms what git bisect already said, that the bug was introduced in c73270c or 61c2cf5.

It’s not environmental differences that cause the issue.

That’s not certain. Ttyplot relies on ncurses, and ncurses uses the TERM environment variable […]

Throughout my testing, $TERM has always been xterm-256color everywhere. I still don’t understand how my environment could be different between two ttyplot versions, but maybe I missed your point.

@hartwork As far as I can tell there is only one ncurses installed: the one included with the macOS developer tools. According to otool -L, all the versions I’ve tested use the same ncurses library, including the one installed by Homebrew (which just uses make during its install process).

❯ ./ttyplot -v
ttyplot 1.5.2

❯ otool -L ./ttyplot
./ttyplot:
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)

❯ /opt/homebrew/bin/ttyplot -v
ttyplot 1.6.1

❯ otool -L /opt/homebrew/bin/ttyplot
/opt/homebrew/bin/ttyplot:
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.61.1)

I tested this again on an older MacBook which I recently reinstalled the OS, so it has a virgin terminal environment (never used, never customized). I installed iTerm and ttyplot for the first time. The issue occurs there!

Also, I discovered it only occurs with the beta version of iTerm (3.5.0beta18), and not the stable release (3.4.23).

The beta version of iTerm is far ahead of the stable release, so I’ve been using it everywhere. I think the iTerm team even recommends it. Searching the changelog of the beta releases I can see many dozens of fixes related to drawing text and screen management, so it’s no surprise that there is a difference.

@hartwork, I checked those two pull requests and briefly scanned the commits, but I’m not familiar enough with ttyplot’s design to know which might be more the cause. If I had to guess on a hunch, it seems like c150198 has the most changes related to screen redraw, so we could start there? If there any way I can get a copy of the source code before these pull requests and merge them myself? (I’m not very familiar with GitHub.)

@edgar-bonet, I can do the random ideas you suggested, but I think it may not be useful since I already discovered ttyplot 1.6.1 does work fine locally on macOS when using the default Terminal.app or the stable version of iTerm, it only occurs with the beta version of iTerm. It’s not environmental differences that cause the issue.

I did more experiments. The issue only occurs on macOS, when using iTerm version 3.5.0beta18, running ttyplot v1.6.1.

• The problem occurs on a clean install of macOS + iTerm 3.5.0beta18 + ttyplot 1.6.1.
• No problem when using the stable version of iTerm (3.4.23).
• No problem when using the standard macOS Terminal.app.
• No problem when running ttyplot 1.6.1 on an Ubuntu server connected from macOS via SSH using iTerm.
• No problem when using ttyplot 1.5.2 with iTerm.
• I’ve ruled out my choice of font (Iosevka) causing the issue; the problem continues even when using the default Menlo.

I know terminal emulation can be difficult to troubleshoot because of all the variables on multiple layers. I would be happy to just blame iTerm because it’s a huge ball of wax. However, the fact that ttyplot v1.5.2 works, and v1.6.1 does not, in identical environments, is evidence that something changed that made ttyplot less resilient to working well in obscure environments, namely, mine.

Here’s a screen recording showing v1.5.2 and v1.6.1 running side-by-side.

Hi @quinncomendant, and thanks for the report. I couldn’t reproduce the issue either. I am using:

• ttyplot 1.6.1 built from sources
• Gnome Terminal @ 90×20
• Ubuntu 22.04

Hi @hartwork! Thanks for the quick reply with suggestions. Yeah, if your internet is stable, ping may not reveal the issue.

In order to reproduce this, you need to generate values that:

• are in a quantity great enough to saturate the x axis (so the screen is full horizontally),
• and contain a range of values such that the scale of the chart will need to redraw occasionally.

Here’s a test case you can use to reproduce the issue:

1. Set your terminal window to about 90 columns by 20 rows.
2. Run this:

for i in 13.669 13.669 13.669 17.235 17.235 17.235 17.235 17.235 17.235 17.235 10.831 10.831 10.831 10.831 10.831 10.831 93.902 93.902 93.902 93.902 93.902 93.902 93.902 93.902 2.984 2.984 2.984 2.984 2.984 2.984 2.984 6.528 6.528 6.528 6.528 6.528 6.528 6.528 6.528 5.143 5.143 5.143 5.143 5.143 5.143 5.143 5.143 8.741 8.741 8.741 8.741 8.741 8.741 8.741 8.741 2.310 2.310 2.310 2.310 2.310 2.310 2.310 10.943 10.943 10.943 10.943 10.943 10.943 10.943 10.943 18.117 18.117 18.117 18.117 18.117 18.117 18.117 18.117 11.670 11.670 11.670 11.670 11.670 11.670 11.670 11.670 5.296 5.296 5.296 5.296 5.296 5.296 5.296 5.296 8.840 8.840 8.840 8.840 8.840 8.840 8.840 8.840 7.446 7.446 7.446 7.446 7.446 7.446 7.446 7.446 7.446 6.027 6.027 6.027 6.027 6.027 6.027 6.027 6.027 4.598 4.598 4.598 4.598 4.598 4.598 4.598 4.598 4.598 8.156 8.156 8.156 8.156 8.156 8.156 8.156 8.156 8.156 6.724 6.724 6.724 6.724 6.724 6.724 6.724 6.724 5.336 5.336 5.336 5.336 5.336 5.336 5.336 5.336 5.336 8.920 8.920 8.920 8.920 8.920 8.920 8.920 8.920 2.485 2.485 2.485 2.485 2.485 2.485 2.485 2.485 2.485 6.056 6.056 6.056 6.056 6.056 6.056 6.056 6.056 3.284 3.284 3.284 3.284 3.284 3.284 3.284 3.284 6.840 6.840 6.840 6.840 6.840 6.840 6.840 6.840 6.840 5.382 5.382 5.382 5.382 5.382 5.382 5.382 5.382 3.986 3.986 3.986 3.986 3.986 3.986 3.986 3.986 3.986 2.608 2.608 2.608 2.608 2.608 2.608 2.608 2.608 1.134 1.134 1.134 1.134 1.134 1.134 1.134 1.134 1.134 4.777 4.777 4.777 4.777 4.777 4.777 4.777 4.777 3.337 3.337 3.337 3.337 3.337 3.337 3.337 3.337 3.337 6.924 6.924 6.924 6.924 6.924 6.924 6.924 6.924 50.511 50.511 50.511 50.511 50.511 50.511 50.511 50.511 50.511 4.095 4.095 4.095 4.095 4.095 4.095 4.095 4.095 2.677 2.677 2.677 2.677 2.677 2.677 2.677 2.677 2.677 1.278 1.278 1.278 1.278 1.278 1.278 1.278 1.278 3.443 3.443 3.443 3.443 3.443 3.443 3.443 3.443 3.443 7.014 7.014 7.014 7.014 7.014 7.014 7.014 7.014 5.616 5.616 5.616 5.616 5.616 5.616 5.616 5.616 5.616 7.731 7.731 7.731 7.731 7.731 7.731 7.731 7.731 7.731 6.371 6.371 6.371 6.371 6.371 6.371 6.371 6.371 4.914 4.914 4.914 4.914 4.914 4.914 4.914 4.914 4.914 3.492 3.492 3.492 3.492 3.492 3.492 3.492 3.492 2.086 2.086 2.086 2.086 2.086 2.086 2.086 2.086 2.086 0.660 0.660 0.660 0.660 0.660 0.660 0.660 0.660 4.234 4.234 4.234 4.234 4.234 4.234 4.234 4.234 4.234 2.853 2.853 2.853 2.853 2.853 2.853 2.853 2.853 2.853 1.382 1.382 1.382 1.382 1.382 1.382 1.382 1.382 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 0.047 8.607 8.607 8.607 8.607 8.607 8.607 8.607 8.607 2.206 2.206 2.206 2.206 2.206 2.206 2.206 2.206 2.206 5.747 5.747 5.747 5.747 5.747 5.747 5.747 5.747 7.980 7.980 7.980 7.980 7.980 7.980 7.980 7.980 6.509 6.509 6.509 6.509 6.509 6.509 6.509 6.509 6.509 5.088 5.088 5.088 5.088 5.088 5.088 5.088 5.088 5.088 3.682 3.682 3.682 3.682 3.682 3.682 3.682 3.682 2.264 2.264 2.264 2.264 2.264 2.264 2.264 2.264 2.264 5.868 5.868 5.868 5.868 5.868 5.868 5.868 5.868 4.424 4.424 4.424 4.424 4.424 4.424 4.424 4.424 4.424 7.988 7.988 7.988 7.988 7.988 7.988 7.988 7.988 6.607 6.607 6.607 6.607 6.607 6.607 6.607 6.607 6.607 5.181 5.181 5.181 5.181 5.181 5.181 5.181 5.181 8.746 8.746 8.746 8.746 8.746 8.746 8.746 8.746 8.746; do echo $i; sleep 0.1; done | ttyplot

This prints a value to the screen every 1/10 seconds. After about 12 seconds, the screen will look like this: