Terminal.Gui: Most Glyphs not showing on macOS - E.g. Menu Glyphs and Character Map

Also, for good measure, we should change setlocale to return an IntPtr, and not an int, in case we ever find a system that uses different calling conventions - it is just good hygiene.

Hello,

@mklement0 found the culprit, the issue is that the LC_ALL definition is wrong for Mac. It should be 0, according to locale.h:

#define LC_ALL          0
#define LC_COLLATE      1
#define LC_CTYPE        2
#define LC_MONETARY     3
#define LC_NUMERIC      4
#define LC_TIME         5
#define LC_MESSAGES     6

I do not know what the value is on Linux, but if it is not zero, then we will need to check the OS and use 0 on Mac, and 6 o Linux.

Hello,

The posters are right, this is a problem with ncurses, not really an issue with the terminal.

What might be happening is that ncurses is being initialized implicitly by some code before setlocale is invoked.

Perhaps an additional dependency, or some invocation to the library is taking place that loads and initializes ncurses before we call setlocale. One quick way of testing this theory would be to call setlocale manually in Main():

                        setlocale(LC_ALL, "");

Before anything else - also, not sure if we are now using global constructors (the ones that run on assembly load), that could also be initializing curses before we have a chance to.

I had some misconceptions:

• It is indeed only the terminal emulator’s character-encoding setting that matters for display, for all programs.

• Typically, terminal emulators are configured to reflect the locale + character encoding in the LANG environment variable (which can be overridden via either category-specific individual LC_* environment variables or for all categories with LC_ALL), e.g. en_US.UTF-8.

  • Locale-aware programs do respect these environment variables, so you can do the following to print the current date in French, for instance (using Bash syntax):

    • LC_ALL=fr_FR date -> French-style date

  • And while programs even respect the character-encoding part of the value, this won’t work for display in the terminal, because the terminal isn’t aware of the requested ad hoc character-encoding change; e.g., on macOS the following correctly translates single byte 0xe8, representing lowercase è in ISO-8859-1 encoding, to single byte 0xc8, its uppercase È equivalent (on Linux, utility tr isn’t locale-aware); however, unless the terminal too is set to ISO-8859-1, the result will print as the replacement character:

    • printf '\xe8' | LC_ALL=fr_FR.ISO8859-1 tr '[:lower:]' '[:upper:]' | od -t x1 -> c8; without | od -t x1, in a UTF-8 terminal window, you’d see � or ?, depending on the terminal.

---

Returning to the ncurses library:

On macOS, there’s only one binary, /usr/lib/libncurses.dylib, which contains both the narrow and wide-character functions.

The version that ships with macOS 10.15.7 is quite old (ncurses5.4-config --version): 5.7.20081102 vs. 6.2.20200212, the current version.

From man ncurses (emphasis added):

       The  library uses the locale which the calling program has initialized.
       That is normally done with setlocale:

	     setlocale(LC_ALL, "");

       If the locale is not initialized, **the library assumes  that  characters
       are  printable  as in ISO-8859-1,** to work with certain legacy programs.

The mystery is that Terminal.Gui does call setlocale(LC_ALL, ""), yet ncurses on macOS seemingly always uses its built-in default, ISO-8859-1.

One thing worth looking into: Is the setlocale(LC_ALL, "") call truly effective on macOS?

@ mklement0 thank you very much for your commitment. Your research is of enormous value. I hope you can come up with the solution to this because I’m already burning the fuses 😃

Thanks, @BDisp, but the logic of LC_ALL is not a matter of bit patterns - as man setlocale on macOS states:

LC_ALL Set the entire locale generically.

That is, the LC_* categories are not flags to be bit-ORed - they are distinct values, with LC_ALL by convention overriding all others.

Re macOS: The setlocale method is misdefined as returning int, according to the man page it is char *, so if you define it as IntPtr and convert it to a string with Marshal.PtrToStringAuto(), it shows that the locale is seemingly correctly picked up from the environment.

As an aside: Window.initscr() is reentered via main_window = new Window (methods.initscr ()); - is that expected?

The strange thing is that I eventually did get the glyphs to render correctly, namely by setting LC_CTYPE (define it as 0) instead of LC_ALL, to target the character-encoding category explicitly - even though LC_ALL should set all categories.

Similarly strangely, querying the effective LC_ALL category (by passing null instead of "") yields the correct value even when only LC_CTYPE was set. Do the other locale categories matter and should they be set explicitly as well?

What terminals are you using here?

WSL Ubuntu itself on Windows and the Ubuntu Terminal on the virtual machine with Ubuntu installed. It seems to me that I am coming to some conclusion. Rune.ColumnWidth has a return that is not always what is expected and I will try to submit a PR on NStack to correct how to interpret what returns ustring.ConsoleWidth. There are Unicode characters that return a width equal to 0 and this induces a strange behavior in the printing of the characters because the calculation of the print spacing is poorly calculated. On the Ubuntu system that is installed on the virtual machine they print well because they interpret the characters well but in WSL and Windows Terminal it causes the disproportionate printing of the characters and their misalignment.

See https://github.com/migueldeicaza/gui.cs/issues/41

I’ve already been down this path…

ncurses isn’t sending UTF-8 to the terminal (it is sending ISO-8859-1, always), so we need to figure out why.

By the way, given that Terminal.Gui can only ever work as expected with UTF-8, the right thing to do would be to refuse startup if the terminal window’s character encoding is found to be something other than UTF-8.

I confirm that it works.

iTerm2-Menlo:

iTerm2-Monaco:

Terminal-Menlo:

Terminal-monaco:

What I posted is a PowerShell command, so it only makes sense to run it from inside PowerShell (iex is the built-in alias for the Invoke-Expression cmdlet; the command also uses the gc alias for the Get-Content cmdlet).

How did you try to install PowerShell? The package downloadable from https://github.com/PowerShell/PowerShell#get-powershell should work.

If you have Homebrew installed, you can also try:

brew tap caskroom/cask; brew cask install powershell

Thanks. Yes the setlocale (LC_ALL, "") is called. It may not be influential on the macOS.

Hey @mklement0 - you seem like you know what you are doing on a Mac. Can you help us with this?

I don’t have time to dive into this today or tomorrow. But it seems there’s some silly configuration related bug in ConsoleDriver/CursesDriver when running on Mac based on this.