terminal: RTL text in conhost is no longer rendered correctly

@j4james Damn that was almost too easy - took like 5 minutes: https://github.com/microsoft/terminal/compare/dev/lhecker/12294-bidi-override

diff --git a/src/renderer/gdi/paint.cpp b/src/renderer/gdi/paint.cpp
index 598ce3489..bc4c4e95a 100644
--- a/src/renderer/gdi/paint.cpp
+++ b/src/renderer/gdi/paint.cpp
@@ -445,9 +445,21 @@ using namespace Microsoft::Console::Render;
         for (size_t i = 0; i != _cPolyText; ++i)
         {
             const auto& t = _pPolyText[i];
-            if (!ExtTextOutW(_hdcMemoryContext, t.x, t.y, t.uiFlags, &t.rcl, t.lpstr, t.n, t.pdx))
+
+            SCRIPT_STATE ss{};
+            ss.fOverrideDirection = TRUE;
+
+            SCRIPT_STRING_ANALYSIS ssa;
+            hr = ScriptStringAnalyse(_hdcMemoryContext, t.lpstr, t.n, 0, -1, SSA_GLYPHS | SSA_FALLBACK | SSA_LINK, 0, nullptr, &ss, t.pdx, nullptr, nullptr, &ssa);
+            if (FAILED(hr))
+            {
+                break;
+            }
+
+            hr = ScriptStringOut(ssa, t.x, t.y, t.uiFlags, &t.rcl, 0, 0, FALSE);
+            ScriptStringFree(&ssa);
+            if (FAILED(hr))
             {
-                hr = E_FAIL;
                 break;
             }
         }

I think typing this message took longer than writing that code. 😄

ScriptStringAnalyse is a handy function that calls ScriptItemize, ScriptShape, ScriptPlace, and ScriptBreak for you. Due to the lack of batching this approach is a lot slower than ExtTextOut though. My plan is to call those 4 functions myself (well 3, because we don’t really need ScriptBreak) and call ScriptIsComplex. If it’s false I can just straight up call TextExtOut to ensure the expected performance in the general case.

If I can’t make it for whatever reason though, I think this is what we could ship, since it works.

As far as I can discern, no one ever actually concerned themselves with Arabic nor Hebrew support in the console host. The targeted languages were basically LTR European type character sets + the CJK trio. Beyond that… it looks like anything else that worked or didn’t was a happy accident.

Furthermore, when our localization team tells us what languages we can pay for in terms of translations for developer utilities today… they limit it to: German, English, Spanish, French, Italian, Japanese, Korean, Brazilian Portuguese, Russian, Simplified Chinese, and Traditional Chinese. I’d therefore have to believe to some degree that research was performed to determine that was the appropriate balance between resources and developer market was to focus on those languages.

Therefore, my consideration here is happiness of those languages as primary goal with anything else being secondary.

Further, one of the most popular issues filed against conhost.exe is the lack of font fallback for Chinese, Japanese, and Korean languages. Switching to ExtTextOut (Option 1) to restore font fallback, therefore, dramatically reduced our inbound bug flow and solved an issue for four of the targeted languages. An issue I’ve never seen filed in Feedback Hub, directly from our OEM customers, our business partners, or otherwise in the last 7-8 years of working on this is anything about Hebrew or Arabic. I know that’s super scientific… to rely on my past experience.

But with the combination of those reasons, I would have to personally opt for Option 1.

I would offer to @lhecker, if he’s interested, that next week is our organization’s “Fix Hack Learn” week again. If he wants to spend a few days hacking Option 3 using Uniscribe to solve this problem and learn more about language processing… he would be free to do so. I think it would be better, though, long-term to focus efforts on supporting those languages fully in the Terminal and the Atlas renderer.

The discussion can continue, I’m not shutting it down. This is just my opinion on the situation.

so that we don’t break applications which rely on our broken behavior

That’s kinda my point - do you already have dozens of reports like “things are broken, the world is falling apart, do something now”? Support for anything non-ASCII in Windows Console has always been like “it depends” - on the OS version, current font, system locale, console codepage, output method, the phase of the Moon etc. Personally I haven’t seen any applications even trying to cover non-trivial cases, but YMMV of course.

Yeah it does! While Uniscribe doesn’t seem to support “liga” in fonts, it does correctly handle ligatures in most languages. You gotta say, this is pretty nice to see in good old conhost, right?

Yeah, I saw the ligatures were working. I meant the other things you were refering to when you said “ZWJ, etc. and all the other fun Unicode stuff”.

And while it looks nice at first glance, it’s not particularly useful as is as far I’m concerned. You’ve got no hope of editing the text - all it’s really good for is displaying a single line of content at best.

I’d be quite sad if we lost that, but I’d understand that it’d be for a good cause.

Yeah, ideally we’d have a solution that was realistically usable and also looked pretty, but I don’t know how feasible that is for languages with ligatures. I thought with something like Arabic, an application might be able to output the appropriate form of each character manually, which might make up for the loss of ligature support, but I don’t know enough about the subject to know if that’s nonsense.

Given that most terminals apart from xterm seem to not draw glyphs in their logical order either

I wouldn’t have said “most”, but I haven’t checked recently. And for those that do draw the glyphs in RTL order, there’s not a standard of any sort that they’re following - they all do things differently. Thankfully some of them at least have a way to turn that functionality off.

if we’d actively not support TUI applications implementing their own BiDi

You realise that just means we’re saying we don’t support TUI applications fullstop (at least for BiDi languages). If that’s the route we want to take, that’s fine - I seem to be in the minority in wanting support for RTL TUI apps. I’d just like to know for definite where we’re going with this, so I can make my own plans accordingly.

BTW for my own curiosity: Do you happen have a specific application at hand that positions Hebrew text manually?

Well there is the command line utility from fribidi, which can be used as a kind bidi-aware version of cat (amongst other things). And there’s also a Hebrew mode in vim (i.e. vim -H). The other applications I have are unfortunately not open source.

@j4james I believe this doesn’t work with font fallback. I think if you try to draw Japanese text for instance, it’ll show just blank / whitespace glyphs.

Yeah, you’re right. I’ve just tested and that’s not working for me either. Oh well.

I would be somewhat surprised if we had supported RTL reordering back then… But if it used to work, then I wonder what the actually correct path forward is.

Yeah, that’s weird. It’s definitely not reordering RTL characters for me in the legacy console.

But I guess applications rely on this now?

That would be because it’s almost impossible to write an RTL application on terminals that don’t work this way. Give it a try. See if you can write some basic RTL applications on one of those terminals that reorders RTL characters. Like a simple RTL form entry system, or something that pops up a dialog or drop-down menu over existing RTL text. Maybe I’m just an idiot, but I can’t see how you can make that work, but it’s fairly straightforward on terminals that leave RTL characters exactly where you put them.

Did you paste them exactly as \u05ea\u05d7\u05d0 into the XP console?

Yes, this as is: תחא

Moreover:

Pulled triage. Sorry @j4james, I’ve been snowed in on e-mail as I had to leave to take care of some family stuff. Thanks for the first pass, Mike. d