rubyinstaller2: unexpected ucrtbase.dll

Hello this is Armin from Microsoft. We encountered the same or at least a very similar issue in our internal compatibility testing. What seems to be the issue is that Ruby is relying on undocumented/unsupported behaviour. It is trying to retrieve a pointer to an internal variable __pioinfo inside urtbase.dll. That itself is not supported, but from our debugging and looking at the code in win32.c it looks like it is doing it by walking back from the ret assembly instruction in _isatty(). But this assumes the ret instruction will be the last instruction in that _isatty().

The issue is that certain optimizations can move function code beyond a ret instruction, so the common/typical instruction path is shorter. And new optimizations we applied to recent ucrtbase.dll builds did exactly that in _isatty(). So it looks like Ruby is now looking at the wrong location and no longer finding __pioinfo.

Short term we’ll be rolling out a fix that for this function that undoes this optimization, which should fix this specific error for the many Ruby users in the wild. But it would be great if Ruby can be fixed to not use undocumented behaviour, or perhaps at least be more robust in its function traversal. Note though that at some point we may break it in some way that is not so easily fixed.

The offending code is in win32\win32.c function set_pioinfo_extra.

Feel free to correct me though if our understanding is incorrect.

thanks @ArminG-MSFT 25250 is now working for me on Windows on Arm Dev Kit 2023 device - thanks!

Update: Build 25252 has just been released. If not offered automatically, actively search for updates will cause it to show.

Update: The issue for ARM64 is of course that Ruby runs emulated, and hence scanning the actual assembly that is executed is never going to yield the expected results. (That is on top of the generic issue that ARM64 does not have a lea-instruction and hence typically will load addresses using a runtime register and the concept itself is tricky.) But when emulated and using GetProcAddress, one will get the ARM64EC forwarder instead of the actual #_isatty inside ucrtbase.dll. Hence currently Ruby is scanning that forwarder when running emulated. That is great, as that is AMD64 assembly. I managed to insert the dummy sequence there, which unblocks the startup check when Ruby runs on Windows on ARM.

I’ll update again when this gets flighted.

Build 25236 has just been released. So all blocked can actively search for updates and will get it offered now.

Update: The fix will be in build 25235.rs_prerelease or higher. I don’t know yet when that will be out publicly.

I implemented __pioinfo pointer detection for (native) windows-arm64 in https://github.com/ruby/ruby/pull/8995. You’re welcome to join and review it.

AMD64 and AMD64 on ARM64 have no longer issues. 32bit on PC does still have issues depending on build (as the issue occurs on and off depending on code gen), as the ucrtbase.dll with fix is not yet flighted yet. Fixed builds are 258xxx builds, but these are not yet released. I’ll report here when it does.

Probably this is useful. I created a patch to win32.c which is based on win32.c taken from Ruby 3.2.x but really it may be applied and was tested with Ruby 2.6.x up to 3.2.x. It fixes ‘unexpected ucrtbase.dll’ issue for WIn32 builds and lets run 32-bit Ruby on Win11 as well as older Windows.

Before making the patch, I have disassembled ucrtbase.dll from no-issues 32-bit windows, 64-bit, and problematic Win11 32-bit and 64-bit. Checked __pioinfo lookup with all the four by looking into asm and finding patterns. The only difference was using LEAVE in the newer 32-bit ucrtbase.dll vs POP EBP in the previous. The main MOV/LEA pattern was not changed and may be used as is. There are no changes in 64-bit asm that would prevent current patterns from working. I did not need 64-bit, but checked asm.

As a result this patch only adds the additional check for LEAVE instruction for 32-bit versions only. Win11 ucrtbase.dll _isatty is longer than 300 bytes, hence changed to 500 which helped.

--- win32.c.orig	2023-02-08 07:02:20.000000000 +0100
+++ win32.c	2023-06-24 00:58:44.000000000 +0100
@@ -2596,8 +2596,8 @@
      * * https://bugs.ruby-lang.org/issues/11118
      * * https://bugs.ruby-lang.org/issues/18605
      */
-    char *p = (char*)get_proc_address(UCRTBASE, "_isatty", NULL);
-    char *pend = p;
+    unsigned char *p = (char*)get_proc_address(UCRTBASE, "_isatty", NULL);
+    unsigned char *pend = p;
     /* _osfile(fh) & FDEV */
 
 # ifdef _WIN64
@@ -2622,10 +2622,16 @@
 #  define PIOINFO_MARK "\x8B\x04\x85"
 # endif
     if (p) {
-        for (pend += 10; pend < p + 300; pend++) {
+        for (pend += 10; pend < p + 500; pend++) {
             // find end of function
+#if _WIN64
             if (memcmp(pend, FUNCTION_BEFORE_RET_MARK, sizeof(FUNCTION_BEFORE_RET_MARK) - 1) == 0 &&
-                (*(pend + (sizeof(FUNCTION_BEFORE_RET_MARK) - 1) + FUNCTION_SKIP_BYTES) & FUNCTION_RET) == FUNCTION_RET) {
+                (*(pend + (sizeof(FUNCTION_BEFORE_RET_MARK) - 1) + FUNCTION_SKIP_BYTES) & FUNCTION_RET) == FUNCTION_RET) 
+#else
+      	    if ((*pend == 0x5d /*pop ebp*/ || *pend == 0xc9 /*leave*/) &&
+		*(pend+1) == 0xc3 /*ret*/)
+#endif	    
+	    {
                 // search backwards from end of function
                 for (pend -= (sizeof(PIOINFO_MARK) - 1); pend > p; pend--) {
                     if (memcmp(pend, PIOINFO_MARK, sizeof(PIOINFO_MARK) - 1) == 0) {

@nurse Thank you for sharing these details! I think it’s best to move the discussion to bugs.ruby-lang.org where I posted my answer. Maybe the title could be somewhat more generic there.

@ArminG-MSFT Thank you for your investigation and your short term solutions! IMHO this particular issue with Windows 11 Insider Preview is solved that way.

Hi, I’m a person who introduced the __pioinfo hack, and notice this thread just now. Why Ruby depends to _pioinfo is

• to associate socket and fd: CRuby creates fd with dummy file handle and set socket to emulate Unix-like behavior
• to implement overlapped I/O for Windows 2000/XP
• to emulate fcntl(2)

I’m very sorry that Visual C++'s optimization is blocked by my hack. But unfortunately our decision to merge the change is not made as a tiny hack. After first I came up with the idea, we investigated other solution for a year. And also I believe many people including you struggled to find a smarter way, but no one proposed a solution.

I agree that Ruby should support ARM64 natively. I’m very welcome to contribute a patch to get __pioinfo on ARM64.

Of course I’m also welcome to propose a better solution. Thanks,

Update: The patch for AMD64 emulation on ARM64 will be in 25250.1000.rs_prerelease or higher. I don’t know the flight schedule yet.

Running Ruby on Windows for ARM64 will indeed run emulated, but that has separated generated assembly code which does not contain my fix as it will load the ARM64 native ucrtbase.dll and look for the EC-export (#_isatty vs _isatty). We really need to get Ruby to use ARM64 natively, but let me see what I can do here too.

@ArminG-MSFT - I just tried rubyinstaller-devkit-3.1.2-1-x64.exe on Window on Arm (Windows DevKit 2023) with Windows Pro 11 Insider Preview (dev channel) on 25236.rs_prerelease.221028-1618 and still get unexpected ucrtbase.dll trying to run ruby. Shouldn’t it run emulated nontheless?

I can confirm same versions (rubyinstaller&windows insider dev) on x64 SurfaceBook works now

@ArminG-MSFT Thank you. Added your comments to https://bugs.ruby-lang.org/issues/18605#note-4