ripgrep: windows doesn't detect stdin automatically

This is the golden ticket for posterity:

/// Returns true if there is an MSYS tty on the given handle.
#[cfg(windows)]
fn msys_tty_on_handle(handle: HANDLE) -> bool {
    use std::ffi::OsString;
    use std::mem;
    use std::os::raw::c_void;
    use std::os::windows::ffi::OsStringExt;
    use std::slice;

    use kernel32::{GetFileInformationByHandleEx};
    use winapi::fileapi::FILE_NAME_INFO;
    use winapi::minwinbase::FileNameInfo;
    use winapi::minwindef::MAX_PATH;

    unsafe {
        let size = mem::size_of::<FILE_NAME_INFO>();
        let mut name_info_bytes = vec![0u8; size + MAX_PATH];
        let res = GetFileInformationByHandleEx(
            handle,
            FileNameInfo,
            &mut *name_info_bytes as *mut _ as *mut c_void,
            name_info_bytes.len() as u32);
        if res == 0 {
            return true;
        }
        let name_info: FILE_NAME_INFO =
            *(name_info_bytes[0..size].as_ptr() as *const FILE_NAME_INFO);
        let name_bytes =
            &name_info_bytes[size..size + name_info.FileNameLength as usize];
        let name_u16 = slice::from_raw_parts(
            name_bytes.as_ptr() as *const u16, name_bytes.len() / 2);
        let name = OsString::from_wide(name_u16)
            .as_os_str().to_string_lossy().into_owned();
        name.contains("msys-") || name.contains("-pty")
    }
}

With that bit done, everything seems to work perfectly now in both cmd.exe and mintty.

Conceptually it seems simple to me: we just check if we have a pipe and if the name of the pipe fits the pattern used by MSYS for the pipes it uses for terminal emulation.

I think the confusing part could be all this _pioinfo stuff, but we wouldn’t need this, the original patch probably was done like this to minimize changes to the existing code, but we don’t have such constraints. So all we need to do is to call NtQueryObject() (which is almost like any other Win32 API, except it’s a wrapper for a kernel function and not a Win32 call) and match the name returned by it again "msys-XXXX-ptyN-XX".

BTW, notice that if we don’t need to support XP (do we?), then we could use Win32 GetFileInformationByHandleEx() function instead. But OTOH NtQueryObject(), despite being officially internal is surely not going anywhere any time this millennium as there is so much code using it…

P.S. Just got the page update with your latest reply. Unfortunately here it’s my lack of Rust knowledge that works against me. I hoped that these function could be used in the same way as GetConsoleMode(), for example. What’s the difference between them from your point of view?

P.P.S. I do understand being sick of it very well though.

@vadz I don’t really understand it and I need to understand it before figuring out how to write it in Rust.

(I am somewhat at the end of my rope on this issue too. For now, at least.)