runtime: Console UTF-8 input is misbehaving on Windows

Description

using System;

namespace Test
{
    static class Program
    {
        static void Main()
        {
            Console.WriteLine(Console.InputEncoding);
            Console.WriteLine(Console.OutputEncoding);
            Console.WriteLine();
            Console.Write("Text   : ");
            Console.WriteLine("Result : {0}", Console.ReadLine());
        }
    }
}

Example run:

$ dotnet run
System.Text.UTF8Encoding+UTF8EncodingSealed
System.Text.UTF8Encoding+UTF8EncodingSealed

Text   : abcæøådef
Result : abc   def

The non-ASCII characters are basically being replaced with NULs for some reason.

This happens in all terminals I’ve tried (CMD, PowerShell, Windows Terminal, mintty). I checked chcp in all conhost-based terminals, and it reported 65001 (UTF-8) everywhere. I’ve also enabled global UTF-8 in Windows region settings just for good measure (enabling/disabling it appears to make no difference).

What is fascinating here is that this only seems to happen in .NET Core processes. No other programs in any of the terminals I’ve tried have issues processing non-ASCII characters. For example, things like this work in all of them:

$ echo abcæøådef
abcæøådef
$ cat
abcæøådef
abcæøådef

What is even more fascinating is that if you P/Invoke ReadFile to read from standard input in the .NET Core program instead of using System.Console, you get the same issue: The read is successful but non-ASCII characters are just replaced with NULs.

So the question is: Why are .NET Core processes special? What does .NET Core do that seemingly makes ReadFile misbehave?

Configuration

$ dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.100-rc.1.20452.10
 Commit:    473d1b592e

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19042
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.100-rc.1.20452.10\

Host (useful for support):
  Version: 5.0.0-rc.1.20451.14
  Commit:  38017c3935

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 23 (21 by maintainers)

Commits related to this issue

Most upvoted comments

@krwq I recommend we get the fix into 6.0.0. After that, if we receive enough reports of users being blocked by this bug, we’d consider down-level servicing. We would need validation from users who have encountered this that the behavior is indeed fixed with the 6.0.0 builds.