runtime: Breaking change proposal: Encoding.UTF8 singleton should not have a BOM

I am a supporter of this proposal. We need to provide a config switch to go back to old behavior if needed.

@krwq The idea is that all UTF-8 factories hanging off Encoding will be no-BOM, unless the caller calls new UTF8Encoding(true).

It seems confusing if Encoding.UTF32 and Encoding.Unicode emit BOM, but Encoding.UTF8 does not.

UTF32 and UTF16 need a Byte Order Mark to indicate the endianness of the data apart from anything else; UTF8 doesn’t have any endianness so doesn’t require it for that purpose.

Also while ASCII text encoded using UTF-8 is backward compatible with ASCII, this is not true when Unicode Standard recommendations are ignored and a BOM is added.

IMO this is such a significant (and difficult to discover) breaking change that “provide a config switch to go back to old behavior” is not sufficient. I propose:

• obsolete Encoding.UTF8 with warning
• add Encoding.UTF8IncludingBom and Encoding.UTF8ExcludingBom (or equivalent)

We discussed a little bit internally the idea of having Encoding.UTF8NoBOM as a first-class citizen alongside Encoding.UTF8. That suggestion has come up a few times in this thread as well.

I’m not sold on that as a good long-term solution. The spirit of this work item is that we want to reduce the number of developers who are exposed to the concept of a BOM. By having static factories for “with BOM” and “without BOM”, we’d be foisting this concept upon every developer who starts typing Encoding.* in their code editor. A developer who is well-versed in these concepts can quickly and correctly answer the question of “do I want a BOM or not?”, but for the majority of the developer audience these terms would be unfamiliar and they wouldn’t know how to answer the question. Ultimately I think exposing these concepts on a primary API would result in a poorer user experience than exists today.

Good catch, @krwq: Encoding.GetEncoding("utf-8") does emit a BOM (though you wouldn’t be able to tell from the the description of the .Preamble property, which doesn’t mention .GetEncoding() and explicitly states that only using System.Text.Encoding.UTF8 and using the argument-less or no-BOM UTF8Encoding constructor results in a BOM).

By contrast, System.Text.Encoding.GetEncoding(0), documented to return UTF-8 in .NET Core, does not emit a BOM.

(A systematic review of the docs with respect to recommending / discouraging a UTF-8 BOM is called for either way, as certain pages contradict each other.)

I think consistency is called for, and my vote is to consistently default to BOM-less UTF-8 and only ever return a with-BOM instance if explicitly requested.

While undoubtedly a breaking change, @GrabYourPitchforks has already made compelling (to me) arguments for it in the initial post; let me add a few points:

• The Unix world has moved to BOM-less UTF-8 a long time ago, and it is primarily tools with a Unix heritage that do not expect a BOM, and in the presence of one either choke or misinterpret the BOM as part of the data.

• The Windows world is undoubtedly moving towards assuming UTF-8 in the absence of a BOM as well:

  • The major (cross-platform) text editors nowadays write and read BOM-less UTF-8 by default - see https://github.com/dotnet/runtime/issues/28218#issuecomment-795925634 for an overview.

  • PowerShell Core (the cross-platform edition built on .NET Core / 5+) too uses BOM-less UTF-8 as its consistent default, both when reading its source code and in its file-processing cmdlets.

  • Node.js (node.exe) - and others? - have chosen to “speak” (BOM-less) UTF-8 by default, irrespective of the active OEM code page as determined by the system locale (aka language for non-Unicode programs).

    • Python chose a different (also nonstandard) approach, defaulting to the active ANSI code page even when called from the console, rather than the OEM code page console applications are expected to use. However, it is is trivial to configure python to use (BOM-less) UTF-8 instead, via an environment variable (PYTHONUTF8) or, situationally, via a CLI parameter (-X utf8).

  • Windows 10 now offers a - still-in-beta as of this writing - feature to switch to (BOM-less) UTF-8 system-wide, by setting the system locale so that both the OEM and the ANSI code pages use code page 65001, i.e. UTF-8; see this Stack Overflow answer for details and a discussion of the ramifications.

    • With this configuration:
      • Even Windows PowerShell and Python, for instance, then default to BOM-less UTF-8 (since the ANSI code page is then effectively UTF-8).
      • So will all conventional console applications that use the OEM code page, with the caveat that legacy applications that aren’t equipped to handle the variable-length aspect of UTF-8 encoding malfunction.

  • Last but not least: .NET’s own default encoding for its System.IO APIs has - commendably - been BOM-less UTF-8 since v1.

I wonder if default encoding can be changed from ANSI to UTF8, as a breaking change. In practice, especially for East Asian users, ASNI codepages are much more annoying than BOM.

Ah, I see. GetBytes never returns a BOM, but a StreamWriter will first write the result of GetPreamble and then the result of GetBytes.

I am confused. The documentation says that it does include a BOM. Which method does that? When I try GetBytes, it doesn’t appear to return a BOM:

var b = System.Text.Encoding.UTF8.GetBytes("Hello world");

foreach (byte bb in b)
    Console.WriteLine(bb.ToString("X2"));

This outputs (on .NET 5):

48
65
6C
6C
6F
20
77
6F
72
6C
64