rexml: REXML don't support UTF8 BOM on Windows 11

Hi!

After update Puppet aggent to 8.9.0 (they updated REXML from 3.2.5 to 3.3.6 version) - puppet chocolatey provider can’t use correct parsing XML witn UTF-8 BOM codding. Only if enable support beta UTF8 WordWide in Windows 10/11 - everything work fine. If disable this feature - puppet send error again. (tried on cyrrilic and english - same issues)

Debug: Gathering sources from 'C:\ProgramData\chocolatey\config\chocolatey.config'.
Error: Could not prefetch chocolateysource provider 'windows': Malformed XML: Content at the start of the document (got '')
Line: 1
Position: 47
Last 80 unconsumed characters:
<?xml version="1.0" encoding="utf-8"?>
Warning: /Stage[main]/Windows::Chocolatey/Chocolateysource[chocolatey]: Skipping because provider prefetch failed
Debug: /Stage[main]/Windows::Chocolatey/Chocolateysource[chocolatey]: Resource is being skipped, unscheduling all events

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 37 (16 by maintainers)

Most upvoted comments

Is there something wrong with the way that the chocolatey.config file is encoded?

Yes.

Really? It seems that chocolatey.config is a valid XML but Puppet reads it with invalid encoding conversion.

Use File.open(‘chocolatey.txt’) or File.read(‘chocolatey.txt’, encoding: Encoding::UTF_8) when calling REXML, please.

Would it be possible to update the documentation? I don’t see any mention of encodings or byte order marks.

A corrupted BOM is considered invalid XML because it is an invalid string.

I know what you’re saying, but this test actually does that. The resulting document children are parsed correctly: https://github.com/ruby/rexml/blob/b97e454ceb2e1719a487bfebaae3da4a706a854b/test/test_document.rb#L331

Also wanted to mention that “just specify the encoding” when calling File.read isn’t always practical on Windows. For example, if the file was created using PowerShell’s Out-File, then it will be UTF-16LE encoded: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_character_encoding?view=powershell-7.5 That said it’s the application’s problem (in this case puppet), not rexml’s problem. I think we’ll just call FIle.open and let rexml autodetect the encoding.

@naitoh thank you for the detailed explanation, I appreciate it!

thanks a lot! I sent them pull request with changing File.read on File.open, but still wait for approving it… thanks again!!!

Malformed XML: Content at the start of the document (got ‘Γê⌐ΓòùΓöÉ’)

'' can be created by this step.

encoding = [Encoding::IBM437, Encoding::IBM865].sample
bom = "\xef\xbb\xbf".b
bom.force_encoding(encoding).encode('utf-8').force_encoding(encoding).encode('utf-8')
#=> ""

So the data passed to REXML is already corrupt by re-encode with multiple encoding several times. I think this is not a bug of REXML.

Since the xml has encoding="utf-8", reading it in ascii, binary or ascii-8bit works fine, but reading with other wrong encoding corrupts the data. Corrupt data can’t be repaired by re-encoding or any other process.

If your Encoding.default_external is not ascii, ascii-8bit or it does not match the file encoding, you need to specify the encoding when reading the file.