rexml: REXML don't support UTF8 BOM on Windows 11
Hi!
After update Puppet aggent to 8.9.0 (they updated REXML from 3.2.5 to 3.3.6 version) - puppet chocolatey provider can’t use correct parsing XML witn UTF-8 BOM codding. Only if enable support beta UTF8 WordWide in Windows 10/11 - everything work fine. If disable this feature - puppet send error again. (tried on cyrrilic and english - same issues)
Debug: Gathering sources from 'C:\ProgramData\chocolatey\config\chocolatey.config'.
Error: Could not prefetch chocolateysource provider 'windows': Malformed XML: Content at the start of the document (got '')
Line: 1
Position: 47
Last 80 unconsumed characters:
<?xml version="1.0" encoding="utf-8"?>
Warning: /Stage[main]/Windows::Chocolatey/Chocolateysource[chocolatey]: Skipping because provider prefetch failed
Debug: /Stage[main]/Windows::Chocolatey/Chocolateysource[chocolatey]: Resource is being skipped, unscheduling all events
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 37 (16 by maintainers)
Really? It seems that
chocolatey.configis a valid XML but Puppet reads it with invalid encoding conversion.Would it be possible to update the documentation? I don’t see any mention of encodings or byte order marks.
I know what you’re saying, but this test actually does that. The resulting document children are parsed correctly: https://github.com/ruby/rexml/blob/b97e454ceb2e1719a487bfebaae3da4a706a854b/test/test_document.rb#L331
Also wanted to mention that “just specify the encoding” when calling
File.readisn’t always practical on Windows. For example, if the file was created using PowerShell’sOut-File, then it will beUTF-16LEencoded: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_character_encoding?view=powershell-7.5 That said it’s the application’s problem (in this case puppet), not rexml’s problem. I think we’ll just callFIle.openand let rexml autodetect the encoding.@naitoh thank you for the detailed explanation, I appreciate it!
thanks a lot! I sent them pull request with changing File.read on File.open, but still wait for approving it… thanks again!!!
''can be created by this step.So the data passed to REXML is already corrupt by re-encode with multiple encoding several times. I think this is not a bug of REXML.
Since the xml has
encoding="utf-8", reading it in ascii, binary or ascii-8bit works fine, but reading with other wrong encoding corrupts the data. Corrupt data can’t be repaired by re-encoding or any other process.If your
Encoding.default_externalis not ascii, ascii-8bit or it does not match the file encoding, you need to specify the encoding when reading the file.