commitizen: `UnicodeDecodeError` if commit messages contain Unicode characters

@gpongelli this issue isn’t really a bug and it’s not about emojis.

The problem in this issue is about a sequence of bytes which contains UTF-8 encoded text, but the bytes’ encoding is mispredicted as Windows 1254 encoding. Based on that misprediction commitizen picks the incorrect codec to decode/interpret the bytes and that fails.

Thus, my proposed solution is to try to decode the bytes using the UTF-8 codec first because that’s the common text encoding across platforms these days. Only if that fails, invoke some statistical analysis (e.g. chardet) to predict the text encoding (see also chardet FAQ).

Python encodes text as UTF-8 by default, but it also provides a large number of other text codecs you should consider when testing. I think, though, that UTF-8 is the common default encoding these days on many platforms.

@KyleKing actually…

I’m not sure if the change would work, but it might be good to add a test case for STDOUT like: bytes([0x73, 0xe2, 0x80, 0x9d]).

That byte sequence encodes two characters:

>>> bytes([0x73, 0xe2, 0x80, 0x9d]).decode()
's”'

where the bytes e2 80 9d are the UTF-8 encoding of ” (U+201D, or “RIGHT DOUBLE QUOTATION MARK”). Injecting U+FFFD whenever a character can’t be decoded using the replace codec may have undesired consequences:

>>> bytes([0x73, 0xe2, 0x80]).decode(encoding="utf8", errors="replace")
's�'

which may be confusing to people—my personal preference would be failure.

Judging from your stacktrace, I see a

  File "/usr/local/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/encodings/cp1254.py", line 15, in decode
    return codecs.charmap_decode(input,errors,decoding_table)
           │                     │     │      └ '\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$...
           │                     │     └ 'strict'
           │                     └ 
           └ 
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 32110: character maps to 

which indicates that decode() tries to interpret your bytes as Windows-1254 encoded string (note the encodings/cp1254.py where the exception originates), and that fails because it’s a UTF-8 encoded string:

>>> bytes([0x73, 0xe2, 0x80, 0x9d]).decode(encoding="utf-8")
's”'
>>> bytes([0x73, 0xe2, 0x80, 0x9d]).decode(encoding="windows-1254")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/encodings/cp1254.py", line 15, in decode
    return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 3: character maps to <undefined>

Take a look at https://github.com/chardet/chardet/issues/148 for some more details, and at PR #545 which addresses this very issue (i.e. mispredicting a bytes object as Windows 1254 encoded string, instead of a UTF-8 encoded string).

@alvercau v2.29.6 just shipped with the fix.

As per PR https://github.com/commitizen-tools/commitizen/issues/545 I rebased my branch. Do as you wish with this fix.

diff --git a/commitizen/cmd.py b/commitizen/cmd.py
index 7f4efb6..71fbe8f 100644
--- a/commitizen/cmd.py
+++ b/commitizen/cmd.py
@@ -3,6 +3,8 @@ from typing import NamedTuple
 
 from charset_normalizer import from_bytes
 
+from commitizen.exceptions import CharacterSetDecodeError
+
 
 class Command(NamedTuple):
     out: str
@@ -12,6 +14,17 @@ class Command(NamedTuple):
     return_code: int
 
 
+def _try_decode(bytes_: bytes) -> str:
+    try:
+        return bytes_.decode("utf-8")
+    except UnicodeDecodeError:
+        charset_match = from_bytes(bytes_).best()
+        try:
+            return bytes_.decode(charset_match.encoding)
+        except UnicodeDecodeError as e:
+            raise CharacterSetDecodeError() from e
+
+
 def run(cmd: str) -> Command:
     process = subprocess.Popen(
         cmd,
@@ -23,8 +36,8 @@ def run(cmd: str) -> Command:
     stdout, stderr = process.communicate()
     return_code = process.returncode
     return Command(
-        str(from_bytes(stdout).best()),
-        str(from_bytes(stderr).best()),
+        _try_decode(stdout),
+        _try_decode(stderr),
         stdout,
         stderr,
         return_code,
diff --git a/commitizen/exceptions.py b/commitizen/exceptions.py
index a95ab3b..16869b5 100644
--- a/commitizen/exceptions.py
+++ b/commitizen/exceptions.py
@@ -26,6 +26,7 @@ class ExitCode(enum.IntEnum):
     INVALID_CONFIGURATION = 19
     NOT_ALLOWED = 20
     NO_INCREMENT = 21
+    UNRECOGNIZED_CHARACTERSET_ENCODING = 22
 
 
 class CommitizenException(Exception):
@@ -148,3 +149,7 @@ class InvalidConfigurationError(CommitizenException):
 
 class NotAllowed(CommitizenException):
     exit_code = ExitCode.NOT_ALLOWED
+
+
+class CharacterSetDecodeError(CommitizenException):
+    exit_code = ExitCode.UNRECOGNIZED_CHARACTERSET_ENCODING
diff --git a/docs/exit_codes.md b/docs/exit_codes.md
index f4c2fa8..a0448a2 100644
--- a/docs/exit_codes.md
+++ b/docs/exit_codes.md
@@ -28,4 +28,5 @@ These exit codes can be found in `commitizen/exceptions.py::ExitCode`.
 | InvalidCommandArgumentError | 18        | The argument provide to command is invalid (e.g. `cz check -commit-msg-file filename --rev-range master..`) |
 | InvalidConfigurationError   | 19        | An error was found in the Commitizen Configuration, such as duplicates in `change_type_order`               |
 | NotAllowed                  | 20        | `--incremental` cannot be combined with a `rev_range`                                                       |
-| NoneIncrementExit           | 21        | The commits found are not elegible to be bumped                                                             |
+| NoneIncrementExit           | 21        | The commits found are not eligible to be bumped                                                             |
+| CharacterSetDecodeError     | 22        | The character encoding of commits could not be determined    

And for laughs the above examples:

>>> charset_normalizer.from_bytes(b'').best().encoding
'utf_8'
>>> charset_normalizer.from_bytes(bytes([0x73, 0xe2, 0x80, 0x9d])).best().encoding
'cp775'  # Mispredicted valid utf-8
>>> charset_normalizer.from_bytes(bytes([0x73, 0xe2, 0x80])).best().encoding
'cp037'

Thanks for taking the time to write such a detailed reply. You’re right, and I look forward to the more robust logic!