pip: pip issues UnicodeDecodeError on Windows 10 for Russian language

Pip version: 9.0.1
Python version: 3.6.0
Operating system: Microsoft Windows 10 Home Edition [Version 10.0.10586] for Russian language

Description:

pip issues UnicodeDecodeError on byte 0x8d in Windows 10 for Russian language. It is not a problem for Windows 7 Ultimate SP1 for English language. Probably has something to do with default CMD encoding, please fix it.

What I’ve run:

C:\WINDOWS\system32>pip install pyyaml
Collecting pyyaml
  Using cached PyYAML-3.12.tar.gz
Building wheels for collected packages: pyyaml
  Running setup.py bdist_wheel for pyyaml ... error
  Failed building wheel for pyyaml
  Running setup.py clean for pyyaml
Failed to build pyyaml
Installing collected packages: pyyaml
  Running setup.py install for pyyaml ... error
Exception:
Traceback (most recent call last):
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8d in position 68: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "c:\program files (x86)\python36-32\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8d in position 68: invalid start byte

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 16 (7 by maintainers)

Commits related to this issue

try fix encode error #4251 — committed to robinxb/pip by robinxb 7 years ago
try fix encode error #4251 — committed to robinxb/pip by robinxb 7 years ago
Add a NEWS entry for #4310 which may cover #4110, #4251 and #3992 — committed to sakurai-youhei/pip by sakurai-youhei 7 years ago

Most upvoted comments

Add a solution here: run a new cmd.exe console chcp it will show the system default code, for example 936. open Lib/site-package/pip/compat/__init__.py around 75 line, change return s.decode('utf_8') to return s.decode('cp936')

It’s just a workaround. I think pip might need solve this issue asap, it’s not easy to find solution.

This may have a general solution using cdll. Not sure if this is the best solution on windows but I still made a PR for this issue.

+78

robinxb on Feb 11, 2017

I thought pip is suposed to be easy for users, is it possible to hide this problems from us ?😃

+20

xtipacko on Jan 24, 2017

Hey @JoeVogel!

pip 10 is currently in beta and has a fix for this. You can upgrade o it (if you don’t mind using a beta version) by running pip install -U --pre pip

pradyunsg on Apr 6, 2018

Encodings are not easy for anyone 😃 It’s certainly possible to deal with this as I said. Just the first time it’s come up (it’s a Python 3.6 change).

pfmoore on Jan 24, 2017

This is likely due to the fact that on Windows Python 3.6 switched to using UTF-8 for console IO. The code is running a subprocess, and then guessing the encoding of the subprocess output as being the same as the encoding of sys.stdout - which was true in Python ❤️.6 (arguably more by luck than anything else) but is no longer true in 3.6+

The simplest fix is probably to use locale.getpreferredencoding(False) for the encoding, as that’s the default encoding used in io.TextIOWrapper and for subprocess when universal_newlines is True.

pfmoore on Jan 24, 2017