pip: [Windows] Could not install packages due to an EnvironmentError: [Errno 42] Illegal byte sequence

Environment

pip version: 18.0
Python version: 2.7.15 64 bit
OS: Windows 10.0.17134.165 x64
The console locale is set to cp872.
The console font is consolas.

Description pip throws an exception when trying to display a progress bar for the download of any package on Windows.

Expected behavior pip should work and install the package correctly.

How to Reproduce

Note that this is environment dependent.

pip install -I --no-cache-dir --verbose wheel

Using --progress-bar ascii doesn’t help. --progress-bar off works, but is sub-optimal as you get no progress.

Analysis The following seems to succeed, while later attempts to actually write the characters to the console fail. 😖 https://github.com/pypa/pip/blob/3e81d8cde142687deda8fb5a07ca3fb38906b007/src/pip/_internal/utils/ui.py#L52-L60

It will work on Python 3 since it has unicode support for the console. There is win-unicode-console which could be used to get similar support for Python 2. But the check should probably be fixed anyhow.

Output

>pip install -I --no-cache-dir --verbose wheel
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Config variable 'Py_UNICODE_SIZE' is unset, Python ABI tag may be incorrect
Created temporary directory: c:\users\user\appdata\local\temp\pip-ephem-wheel-cache-5axu12
Created temporary directory: c:\users\user\appdata\local\temp\pip-req-tracker-r7iyls
Created requirements tracker 'c:\\users\\user\\appdata\\local\\temp\\pip-req-tracker-r7iyls'
Created temporary directory: c:\users\user\appdata\local\temp\pip-install-5priwk
Collecting wheel
  1 location(s) to search for versions of wheel:
  * https://pypi.org/simple/wheel/
  Getting page https://pypi.org/simple/wheel/
  Starting new HTTPS connection (1): pypi.org:443
  https://pypi.org:443 "GET /simple/wheel/ HTTP/1.1" 200 7077
  Analyzing links from page https://pypi.org/simple/wheel/
    Found link https://files.pythonhosted.org/packages/51/15/d31364732deb6e7d1490f8a08f011997b52b4b106d592d36e917404be613/wheel-0.1.tar.gz#sha256=2cc0dd3d9465ae323ad6e85fc9dc54ab3664da2c60ce42189b9ae4c44de7d8bd (from https://pypi.org/simple/wheel/), version: 0.1
    Found link https://files.pythonhosted.org/packages/e2/1e/aedf6a9c6c0a588f96e9fe6e953de0c9f9084baad05f3ed0bd4a9b4a469f/wheel-0.2.tar.gz#sha256=82026a421ca379affefa9a0cb85807047e7184574a92f406670b2dcc3384da36 (from https://pypi.org/simple/wheel/), version: 0.2
    <snip>
    Found link https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl#sha256=80044e51ec5bbf6c894ba0bc48d26a8c20a9ba629f4ca19ea26ecfcf87685f5f (from https://pypi.org/simple/wheel/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*), version: 0.31.1
    Found link https://files.pythonhosted.org/packages/2a/fb/aefe5d5dbc3f4fe1e815bcdb05cbaab19744d201bbc9b59cfa06ec7fc789/wheel-0.31.1.tar.gz#sha256=0a2e54558a0628f2145d2fc822137e322412115173e8a2ddbe1c9024338ae83c (from https://pypi.org/simple/wheel/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*), version: 0.31.1
  Using version 0.31.1 (newest of versions: 0.1, 0.2, 0.3, 0.4, 0.4.1, 0.4.2, 0.5, 0.6, 0.7, 0.8, 0.9, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0, 0.17.0, 0.18.0, 0.19.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.25.0, 0.26.0, 0.27.0, 0.28.0, 0.29.0, 0.30.0, 0.31.0, 0.31.1)
  Created temporary directory: c:\users\user\appdata\local\temp\pip-unpack-euk6au
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl HTTP/1.1" 200 41567
  Downloading https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl (41kB)
  Downloading from URL https://files.pythonhosted.org/packages/81/30/e935244ca6165187ae8be876b6316ae201b71485538ffac1d718843025a9/wheel-0.31.1-py2.py3-none-any.whl#sha256=80044e51ec5bbf6c894ba0bc48d26a8c20a9ba629f4ca19ea26ecfcf87685f5f (from https://pypi.org/simple/wheel/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*)
    24% |Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pip\_internal\commands\install.py", line 299, in run
    resolver.resolve(requirement_set)
  File "c:\python27\lib\site-packages\pip\_internal\resolve.py", line 102, in resolve
    self._resolve_one(requirement_set, req)
  File "c:\python27\lib\site-packages\pip\_internal\resolve.py", line 256, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "c:\python27\lib\site-packages\pip\_internal\resolve.py", line 209, in _get_abstract_dist_for
    self.require_hashes
  File "c:\python27\lib\site-packages\pip\_internal\operations\prepare.py", line 283, in prepare_linked_requirement
    progress_bar=self.progress_bar
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 836, in unpack_url
    progress_bar=progress_bar
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 673, in unpack_http_url
    progress_bar)
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 897, in _download_http_url
    _download_url(resp, link, content_file, hashes, progress_bar)
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 617, in _download_url
    hashes.check_against_chunks(downloaded_chunks)
  File "c:\python27\lib\site-packages\pip\_internal\utils\hashes.py", line 48, in check_against_chunks
    for chunk in chunks:
  File "c:\python27\lib\site-packages\pip\_internal\download.py", line 585, in written_chunks
    for chunk in chunks:
  File "c:\python27\lib\site-packages\pip\_internal\utils\ui.py", line 161, in iter
    self.next(n)
  File "c:\python27\lib\site-packages\pip\_vendor\progress\__init__.py", line 73, in next
    self.update()
  File "c:\python27\lib\site-packages\pip\_vendor\progress\bar.py", line 86, in update
    self.writeln(line)
  File "c:\python27\lib\site-packages\pip\_vendor\progress\helpers.py", line 68, in writeln
    print(line, end='', file=self.file)
  File "c:\python27\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 141, in write
    self.write_and_convert(text)
  File "c:\python27\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 169, in write_and_convert
    self.write_plain_text(text, cursor, len(text))
  File "c:\python27\lib\site-packages\pip\_vendor\colorama\ansitowin32.py", line 174, in write_plain_text
    self.wrapped.write(text[start:end])
IOError: [Errno 42] Illegal byte sequence
Cleaning up...
Removed build tracker 'c:\\users\\user\\appdata\\local\\temp\\pip-req-tracker-r7iyls'
There was an error checking the latest version of pip
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pip\_internal\utils\outdated.py", line 90, in pip_version_check
    state = SelfCheckState(cache_dir=options.cache_dir)
  File "c:\python27\lib\site-packages\pip\_internal\utils\outdated.py", line 25, in __init__
    self.statefile_path = os.path.join(cache_dir, "selfcheck.json")
  File "c:\python27\lib\ntpath.py", line 65, in join
    result_drive, result_path = splitdrive(path)
  File "c:\python27\lib\ntpath.py", line 115, in splitdrive
    if len(p) > 1:
TypeError: object of type 'bool' has no len()

P.S. The second exception is an entirely different bug… This happens when you use --no-cache-dir. Not sure if reported, if not, then a separate issue needs to be opened for that.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 4
Comments: 15 (12 by maintainers)

Commits related to this issue

Make the ascii progress bar really be ascii, not Unicode Found while debugging #5665 — committed to segevfiner/pip by segevfiner 6 years ago

Most upvoted comments

--progress-bar off also works.

segevfiner on Sep 22, 2018

One workaround that worked for me: use pip install <package> -q. This made pip to not display progressbar which were triggering issue. I had no output displayed, but at least it has installed the package.

undera on Sep 16, 2018

I also get this error on an updated Windows 10 with pip 18.0 and 18.1, didn’t have it in 9.0.1 or 10.0.1… I use --progress-bar=off, but as it was stated before - this is sub-optimal, there should be a fix rather than a workaround…

eranzim on Oct 15, 2018

Thanks to @segevfiner for fixing --progress-bar ascii to actually being ASCII (in #5671). With the next pip release (i.e. later this month), using --progress-bar ascii will be a better alternative to using -q/--quiet, --progress-bar off etc.

Other than that, I’m not sure it’d be worth anyone’s time to fix the unicode bug/problem that’s Python 2 only. A better option might just be to detect “invalid byte sequence” and tell the user in the error message, that they might want to use --progress-bar ascii.

pradyunsg on Jul 6, 2019

Essentially yes.

In my opinion, It’s a core issue in the way Python 2.7 (and below, most likely) handle the console code page if you call setlocale. I kinda opened the issue for documentation purposes (So that people can find it and get linked to workarounds), I doubted anyone would want to make any changes to encodings in Python 2.7 at this point. Let alone a potentially non-backwards compatible fix. Python 3 and all… 😁

#5671 doesn’t fix this. It only fixes the option for the user to override pip’s automatic choice and use an ASCII only progress bar. It will still be a required to use it manually, and placing it in a config file will have it take effect for Python 3 too, which the user likely doesn’t want.

Without implementing a workaround, pip will be broken by default for many new users on Python 2.7 on Windows (Ones who use code pages which trigger this, I’m not sure it happens with all of them), which I don’t think is particularly nice. Things should really work out of the box. Users shouldn’t need to apply workarounds themselves to get things to work with supported Python versions. Python 2.7 is still supported. For now… 😉

segevfiner on Jul 30, 2018