pip: Problems on Windows with username or hostname containing non-ASCII characters

When organizing a Python training recently, one participant failed to use pip after a fresh Python 2.7.11 installation on Windows. Quick investigation showed that the reason problem was ä in her username. We failed to workaround that even by creating a new account and needed to use python setup.py install instead.

I now tried to reproduce the problem on my virtual machine. My main account there has only ASCII characters in the username but I created another for testing purposes. Clearly everything is not correct:

C:\Users\Ürjö>pip
Traceback (most recent call last):
  File "c:\python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\Scripts\pip.exe\__main__.py", line 9, in <module>
  File "c:\python27\lib\site-packages\pip\__init__.py", line 210, in main
    cmd_name, cmd_args = parseopts(args)
  File "c:\python27\lib\site-packages\pip\__init__.py", line 165, in parseopts
    parser.print_help()
  File "c:\python27\lib\optparse.py", line 1670, in print_help
    file.write(self.format_help().encode(encoding, "replace"))
  File "c:\python27\lib\optparse.py", line 1650, in format_help
    result.append(self.format_option_help(formatter))
  File "c:\python27\lib\optparse.py", line 1633, in format_option_help
    result.append(group.format_help(formatter))
  File "c:\python27\lib\optparse.py", line 1114, in format_help
    result += OptionContainer.format_help(self, formatter)
  File "c:\python27\lib\optparse.py", line 1085, in format_help
    result.append(self.format_option_help(formatter))
  File "c:\python27\lib\optparse.py", line 1074, in format_option_help
    result.append(formatter.format_option(option))
  File "c:\python27\lib\optparse.py", line 316, in format_option
    help_text = self.expand_default(option)
  File "c:\python27\lib\site-packages\pip\baseparser.py", line 112, in expand_default
    return optparse.IndentedHelpFormatter.expand_default(self, option)
  File "c:\python27\lib\optparse.py", line 288, in expand_default
    return option.help.replace(self.default_tag, str(default_value))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 9: ordinal not in range(128)

Interestingly installation and uninstallation seem to work fine also with this account. I guess the difference with the problem I saw earlier could be that my main user/admin doesn’t have non-ASCII characters.


UPDATE: It later turned out that pip install is totally broken if the hostname has non-ASCII characters. That explains why creating a new account with just ASCII characters in the username didn’t didn’t work when I encountered this first time and also why I couldn’t reproduce that more severe problem with just an account with non-ASCII username.

A workaround for both of these problems is using --no-cache-dir. Both problems are also fixed by PR #3970 that hopefully gets merged and released at some point.

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 39 (34 by maintainers)

Commits related to this issue

Most upvoted comments

PR #3970 fixes both the more severe problem with pip install crashing if hostname has non-ASCII characters, and the less severe but annoying crash with pip --help if username has non-ASCII characters.

Yes, --cache-dir seems to be causing this. I debugged the problem on my machine (see the original description) and noticed that it crashes when formatting help text for exactly that option. Looking at the pip and optparse code I think I found the root cause:

  • pip sets default value for --cache-dir based on what pip.utils.appdirs.user_cache_dir returns. It is a Unicode string and in my case it is u'C:\\Users\\\xdcrj\xf6\\AppData\\Local\\pip\\Cache'.
  • optparse formats the default value in HelpFormatter.expand_default (line 288) and uses str(default_value) there.

Based on this analysis the real bug would be in optparse. Not sure what’s the best way to avoid it on pip side.