pipenv: 2018.11.26: pipenv emits pythons errors when creating a virtualenv

Issue description

Possibly related to #3229, pipenv continues to throw errors when creating a virtualenv:

$ pipenv --three
['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 160, in _create_subprocess\n    combine_stderr=combine_stderr)\n', '  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 134, in _spawn_subprocess\n    return subprocess.Popen(cmd, **options)\n', '  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 709, in __init__\n    restore_signals, start_new_session)\n', '  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1344, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\n', "FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/bin/pythonz': '/usr/local/bin/pythonz'\n", '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/contextmanagers.py", line 150, in spinner\n    yield _spinner\n', '  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 314, in run\n    write_to_stdout=True\n', '  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 162, in _create_subprocess\n    sys.stderr.write("Error %s while executing command %s", exc, " ".join(cmd._parts))\n', 'TypeError: write() takes exactly one argument (3 given)\n']
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 160, in _create_subprocess
    combine_stderr=combine_stderr)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 134, in _spawn_subprocess
    return subprocess.Popen(cmd, **options)
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/bin/pythonz': '/usr/local/bin/pythonz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1114, in invoke
    return Command.invoke(self, ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli/command.py", line 208, in cli
    clear=state.clear,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 574, in ensure_project
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 516, in ensure_virtualenv
    ensure_python(three=three, python=python)
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 397, in ensure_python
    path_to_python = find_a_system_python(python)
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 360, in find_a_system_python
    python_entry = finder.find_python_version(line)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/pythonfinder.py", line 114, in find_python_version
    major=major, minor=minor, patch=patch, pre=pre, dev=dev, arch=arch, name=name
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 396, in find_python_version
    ver = next(iter(self.get_pythons(sub_finder)), None)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 279, in get_pythons
    reverse=True
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 277, in <genexpr>
    (p for p in self._filter_paths(finder) if p.is_python),
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 264, in <genexpr>
    pth for pth in unnest(finder(p) for p in self.path_entries if p is not None)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/utils.py", line 251, in unnest
    for el in target:
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 264, in <genexpr>
    pth for pth in unnest(finder(p) for p in self.path_entries if p is not None)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/mixins.py", line 121, in find_python_version
    for child in unnest(self.pythons.values())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 531, in pythons
    for path, entry in self.children.items():
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/cached_property.py", line 35, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 501, in children
    for child_key, child_val in self._gen_children():
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 493, in _gen_children
    entry = PathEntry.create(path=child, **pass_args)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 575, in create
    _new = cls(**creation_args)
  File "<attrs generated init b90d7581ea07925e94241736776cf96c889eb52c>", line 16, in __init__
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/path.py", line 518, in get_py_version
    py_version = PythonVersion.from_path(path=self, name=self.name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/models/python.py", line 395, in from_path
    py_version = get_python_version(path.path.absolute().as_posix())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/utils.py", line 68, in get_python_version
    combine_stderr=False, write_to_stdout=False)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 314, in run
    write_to_stdout=True
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 162, in _create_subprocess
    sys.stderr.write("Error %s while executing command %s", exc, " ".join(cmd._parts))
TypeError: write() takes exactly one argument (3 given)

Expected result

Expected no error logging while simply creating a virtualenv.

Actual result

virtualenv creation fails.

Steps to replicate

$ pipenv --three

pipenv --support also emits these errors and fails to collect support information!

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 4
Comments: 30 (5 by maintainers)

Commits related to this issue

Update failure tests README I keep finding this page when trying to run failure tests, so updating the README that way: https://github.com/pypa/pipenv/issues/3363#issuecomment-452171564 — committed to citusdata/citus by SaitTalhaNisanci 3 years ago
Update failure tests README (#5197) * Update failure tests README I keep finding this page when trying to run failure tests, so updating the README that way: https://github.com/pypa/pipenv/issues... — committed to citusdata/citus by SaitTalhaNisanci 3 years ago
Update failure tests README (#5197) * Update failure tests README I keep finding this page when trying to run failure tests, so updating the README that way: https://github.com/pypa/pipenv/issues... — committed to citusdata/citus by SaitTalhaNisanci 3 years ago

Most upvoted comments

The current release sure seems hosed to me. Another temporary workaround I saw in a past issue was to specifically pass in a python path like so:

pipenv --three --python=`which python3`

I’ve been doing this with success for the time being 😬

+89

command-tab on Jan 8, 2019

Ah ok. And yeah the bug makes sense. I’ll just tag it upstream and make sure it gets fixed

Still issue persists, any idea when this fix will be released.

+10

pavithrarajs on Aug 4, 2019

Same issue with Ubuntu 18 LTS installed in Windows Subsystem for Linux.

Workaround works for me.

dahifi on Sep 23, 2019

Me too:

Catalina
Python 3.7.

Everything works fine once I comment out this line in my Pipfile:

[requires]

#python_version = "3.7.4"

edsu on Nov 19, 2019

That workaround wasn’t working for me. I had to downgrade to 2018.10.13 and then use the workaround on that thread :

pipenv install -d --python=$HOME/.pyenv/versions/3.7.1/bin/python

As far as I can tell, pipenv has been completely broken since October.

jarshwah on Jan 8, 2019

I m using wsl2 with arclinux (manjaro) It seems pipenv is calling the python in host env windows to install so we need to define where to find the python with: pipenv --python=<PATH_TO_PYTHON> or pipenv --python=which python3

works for me.

lencshu on Feb 26, 2020

This definitely still happens with homebrew pipenv and python… kind of surprised there isn’t a test to cover it seeing how long this issue has been around…

gormster on Sep 2, 2019

Can confirm this happens under WSL with Ubuntu 18.04 LTS. It seems like pipenv is going in search of a python interpreter rather than using the one that it is being executed under, and in-so-doing is traversing PATH in backwards order. I have multiple python environments installed, and whenever I manually remove one from PATH, pipenv will find a different one and complain about that one instead.

wbierman on Mar 11, 2020

@JarredStanford @edsu This worked for me as well on WSL in VSCode (I didn’t try an outside shell as it worked fine for a different project).

@command-tab Adding “–python=which python3” also worked. I did not try adding “–three” and it seems not necessary for my use case / version.

After getting both workarounds and I was able to get it to work, I removed them and tried a third time to ensure the environment was still causing an issue (narrator: it was).

@techalchemy I’m thinking this may not be “fully fixed” in 3330? It could be a tangential issue causing the same symptoms. In my case I’m using WSL and wondering if ‘python3’ and ‘python3.exe’ both being in my WSL path seems suspicious.

Would you think it best I open a new issue, would you like me to run additional diagnostics, or something else?

josh-dix on Dec 27, 2019

Me too:

Catalina

Python 3.7.

Everything works fine once I comment out this line in my Pipfile:
[requires]

#python_version = "3.7.4"

ahhhh this worked for me, thank you!

JarredStanford on Dec 1, 2019

Not sure how much this helps, but deleting the Pipfile, running pipenv lock, and then putting the contents of the old Pipfile back in and running pipenv install works for me permanently.

paramkpr on May 15, 2020

The reason for the last part of the exception, i.e. this bit:

  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pythonfinder/utils.py", line 68, in get_python_version
    combine_stderr=False, write_to_stdout=False)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 314, in run
    write_to_stdout=True
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/vistir/misc.py", line 162, in _create_subprocess
    sys.stderr.write("Error %s while executing command %s", exc, " ".join(cmd._parts))
TypeError: write() takes exactly one argument (3 given)

… is because there is a bug in the call to print an error when handling the first exception. This bug appears to have already been fixed in master back in Jan 2019 - https://github.com/pypa/pipenv/commit/574fe7308d9ac81d64da954722f35c9eee0dd467#diff-a59595db75020aeb9f688d6a0b4818e6L162. But since the latest version of pipenv is 2018.11.26, I imagine most don’t have a fixed version.

One can patch it manually. Open /usr/local/lib/<your-python-version>/site-packages/pipenv/vendor/vistir/misc.py, navigate to line 162, and change this:

sys.stderr.write("Error %s while executing command %s", exc, " ".join(cmd._parts))

… to this:

sys.stderr.write(f"Error {exc} while executing command " + " ".join(cmd._parts))

You will now get a more accurate exception report. In my case, the issue is to do with WSL Linux-based pipenv detecting Windows-based Python installations and having trouble running them, which is kind of obvious:

Error [Errno 8] Exec format error: '/mnt/c/Users/<user>/AppData/Local/Microsoft/WindowsApps/python3.exe' while executing command /mnt/c/Users/<user>/AppData/Local/Microsoft/WindowsApps/python3.exe -c import sys; print(['Traceback (most recent call last):\n', '  File "/usr/local/lib/python3.6/dist-packages/pipenv/vendor/vistir/contextmanagers.py", line 150, in spinner\n    yield _spinner\n', '  File "/usr/local/lib/python3.6/dist-packages/pipenv/vendor/vistir/misc.py", line 314, in run\n    write_to_stdout=True\n', '  File "/usr/local/lib/python3.6/dist-packages/pipenv/vendor/vistir/misc.py", line 160, in _create_subprocess\n    combine_stderr=combine_stderr)\n', '  File "/usr/local/lib/python3.6/dist-packages/pipenv/vendor/vistir/misc.py", line 134, in _spawn_subprocess\n    return subprocess.Popen(cmd, **options)\n', '  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__\n    restore_signals, start_new_session)\n', '  File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\n', "OSError: [Errno 8] Exec format error: '/mnt/c/Users/<user>/AppData/Local/Microsoft/WindowsApps/python3.exe'\n"]

For WSL users, I believe the issue can be alleviated by ensuring that any requirements on your Python version in Pipfile are exactly matched by a Linux-based Python installation, e.g.

[requires]
python_version = "3.6.8"  # Make sure this exactly matches an installed version, or remove it.

Hope this helps.

mr-smidge on Apr 21, 2020

Same issue with Ubuntu 18 LTS installed in Windows Subsystem for Linux.

same here

AlphaWong on Oct 28, 2019

@techalchemy this issue still seems to exist. Can you please either reopen it or make it clear that you want a new issue to be created?

Thanks.

pcolmer on Oct 18, 2019

I can also confirm that this problem is still happening, despite this GitHub issue being closed.

Do the maintainers need a new issue to be created?

pcolmer on Sep 16, 2019

I can not get pipenv to work at all with the current version.

$ which pipenv
/Users/josh/.pyenv/shims/pipenv

$ python -m pipenv --version
pipenv, version 2018.11.26

I tried installing with python3.7 from homebrew, and same issue.

It seems like there’s no urgency here, or that it’s not a big problem. Am I missing something, or is pipenv completely broken with the current version?

jarshwah on Jan 8, 2019

Any word on when a fix will be released for this? I am having this problem its preventing me from installing the pipenv requirements. I have tried @commandtab 's workaround with no such luck.

jasonrhaas on Dec 21, 2018