pipenv: Pipenv fails to install latest version of pyarrow on linux

Pipenv fails to install and sync latest version of pyarrow on linux. As far as I could tell this is due to a difference between their versioning of the latest release:

linux, windows: 0.9.0 (https://pypi.python.org/pypi/pyarrow)
macOS: 0.9.0.post1 (https://pypi.python.org/pypi/pyarrow/0.9.0.post1)

When I attempt to install pyarrow on a linux, pipenv returns the following:

And on a macOS, it works:

Howerver if I specify the version to: pyarrow==0.9.0 On linux, it works:

And on macOS, it does not:

I know that this is a edge case, and according to here, it should be fixed when the pyarrow team releases a working macOS wheel of the 0.9.0 version. However pip can correctly identify and install each OS specific version. So, I, personally, think this should work on pipenv too.

$ python -m pipenv.help output

Pipenv version: '11.9.0'

Pipenv location: '/usr/local/lib/python3.6/dist-packages/pipenv'

Python location: '/home/vasv/.local/bin/python'

Other Python installations in PATH:

2.7: /usr/bin/python2.7
2.7: /usr/bin/python2.7
3.6: /usr/bin/python3.6m
3.6: /usr/bin/python3.6
3.6.3: /home/vasv/.local/bin/python
2.7.14: /usr/bin/python
2.7.14: /usr/bin/python2
3.6.3: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.13.0-38-generic',
 'platform_system': 'Linux',
 'platform_version': '#43-Ubuntu SMP Wed Mar 14 15:20:44 UTC 2018',
 'python_full_version': '3.6.3',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

LS_COLORS
LESSCLOSE
LANG
GDM_LANG
HISTIGNORE
DISPLAY
COLORTERM
NO_AT_BRIDGE
XDG_VTNR
SSH_AUTH_SOCK
MANDATORY_PATH
S_COLORS
XDG_SESSION_ID
XDG_GREETER_DATA_DIR
USER
DESKTOP_SESSION
DEFAULTS_PATH
QT_QPA_PLATFORMTHEME
PWD
HOME
SSH_AGENT_PID
QT_ACCESSIBILITY
XDG_SESSION_TYPE
XDG_DATA_DIRS
XDG_SESSION_DESKTOP
TERM
SHELL
XDG_SEAT_PATH
SHLVL
XDG_SEAT
LANGUAGE
WINDOWID
GDMSESSION
TERMINFO
LOGNAME
DBUS_SESSION_BUS_ADDRESS
XDG_RUNTIME_DIR
XAUTHORITY
XDG_SESSION_PATH
XDG_CONFIG_DIRS
PATH
PS1
PKG_CONFIG_PATH
PYTHON
GCC_COLORS
LESSOPEN
_
OLDPWD
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH
PYTHONUNBUFFERED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /home/vasv/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /home/vasv/Workspace/antenna_protocol

Contents of Pipfile (‘/home/vasv/Workspace/antenna_protocol/Pipfile’):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pyarrow = ">=0.9.0,<0.9.1"
logzero = "*"
async-timeout = "*"
aioreactive = "*"
uvloop = {version = "*", sys_platform = "!= 'win32'"}

[dev-packages]
pipenv = "*"

[requires]
python_version = "3.6"

Contents of Pipfile.lock (‘/home/vasv/Workspace/antenna_protocol/Pipfile.lock’):

{
    "_meta": {
        "hash": {
            "sha256": "840ae8c8bc18ece383055eb1462561e4490412e84042a0bc25d5feb13b09f693"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "aioreactive": {
            "hashes": [
                "sha256:201de56d7bc12854dda45a7e289dce49c99f7d3a50296feb48a401992f37b678",
                "sha256:6c31397f2be441fdcf951f74298ddb7bf64670b8e3b34cd0f537b02016eb40a9"
            ],
            "index": "pypi",
            "version": "==0.5.0"
        },
        "async-timeout": {
            "hashes": [
                "sha256:00cff4d2dce744607335cba84e9929c3165632da2d27970dbc55802a0c7873d0",
                "sha256:9093db5b8ddbe4b8f6885d1a6e0ad84ae3155464cbf6877c387605244c285f3c"
            ],
            "index": "pypi",
            "version": "==2.0.1"
        },
        "futures": {
            "hashes": [
                "sha256:9ec02aa7d674acb8618afb127e27fde7fc68994c0437ad759fa094a574adb265",
                "sha256:ec0a6cb848cc212002b9828c3e34c675e0c9ff6741dc445cab6fdd4e1085d1f1"
            ],
            "version": "==3.2.0"
        },
        "logzero": {
            "hashes": [
                "sha256:34fa1e2e436dfa9f37e5ff8750e932bafe0c5abbb42e1f669e4cf5ce1f179142",
                "sha256:818072e4fcb53a3f6fb4114a92f920e1135fe6f47bffd9dc2b6c4d10eedacf27"
            ],
            "index": "pypi",
            "version": "==1.5.0"
        },
        "numpy": {
            "hashes": [
                "sha256:0739146eaf4985962f07c62f7133aca89f3a600faac891ce6c7f3a1e2afe5272",
                "sha256:07e21f14490324cc1160db101e9b6c1233c33985af4cb1d301dd02650fea1d7f",
                "sha256:0f6a5ed0cd7ab1da11f5c07a8ecada73fc55a70ef7bb6311a4109891341d7277",
                "sha256:0fd65cbbfdbf76bbf80c445d923b3accefea0fe2c2082049e0ce947c81fe1d3f",
                "sha256:20cac3123d791e4bf8482a580d98d6b5969ba348b9d5364df791ba3a666b660d",
                "sha256:528ce59ded2008f9e8543e0146acb3a98a9890da00adf8904b1e18c82099418b",
                "sha256:56e392b7c738bd70e6f46cf48c8194d3d1dd4c5a59fae4b30c58bb6ef86e5233",
                "sha256:675e0f23967ce71067d12b6944add505d5f0a251f819cfb44bdf8ee7072c090d",
                "sha256:6be6b0ca705321c178c9858e5ad5611af664bbdfae1df1541f938a840a103888",
                "sha256:719d914f564f35cce4dc103808f8297c807c9f0297ac183ed81ae8b5650e698e",
                "sha256:768e777cc1ffdbf97c507f65975c8686ebafe0f3dc8925d02ac117acc4669ce9",
                "sha256:7f76d406c6b998d6410198dcb82688dcdaec7d846aa87e263ccf52efdcfeba30",
                "sha256:8c18ee4dddd5c6a811930c0a7c7947bf16387da3b394725f6063f1366311187d",
                "sha256:99051e03b445117b26028623f1a487112ddf61a09a27e2d25e6bc07d37d94f25",
                "sha256:a1413d06abfa942ca0553bf3bccaff5fdb36d55b84f2248e36228db871147dab",
                "sha256:a7157c9ac6bddd2908c35ef099e4b643bc0e0ebb4d653deb54891d29258dd329",
                "sha256:a958bf9d4834c72dee4f91a0476e7837b8a2966dc6fcfc42c421405f98d0da51",
                "sha256:bb370120de6d26004358611441e07acda26840e41dfedc259d7f8cc613f96495",
                "sha256:d0928076d9bd8a98de44e79b1abe50c1456e7abbb40af7ef58092086f1a6c729",
                "sha256:d858423f5ed444d494b15c4cc90a206e1b8c31354c781ac7584da0d21c09c1c3",
                "sha256:e6120d63b50e2248219f53302af7ec6fa2a42ed1f37e9cda2c76dbaca65036a7",
                "sha256:f2b1378b63bdb581d5d7af2ec0373c8d40d651941d283a2afd7fc71184b3f570",
                "sha256:facc6f925c3099ac01a1f03758100772560a0b020fb9d70f210404be08006bcb"
            ],
            "version": "==1.14.2"
        },
        "pyarrow": {
            "hashes": [
                "sha256:5d06a1df6c1c43a122cd05077a3d47ae8a53dabc0f3620aba78c7d434075b2c8",
                "sha256:92c7441d3a57870e34a47bd30d55dfd4c5d49d29893e3221955608e4a10fa57d",
                "sha256:df1b0b67ab4029628264219c5f236f3c828c030e14b455cfd2b2a9d41cf3609b"
            ],
            "index": "pypi",
            "version": "==0.9.0.post1"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        },
        "uvloop": {
            "hashes": [
                "sha256:01cf7199728867f406ba5af78cc47c80acd663ccc52cae105e737a997f1b2bca",
                "sha256:68574150720a380509a3409bf2941be0199cfdacff144a97502fb29c250ba927",
                "sha256:7ee14835a75c72227d3f8a3f370519a3106a6f02e5453f275f16437ebdb92953",
                "sha256:7fba5f390db607b2f026bc598df6b2a2a9e062bffe82910b5ffe2b88560135e5",
                "sha256:89c3bfaad77625490c42a6b99af1879234767ab0c31dd193486a909506e5e549",
                "sha256:b057ef2b0d0162c1ef257f43a95f59bfec37ee9a75cc5412d6b7f9ac6d1d69cb",
                "sha256:e7c871ba3edd5fcf2afb756de88a9a65245070161e24f75abe79c0a241bb8c76"
            ],
            "index": "pypi",
            "markers": "sys_platform != 'win32'",
            "version": "==0.9.1"
        }
    },
    "develop": {
        "certifi": {
            "hashes": [
                "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296",
                "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d"
            ],
            "version": "==2018.1.18"
        },
        "pipenv": {
            "hashes": [
                "sha256:325b26672172f11cf46ee5fa31cc4c751df24f920617ca21eeb68fce70a394e2",
                "sha256:dab6b2e5347757cda7973e6c6065c30c020c5ff8b0687306b83e962aea4adc6c"
            ],
            "index": "pypi",
            "version": "==11.10.0"
        },
        "virtualenv": {
            "hashes": [
                "sha256:1d7e241b431e7afce47e77f8843a276f652699d1fa4f93b9d8ce0076fd7b0b54",
                "sha256:e8e05d4714a1c51a2f5921e62f547fcb0f713ebbe959e0a7f585cc8bef71d11f"
            ],
            "version": "==15.2.0"
        },
        "virtualenv-clone": {
            "hashes": [
                "sha256:4507071d81013fd03ea9930ec26bc8648b997927a11fa80e8ee81198b57e0ac7",
                "sha256:b5cfe535d14dc68dfc1d1bb4ac1209ea28235b91156e2bba8e250d291c3fb4f8"
            ],
            "version": "==0.3.0"
        }
    }
}

$ python -m pipenv.help (Linux):

$ python -m pipenv.help output

Pipenv version: '11.9.0'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.6'

Other Python installations in PATH:

2.7: /usr/bin/python2.7
2.7: /usr/bin/python2.7
3.6: /usr/local/bin/python3.6m
3.6: /usr/local/bin/python3.6
3.6.5: /usr/local/opt/python/libexec/bin/python
2.7.10: /usr/bin/python
3.6.5: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.5.0: Mon Mar  5 22:24:32 PST '
                     '2018; root:xnu-4570.51.1~1/RELEASE_X86_64',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

LANG
XPC_FLAGS
HISTIGNORE
DISPLAY
TERM_SESSION_ID
EDITOR
JAVA_HOME
SSH_AUTH_SOCK
USER
LSCOLORS
PWD
HOME
LC_CTYPE
TERM_PROGRAM
TERM_PROGRAM_VERSION
Apple_PubSub_Socket_Render
TMPDIR
XPC_SERVICE_NAME
SHELL
TERM
SHLVL
CPPFLAGS
LOGNAME
LDFLAGS
PATH
PS1
PYTHON
HOMEBREW_GITHUB_API_TOKEN
_
OLDPWD
__CF_USER_TEXT_ENCODING
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH
PYTHONUNBUFFERED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /usr/local/bin:/usr/local/sbin:/usr/local/opt/python/libexec/bin:/usr/local/opt/go/libexec/bin:/usr/bin:/bin:/usr/sbin:/sbin
SHELL: /usr/local/bin/bash
EDITOR: code
LANG: en_US.UTF-8
PWD: /Users/hvolkoff/Workspace/Python/antenna_protocol

Contents of Pipfile (‘/Users/hvolkoff/Workspace/Python/antenna_protocol/Pipfile’):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pyarrow = ">=0.9.0,<0.9.1"
logzero = "*"
async-timeout = "*"
aioreactive = "*"
uvloop = {version = "*", sys_platform = "!= 'win32'"}

[dev-packages]
pipenv = "*"

[requires]
python_version = "3.6"

Contents of Pipfile.lock (‘/Users/hvolkoff/Workspace/Python/antenna_protocol/Pipfile.lock’):

{
    "_meta": {
        "hash": {
            "sha256": "840ae8c8bc18ece383055eb1462561e4490412e84042a0bc25d5feb13b09f693"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "aioreactive": {
            "hashes": [
                "sha256:201de56d7bc12854dda45a7e289dce49c99f7d3a50296feb48a401992f37b678",
                "sha256:6c31397f2be441fdcf951f74298ddb7bf64670b8e3b34cd0f537b02016eb40a9"
            ],
            "index": "pypi",
            "version": "==0.5.0"
        },
        "async-timeout": {
            "hashes": [
                "sha256:00cff4d2dce744607335cba84e9929c3165632da2d27970dbc55802a0c7873d0",
                "sha256:9093db5b8ddbe4b8f6885d1a6e0ad84ae3155464cbf6877c387605244c285f3c"
            ],
            "index": "pypi",
            "version": "==2.0.1"
        },
        "logzero": {
            "hashes": [
                "sha256:34fa1e2e436dfa9f37e5ff8750e932bafe0c5abbb42e1f669e4cf5ce1f179142",
                "sha256:818072e4fcb53a3f6fb4114a92f920e1135fe6f47bffd9dc2b6c4d10eedacf27"
            ],
            "index": "pypi",
            "version": "==1.5.0"
        },
        "numpy": {
            "hashes": [
                "sha256:0739146eaf4985962f07c62f7133aca89f3a600faac891ce6c7f3a1e2afe5272",
                "sha256:07e21f14490324cc1160db101e9b6c1233c33985af4cb1d301dd02650fea1d7f",
                "sha256:0f6a5ed0cd7ab1da11f5c07a8ecada73fc55a70ef7bb6311a4109891341d7277",
                "sha256:0fd65cbbfdbf76bbf80c445d923b3accefea0fe2c2082049e0ce947c81fe1d3f",
                "sha256:20cac3123d791e4bf8482a580d98d6b5969ba348b9d5364df791ba3a666b660d",
                "sha256:528ce59ded2008f9e8543e0146acb3a98a9890da00adf8904b1e18c82099418b",
                "sha256:56e392b7c738bd70e6f46cf48c8194d3d1dd4c5a59fae4b30c58bb6ef86e5233",
                "sha256:675e0f23967ce71067d12b6944add505d5f0a251f819cfb44bdf8ee7072c090d",
                "sha256:6be6b0ca705321c178c9858e5ad5611af664bbdfae1df1541f938a840a103888",
                "sha256:719d914f564f35cce4dc103808f8297c807c9f0297ac183ed81ae8b5650e698e",
                "sha256:768e777cc1ffdbf97c507f65975c8686ebafe0f3dc8925d02ac117acc4669ce9",
                "sha256:7f76d406c6b998d6410198dcb82688dcdaec7d846aa87e263ccf52efdcfeba30",
                "sha256:8c18ee4dddd5c6a811930c0a7c7947bf16387da3b394725f6063f1366311187d",
                "sha256:99051e03b445117b26028623f1a487112ddf61a09a27e2d25e6bc07d37d94f25",
                "sha256:a1413d06abfa942ca0553bf3bccaff5fdb36d55b84f2248e36228db871147dab",
                "sha256:a7157c9ac6bddd2908c35ef099e4b643bc0e0ebb4d653deb54891d29258dd329",
                "sha256:a958bf9d4834c72dee4f91a0476e7837b8a2966dc6fcfc42c421405f98d0da51",
                "sha256:bb370120de6d26004358611441e07acda26840e41dfedc259d7f8cc613f96495",
                "sha256:d0928076d9bd8a98de44e79b1abe50c1456e7abbb40af7ef58092086f1a6c729",
                "sha256:d858423f5ed444d494b15c4cc90a206e1b8c31354c781ac7584da0d21c09c1c3",
                "sha256:e6120d63b50e2248219f53302af7ec6fa2a42ed1f37e9cda2c76dbaca65036a7",
                "sha256:f2b1378b63bdb581d5d7af2ec0373c8d40d651941d283a2afd7fc71184b3f570",
                "sha256:facc6f925c3099ac01a1f03758100772560a0b020fb9d70f210404be08006bcb"
            ],
            "version": "==1.14.2"
        },
        "pyarrow": {
            "hashes": [
                "sha256:5d06a1df6c1c43a122cd05077a3d47ae8a53dabc0f3620aba78c7d434075b2c8",
                "sha256:92c7441d3a57870e34a47bd30d55dfd4c5d49d29893e3221955608e4a10fa57d",
                "sha256:df1b0b67ab4029628264219c5f236f3c828c030e14b455cfd2b2a9d41cf3609b"
            ],
            "index": "pypi",
            "version": "==0.9.0.post1"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        },
        "uvloop": {
            "hashes": [
                "sha256:01cf7199728867f406ba5af78cc47c80acd663ccc52cae105e737a997f1b2bca",
                "sha256:68574150720a380509a3409bf2941be0199cfdacff144a97502fb29c250ba927",
                "sha256:7ee14835a75c72227d3f8a3f370519a3106a6f02e5453f275f16437ebdb92953",
                "sha256:7fba5f390db607b2f026bc598df6b2a2a9e062bffe82910b5ffe2b88560135e5",
                "sha256:89c3bfaad77625490c42a6b99af1879234767ab0c31dd193486a909506e5e549",
                "sha256:b057ef2b0d0162c1ef257f43a95f59bfec37ee9a75cc5412d6b7f9ac6d1d69cb",
                "sha256:e7c871ba3edd5fcf2afb756de88a9a65245070161e24f75abe79c0a241bb8c76"
            ],
            "index": "pypi",
            "markers": "sys_platform != 'win32'",
            "version": "==0.9.1"
        }
    },
    "develop": {
        "certifi": {
            "hashes": [
                "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296",
                "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d"
            ],
            "version": "==2018.1.18"
        },
        "pipenv": {
            "hashes": [
                "sha256:325b26672172f11cf46ee5fa31cc4c751df24f920617ca21eeb68fce70a394e2",
                "sha256:dab6b2e5347757cda7973e6c6065c30c020c5ff8b0687306b83e962aea4adc6c"
            ],
            "index": "pypi",
            "version": "==11.10.0"
        },
        "virtualenv": {
            "hashes": [
                "sha256:1d7e241b431e7afce47e77f8843a276f652699d1fa4f93b9d8ce0076fd7b0b54",
                "sha256:e8e05d4714a1c51a2f5921e62f547fcb0f713ebbe959e0a7f585cc8bef71d11f"
            ],
            "version": "==15.2.0"
        },
        "virtualenv-clone": {
            "hashes": [
                "sha256:4507071d81013fd03ea9930ec26bc8648b997927a11fa80e8ee81198b57e0ac7",
                "sha256:b5cfe535d14dc68dfc1d1bb4ac1209ea28235b91156e2bba8e250d291c3fb4f8"
            ],
            "version": "==0.3.0"
        }
    }
}

$ python -m pipenv.help (macOS):

Expected result

Pipenv should correctly install each OS specific version.

Actual result

Pipenv fails to install latest version of pyarrow under Linux

Steps to replicate

Install pyarrow under Linux with pipenv

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 18 (9 by maintainers)

Most upvoted comments

pyarrow 0.9.0 contains a packaging problem (iirc), and the wheels they built for macOS couldn’t be installed. To remedy this (you cannot re-upload a file with the same name on PyPI, in case you’re wondering), they released 0.9.0.post1, which only contains macOS wheels. This means that when you pip install pyarrow, you get 0.9.0.post1 on macOS, and 0.9.0 everywhere else.

When Pipenv does locking, it needs to (well, tries to) make the lock file cross-platform. With a wildcard installation (pyarrow = "*"), it needs to find a latest version for all platforms—but there isn’t one! pyarrow 0.9.0.post1 only works on macOS, and 0.9.0 does not work on macOS (they deleted the broken macOS wheels). That’s why locking fails.

pip works because it only needs to care about the platform you’re running on. There is a valid latest version no matter where it is running, and that’s good enough for pip.

Knowing the root cause, you can actually work around this problem by pipenv install "pyarrow!=0.9.0,!=0.9.0.post1". As long as Pipenv does not need to consider those versions, it can resolve pyarrow correctly (to 0.8.0, until a newer version is released). But please, please acknowledge this issue to the pyarrow team. They need to know this is a problem so not to do this again in the future.

uranusjr on May 16, 2018

Hi, I am on the latest pipenv release, and it still fails:

HeavenVolkoff on Apr 15, 2018

@kadrach Thanks for the effort!

uranusjr on May 17, 2018

So, that’s the weird thing. if I install pyarrow using only pip, inside the pipenv venv, it install correctly. Here’s the log: install.pip.txt It also installs correctly with pipenv install --skip-lock pyarrow. Here’s the log: install.skip.txt

HeavenVolkoff on Apr 14, 2018