pipenv: Release v2023.7.23 not resolving all platform hashes when pulling from a custom index

Issue description

After upgrading to 2023.7.23 my Pipfile.lock is not populated with all platform hashes when a custom repo is configured. Only the hashes for my OS platform are generated in Pipfile.lock. When I disabled the custom repo, all platform hashes are generated. I have not seen this behavior with earlier versions.

Expected result

I would expect all platform hashes to be generated in Pipfile.lock

{
    "_meta": {
        "hash": {
            "sha256": "90025a8184f3365cfb0e9f07920c49ce086e43ac5335361e1ca7a562e9ab9e30"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "cffi": {
            "hashes": [
                "sha256:00a9ed42e88df81ffae7a8ab6d9356b371399b91dbdf0c3cb1e84c03a13aceb5",
                "sha256:03425bdae262c76aad70202debd780501fabeaca237cdfddc008987c0e0f59ef",
                "sha256:04ed324bda3cda42b9b695d51bb7d54b680b9719cfab04227cdd1e04e5de3104",
                "sha256:0e2642fe3142e4cc4af0799748233ad6da94c62a8bec3a6648bf8ee68b1c7426",
                "sha256:173379135477dc8cac4bc58f45db08ab45d228b3363adb7af79436135d028405",
                "sha256:198caafb44239b60e252492445da556afafc7d1e3ab7a1fb3f0584ef6d742375",
                "sha256:1e74c6b51a9ed6589199c787bf5f9875612ca4a8a0785fb2d4a84429badaf22a",
                "sha256:2012c72d854c2d03e45d06ae57f40d78e5770d252f195b93f581acf3ba44496e",
                "sha256:21157295583fe8943475029ed5abdcf71eb3911894724e360acff1d61c1d54bc",
                "sha256:2470043b93ff09bf8fb1d46d1cb756ce6132c54826661a32d4e4d132e1977adf",
                "sha256:285d29981935eb726a4399badae8f0ffdff4f5050eaa6d0cfc3f64b857b77185",
                "sha256:30d78fbc8ebf9c92c9b7823ee18eb92f2e6ef79b45ac84db507f52fbe3ec4497",
                "sha256:320dab6e7cb2eacdf0e658569d2575c4dad258c0fcc794f46215e1e39f90f2c3",
                "sha256:33ab79603146aace82c2427da5ca6e58f2b3f2fb5da893ceac0c42218a40be35",
                "sha256:3548db281cd7d2561c9ad9984681c95f7b0e38881201e157833a2342c30d5e8c",
                "sha256:3799aecf2e17cf585d977b780ce79ff0dc9b78d799fc694221ce814c2c19db83",
                "sha256:39d39875251ca8f612b6f33e6b1195af86d1b3e60086068be9cc053aa4376e21",
                "sha256:3b926aa83d1edb5aa5b427b4053dc420ec295a08e40911296b9eb1b6170f6cca",
                "sha256:3bcde07039e586f91b45c88f8583ea7cf7a0770df3a1649627bf598332cb6984",
                "sha256:3d08afd128ddaa624a48cf2b859afef385b720bb4b43df214f85616922e6a5ac",
                "sha256:3eb6971dcff08619f8d91607cfc726518b6fa2a9eba42856be181c6d0d9515fd",
                "sha256:40f4774f5a9d4f5e344f31a32b5096977b5d48560c5592e2f3d2c4374bd543ee",
                "sha256:4289fc34b2f5316fbb762d75362931e351941fa95fa18789191b33fc4cf9504a",
                "sha256:470c103ae716238bbe698d67ad020e1db9d9dba34fa5a899b5e21577e6d52ed2",
                "sha256:4f2c9f67e9821cad2e5f480bc8d83b8742896f1242dba247911072d4fa94c192",
                "sha256:50a74364d85fd319352182ef59c5c790484a336f6db772c1a9231f1c3ed0cbd7",
                "sha256:54a2db7b78338edd780e7ef7f9f6c442500fb0d41a5a4ea24fff1c929d5af585",
                "sha256:5635bd9cb9731e6d4a1132a498dd34f764034a8ce60cef4f5319c0541159392f",
                "sha256:59c0b02d0a6c384d453fece7566d1c7e6b7bae4fc5874ef2ef46d56776d61c9e",
                "sha256:5d598b938678ebf3c67377cdd45e09d431369c3b1a5b331058c338e201f12b27",
                "sha256:5df2768244d19ab7f60546d0c7c63ce1581f7af8b5de3eb3004b9b6fc8a9f84b",
                "sha256:5ef34d190326c3b1f822a5b7a45f6c4535e2f47ed06fec77d3d799c450b2651e",
                "sha256:6975a3fac6bc83c4a65c9f9fcab9e47019a11d3d2cf7f3c0d03431bf145a941e",
                "sha256:6c9a799e985904922a4d207a94eae35c78ebae90e128f0c4e521ce339396be9d",
                "sha256:70df4e3b545a17496c9b3f41f5115e69a4f2e77e94e1d2a8e1070bc0c38c8a3c",
                "sha256:7473e861101c9e72452f9bf8acb984947aa1661a7704553a9f6e4baa5ba64415",
                "sha256:8102eaf27e1e448db915d08afa8b41d6c7ca7a04b7d73af6514df10a3e74bd82",
                "sha256:87c450779d0914f2861b8526e035c5e6da0a3199d8f1add1a665e1cbc6fc6d02",
                "sha256:8b7ee99e510d7b66cdb6c593f21c043c248537a32e0bedf02e01e9553a172314",
                "sha256:91fc98adde3d7881af9b59ed0294046f3806221863722ba7d8d120c575314325",
                "sha256:94411f22c3985acaec6f83c6df553f2dbe17b698cc7f8ae751ff2237d96b9e3c",
                "sha256:98d85c6a2bef81588d9227dde12db8a7f47f639f4a17c9ae08e773aa9c697bf3",
                "sha256:9ad5db27f9cabae298d151c85cf2bad1d359a1b9c686a275df03385758e2f914",
                "sha256:a0b71b1b8fbf2b96e41c4d990244165e2c9be83d54962a9a1d118fd8657d2045",
                "sha256:a0f100c8912c114ff53e1202d0078b425bee3649ae34d7b070e9697f93c5d52d",
                "sha256:a591fe9e525846e4d154205572a029f653ada1a78b93697f3b5a8f1f2bc055b9",
                "sha256:a5c84c68147988265e60416b57fc83425a78058853509c1b0629c180094904a5",
                "sha256:a66d3508133af6e8548451b25058d5812812ec3798c886bf38ed24a98216fab2",
                "sha256:a8c4917bd7ad33e8eb21e9a5bbba979b49d9a97acb3a803092cbc1133e20343c",
                "sha256:b3bbeb01c2b273cca1e1e0c5df57f12dce9a4dd331b4fa1635b8bec26350bde3",
                "sha256:cba9d6b9a7d64d4bd46167096fc9d2f835e25d7e4c121fb2ddfc6528fb0413b2",
                "sha256:cc4d65aeeaa04136a12677d3dd0b1c0c94dc43abac5860ab33cceb42b801c1e8",
                "sha256:ce4bcc037df4fc5e3d184794f27bdaab018943698f4ca31630bc7f84a7b69c6d",
                "sha256:cec7d9412a9102bdc577382c3929b337320c4c4c4849f2c5cdd14d7368c5562d",
                "sha256:d400bfb9a37b1351253cb402671cea7e89bdecc294e8016a707f6d1d8ac934f9",
                "sha256:d61f4695e6c866a23a21acab0509af1cdfd2c013cf256bbf5b6b5e2695827162",
                "sha256:db0fbb9c62743ce59a9ff687eb5f4afbe77e5e8403d6697f7446e5f609976f76",
                "sha256:dd86c085fae2efd48ac91dd7ccffcfc0571387fe1193d33b6394db7ef31fe2a4",
                "sha256:e00b098126fd45523dd056d2efba6c5a63b71ffe9f2bbe1a4fe1716e1d0c331e",
                "sha256:e229a521186c75c8ad9490854fd8bbdd9a0c9aa3a524326b55be83b54d4e0ad9",
                "sha256:e263d77ee3dd201c3a142934a086a4450861778baaeeb45db4591ef65550b0a6",
                "sha256:ed9cb427ba5504c1dc15ede7d516b84757c3e3d7868ccc85121d9310d27eed0b",
                "sha256:fa6693661a4c91757f4412306191b6dc88c1703f780c8234035eac011922bc01",
                "sha256:fcd131dd944808b5bdb38e6f5b53013c5aa4f334c5cad0c72742f6eba4b73db0"
            ],
            "version": "==1.15.1"
        },
        "cryptography": {
            "hashes": [
                "sha256:0d09fb5356f975974dbcb595ad2d178305e5050656affb7890a1583f5e02a306",
                "sha256:23c2d778cf829f7d0ae180600b17e9fceea3c2ef8b31a99e3c694cbbf3a24b84",
                "sha256:3fb248989b6363906827284cd20cca63bb1a757e0a2864d4c1682a985e3dca47",
                "sha256:41d7aa7cdfded09b3d73a47f429c298e80796c8e825ddfadc84c8a7f12df212d",
                "sha256:42cb413e01a5d36da9929baa9d70ca90d90b969269e5a12d39c1e0d475010116",
                "sha256:4c2f0d35703d61002a2bbdcf15548ebb701cfdd83cdc12471d2bae80878a4207",
                "sha256:4fd871184321100fb400d759ad0cddddf284c4b696568204d281c902fc7b0d81",
                "sha256:5259cb659aa43005eb55a0e4ff2c825ca111a0da1814202c64d28a985d33b087",
                "sha256:57a51b89f954f216a81c9d057bf1a24e2f36e764a1ca9a501a6964eb4a6800dd",
                "sha256:652627a055cb52a84f8c448185922241dd5217443ca194d5739b44612c5e6507",
                "sha256:67e120e9a577c64fe1f611e53b30b3e69744e5910ff3b6e97e935aeb96005858",
                "sha256:6af1c6387c531cd364b72c28daa29232162010d952ceb7e5ca8e2827526aceae",
                "sha256:6d192741113ef5e30d89dcb5b956ef4e1578f304708701b8b73d38e3e1461f34",
                "sha256:7efe8041897fe7a50863e51b77789b657a133c75c3b094e51b5e4b5cec7bf906",
                "sha256:84537453d57f55a50a5b6835622ee405816999a7113267739a1b4581f83535bd",
                "sha256:8f09daa483aedea50d249ef98ed500569841d6498aa9c9f4b0531b9964658922",
                "sha256:95dd7f261bb76948b52a5330ba5202b91a26fbac13ad0e9fc8a3ac04752058c7",
                "sha256:a74fbcdb2a0d46fe00504f571a2a540532f4c188e6ccf26f1f178480117b33c4",
                "sha256:a983e441a00a9d57a4d7c91b3116a37ae602907a7618b882c8013b5762e80574",
                "sha256:ab8de0d091acbf778f74286f4989cf3d1528336af1b59f3e5d2ebca8b5fe49e1",
                "sha256:aeb57c421b34af8f9fe830e1955bf493a86a7996cc1338fe41b30047d16e962c",
                "sha256:ce785cf81a7bdade534297ef9e490ddff800d956625020ab2ec2780a556c313e",
                "sha256:d0d651aa754ef58d75cec6edfbd21259d93810b73f6ec246436a21b7841908de"
            ],
            "index": "pypi",
            "version": "==41.0.3"
        },
        "pycparser": {
            "hashes": [
                "sha256:8ee45429555515e1f6b185e78100aea234072576aa43ab53aefcae078162fca9",
                "sha256:e644fdec12f7872f86c58ff790da456218b10f863970249516d60a5eaca77206"
            ],
            "version": "==2.21"
        }
    },
    "develop": {}
}

Actual result

Hashes in Pipfile.lock are only for my current OS, where I’m running pipenv install

{
    "_meta": {
        "hash": {
            "sha256": "90025a8184f3365cfb0e9f07920c49ce086e43ac5335361e1ca7a562e9ab9e30"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "cffi": {
            "hashes": [
                "sha256:39d39875251ca8f612b6f33e6b1195af86d1b3e60086068be9cc053aa4376e21",
                "sha256:d400bfb9a37b1351253cb402671cea7e89bdecc294e8016a707f6d1d8ac934f9"
            ],
            "version": "==1.15.1"
        },
        "cryptography": {
            "hashes": [
                "sha256:652627a055cb52a84f8c448185922241dd5217443ca194d5739b44612c5e6507",
                "sha256:6d192741113ef5e30d89dcb5b956ef4e1578f304708701b8b73d38e3e1461f34",
                "sha256:8f09daa483aedea50d249ef98ed500569841d6498aa9c9f4b0531b9964658922"
            ],
            "index": "pypi",
            "version": "==41.0.3"
        },
        "pycparser": {
            "hashes": [
                "sha256:8ee45429555515e1f6b185e78100aea234072576aa43ab53aefcae078162fca9",
                "sha256:e644fdec12f7872f86c58ff790da456218b10f863970249516d60a5eaca77206"
            ],
            "version": "==2.21"
        }
    },
    "develop": {}
}

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

  1. Create a Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.10"

[packages]
cryptography = "*"
  1. set pipenv version to 2023.7.23
  2. set PIPENV_PYPI_MIRROR to a custom repo
  3. run pipenv install
  4. inspect Pipfile.lock
$ pipenv --support

Pipenv version: '2023.7.23'

Pipenv location: '/Users/foo/.pyenv/versions/3.10.2/lib/python3.10/site-packages/pipenv'

Python location: '/Users/foo/.pyenv/versions/3.10.2/bin/python3.10'

OS Name: 'posix'

User pip version: '21.2.4'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.10.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '22.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 22.6.0: Wed Jul  5 22:21:56 PDT '
                     '2023; root:xnu-8796.141.3~6/RELEASE_X86_64',
 'python_full_version': '3.10.2',
 'python_version': '3.10',
 'sys_platform': 'darwin'}

System environment variables:

  • TERM_PROGRAM
  • PYENV_ROOT
  • TERM
  • SHELL
  • TMPDIR
  • TERM_PROGRAM_VERSION
  • AWS_CREDENTIAL_EXPIRATION
  • PIP_INDEX_URL
  • AWS_SESSION_TOKEN
  • TERM_SESSION_ID
  • AWS_VAULT
  • PYENV_VERSION
  • WORKSPACE
  • ZSH
  • USER
  • COMMAND_MODE
  • CODEARTIFACT_AUTH_TOKEN
  • SSH_AUTH_SOCK
  • PYENV_DIR
  • __CF_USER_TEXT_ENCODING
  • PAGER
  • LSCOLORS
  • PATH
  • LaunchInstanceID
  • AWS_DEFAULT_REGION
  • COMPOSE_FILES_DIR
  • __CFBundleIdentifier
  • PWD
  • AWS_SECRET_ACCESS_KEY
  • LANG
  • ITERM_PROFILE
  • AWS_REGION
  • PYENV_HOOK_PATH
  • XPC_FLAGS
  • RBENV_SHELL
  • PIPENV_PYPI_MIRROR
  • XPC_SERVICE_NAME
  • AWS_ACCESS_KEY_ID
  • PYENV_SHELL
  • SHLVL
  • HOME
  • COLORFGBG
  • LC_TERMINAL_VERSION
  • ITERM_SESSION_ID
  • LESS
  • LOGNAME
  • LC_TERMINAL
  • SECURITYSESSIONID
  • SQLITE_EXEMPT_PATH_FROM_VNODE_GUARDS
  • ASDF_HASHICORP_TERRAFORM_VERSION_FILE
  • COLORTERM
  • PYTHONDONTWRITEBYTECODE
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

  • PIPENV_PYPI_MIRROR: https://aws:....codeartifact.us-east-2.amazonaws.com/pypi/foo-python/simple/

Debug–specific environment variables:

  • PATH: /Users/foo/.pyenv/versions/3.8.7/bin:/Users/foo/.pyenv/versions/3.9.13/bin:/Users/foo/.pyenv/versions/3.10.2/bin:/Users/foo/.pyenv/versions/3.9.10/bin:/Users/foo/.pyenv/versions/3.8.10/bin:/Users/foo/.pyenv/versions/3.10.2/bin:/usr/local/Cellar/pyenv/2.3.16/libexec:/usr/local/Cellar/pyenv/2.3.16/plugins/python-build/bin:/Users/foo/.pyenv/shims:/Users/foo/bin:/Users/foo/.pyenv/bin:/Users/foo/.rbenv/shims:/Users/foo/bin:/Users/foo/.pyenv/bin:/Users/foo/.rbenv/shims:/Users/foo/.pyenv/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/foo/.asdf/shims:/Users/foo/.asdf/shims
  • SHELL: /bin/zsh
  • LANG: en_US.UTF-8

Contents of Pipfile (‘/Users/foo/workspace/foo-test-cryptograph-all-platform-hashes/Pipfile’):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.10"

[packages]
cryptography = "*"

Contents of Pipfile.lock (‘/Users/foo/workspace/foo-test-cryptograph-all-platform-hashes/Pipfile.lock’):

{
    "_meta": {
        "hash": {
            "sha256": "90025a8184f3365cfb0e9f07920c49ce086e43ac5335361e1ca7a562e9ab9e30"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "cffi": {
            "hashes": [
                "sha256:39d39875251ca8f612b6f33e6b1195af86d1b3e60086068be9cc053aa4376e21",
                "sha256:d400bfb9a37b1351253cb402671cea7e89bdecc294e8016a707f6d1d8ac934f9"
            ],
            "version": "==1.15.1"
        },
        "cryptography": {
            "hashes": [
                "sha256:652627a055cb52a84f8c448185922241dd5217443ca194d5739b44612c5e6507",
                "sha256:6d192741113ef5e30d89dcb5b956ef4e1578f304708701b8b73d38e3e1461f34",
                "sha256:8f09daa483aedea50d249ef98ed500569841d6498aa9c9f4b0531b9964658922"
            ],
            "index": "pypi",
            "version": "==41.0.3"
        },
        "pycparser": {
            "hashes": [
                "sha256:8ee45429555515e1f6b185e78100aea234072576aa43ab53aefcae078162fca9",
                "sha256:e644fdec12f7872f86c58ff790da456218b10f863970249516d60a5eaca77206"
            ],
            "version": "==2.21"
        }
    },
    "develop": {}
}

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 21 (10 by maintainers)

Most upvoted comments

2023.9.7 included this change.

+1
matteius on Sep 7, 2023

@jermbop I may have some time next weekend given its calling for rain here. If you click on my github profile, I have my email you can send me and I can invite you to a python developers slack group …

+1
matteius on Aug 22, 2023

@matteius Just tested with 2023.8.19. I’m still seeing the same issue. I’m open for a debugging session!

+1
jermbop on Aug 21, 2023

@jermbop thats a different docs link, I think it would be this one: https://pipenv.pypa.io/en/latest/indexes/#using-a-pypi-mirror

To my understanding I can either run pipenv install --pypi-mirror <mirror_url> or set PIPENV_PYPI_MIRROR.

Yes that is correct, but my suspicion is that is also where there may still be a bug. Since I do not have access to your private pypi mirror, its harder for me to test this which is why I am asking if you can try to not use the pypi-mirror in testing my branch and instead change the Pipfile or lockfile source URL to be the mirror and run a couple checks. This will help me determine:

  1. Is it just a bug with pypi-mirror (ok we can look more deeply into that if so) OR
  2. Is it perhaps something isn’t matching when we scrape the index for the hashes.

On my branch there should be good output with -v or --verbose that should show where its trying to obtain the hashes, if there is an error there that would also be useful to know, or maybe its skipping that because the source isn’t in the lockfile directly.

Its not really going to be too useful for me to test on main/current release because my future looking branch is likely to get merged soon, and so I’ll want to find the path forward from starting from that code. Nothing stood out to me comparing 2023.7.11 to 2023.7.9 anyway and plus a lot more fixes in the draft-no-reqlib branch including the fix from today that I verified against the pytorch index.

Just to reiterate, I think there is a bug and it should be fixed, but it somewhat non-trivial and so some help pin-pointing it will help me do a better job with it.

+1
matteius on Aug 17, 2023

I am also seeing this with version 2023.7.23 while using the extra index https://download.pytorch.org/whl/cpu to install torch. Downgrading to version 2023.7.9 has fixed the problem.

+1
boonware on Aug 15, 2023
Read more comments on GitHub
← pipenv: Release v2023.6.26: error: invalid command 'egg_info'
pipenv: Running `pipenv --completion` is very slow →