pipenv: Using: keyrings.google-artifactregistry-auth pipenv update -d fails hard.

Be sure to check the existing issues (both open and closed!), and make sure you are running the latest version of Pipenv.

Check the diagnose documentation for common issues before posting! We may close your issue if it is very similar to one of them. Please be considerate, or be on your way.

Make sure to mention your debugging experience if the documented solution failed.

Issue description

Google Artifact Repositories use keyrings with their own addon to make git and twine work perfectly, but when using pipenv it fails hard to a point where we cant use pipenv.

Note I also tried with 20.11.15 with same result

Expected result

pipenv update -d should update dependencies and make an update Pipfile.lock

Actual result

(applied_mapper) klaus@new-server ~/src/applied_mapper master> pipenv update -d -v Running $ pipenv lock then $ pipenv sync. Locking [dev-packages] dependencies… Building requirements… Resolving dependencies… ⠧ Locking… ROUND 1
✘ Locking Failed! Current constraints: aadomain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 10)) applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14)) applied-domain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 9)) geojson (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 13)) keyring (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 7)) keyrings.google-artifactregistry-auth (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 15)) pbr (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 5)) pylint (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 4)) pytest (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 8)) pytest-flake8 (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 6)) python-dateutil (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 12)) strenum (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 11))

Finding the best candidates: found candidate aadomain==1.4.42 (constraint was <any>) Traceback (most recent call last): File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py”, line 808, in resolve results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS) File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py”, line 180, in resolve has_changed, best_matches = self._resolve_one_round() File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py”, line 260, in _resolve_one_round best_matches = {self.get_best_match(ireq) for ireq in constraints} File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py”, line 260, in <setcomp> best_matches = {self.get_best_match(ireq) for ireq in constraints} File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py”, line 319, in get_best_match best_match = self.repository.find_best_match( File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py”, line 202, in find_best_match raise NoCandidateFound(ireq, all_candidates, self.finder) pipenv.patched.piptools.exceptions.NoCandidateFound: Could not find a version that matches applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14)) No versions found Were https://europe-north1-pypi.pkg.dev/new-server/pypi/simple or https://legacy-pypi.appliedautonomy.no:8443/simple or https://pypi.org/simple reachable?

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py”, line 807, in <module> main() File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py”, line 802, in main _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write, File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py”, line 785, in _main resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages) File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py”, line 746, in resolve_packages results, resolver = resolve( File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/resolver.py”, line 728, in resolve return resolve_deps( File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py”, line 1378, in resolve_deps results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps( File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py”, line 1093, in actually_resolve_deps resolver.resolve() File “/home/klaus/.local/lib/python3.8/site-packages/pipenv/utils.py”, line 818, in resolve raise ResolutionFailure(message=str(e)) pipenv.exceptions.ResolutionFailure: ERROR: Could not find a version that matches applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14)) No versions found Were https://europe-north1-pypi.pkg.dev/new-server/pypi/simple or https://legacy-pypi.appliedautonomy.no:8443/simple or https://pypi.org/simple reachable? ROUND 1
Current constraints: aadomain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 10)) applied-core (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 14)) applied-domain (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 9)) geojson (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 13)) keyring (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 7)) keyrings.google-artifactregistry-auth (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 15)) pbr (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 5)) pylint (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 4)) pytest (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 8)) pytest-flake8 (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 6)) python-dateutil (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 12)) strenum (from -r /tmp/pipenvg2_zajwhrequirements/pipenv-2_qs5aav-constraints.txt (line 11))

During handling of the above exception, another exception occurred:

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

Its hard to replicate as you need alpha access to artifact repo pypi, but using a Pipfile like this: [[source]] name = “applied-backend” url = “https://europe-north1-pypi.pkg.dev/new-server/pypi/simple” verify_ssl = true

[[source]] name = “applied” url = “https://legacy-pypi.appliedautonomy.no:8443/simple” verify_ssl = true

[[source]] name = “pypi” url = “https://pypi.org/simple” verify_ssl = true

[dev-packages] keyring = “" “keyrings.google-artifactregistry-auth” = "” pytest = “" pytest-flake8 = "” pbr = “" pylint = "” StrEnum = “" python-dateutil = "” geojson = “" aadomain = "” applied-core = “" applied-domain = "”

[packages]

[requires] python_version = “3.9”

then running pipenv update -d you will get the hard fail listed above. I do believe its connected to the keyring as it works as a charm in pip without any drama.

Please run $ pipenv --support, and paste the results here. Don’t put backticks (`) around it! The output already contains Markdown formatting.

$ pipenv --support

Pipenv version: '2020.8.13'

Pipenv location: '/home/klaus/.local/lib/python3.8/site-packages/pipenv'

Python location: '/usr/bin/python3'

Python installations found:

3.9.0: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python3.9
3.9.0: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python3
3.9.0: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python3.9
3.9.0: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin/python3
3.9.0: /usr/bin/python3.9
3.9.0: /bin/python3.9
3.8.5: /usr/bin/python3.8
3.8.5: /usr/bin/python3
3.8.5: /bin/python3.8
3.8.5: /bin/python3
2.7.18: /usr/bin/python2
2.7.18: /usr/bin/python2.7
2.7.18: /bin/python2
2.7.18: /bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.4.0-73-generic',
 'platform_system': 'Linux',
 'platform_version': '#82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021',
 'python_full_version': '3.8.5',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

GJS_DEBUG_TOPICS
SSH_AUTH_SOCK
SESSION_MANAGER
GNOME_TERMINAL_SCREEN
SSH_AGENT_PID
XDG_CURRENT_DESKTOP
LANG
LC_IDENTIFICATION
DEFAULTS_PATH
XDG_SESSION_CLASS
COLORTERM
LIBVIRT_DEFAULT_URI
GPG_AGENT_INFO
DESKTOP_SESSION
GJS_DEBUG_OUTPUT
XDG_MENU_PREFIX
USER
QT_IM_MODULE
LC_MEASUREMENT
VTE_VERSION
DBUS_SESSION_BUS_ADDRESS
PWD
LC_NUMERIC
GTK_MODULES
_
WINDOWPATH
XDG_SESSION_DESKTOP
JOURNAL_STREAM
QT_ACCESSIBILITY
HOME
GNOME_DESKTOP_SESSION_ID
MANAGERPID
LC_TIME
XDG_DATA_DIRS
GNOME_TERMINAL_SERVICE
LC_PAPER
LOGNAME
MANDATORY_PATH
XDG_RUNTIME_DIR
XDG_CONFIG_DIRS
XDG_SESSION_TYPE
XMODIFIERS
PATH
LC_TELEPHONE
LC_MONETARY
SHELL
GNOME_SHELL_SESSION_MODE
USERNAME
INVOCATION_ID
SHLVL
XAUTHORITY
LC_NAME
IM_CONFIG_PHASE
TERM
LC_ADDRESS
DISPLAY
GDMSESSION
ZSH
PAGER
LESS
LSCOLORS
LS_COLORS
CLOUDSDK_HOME
APPLIED_SCRIPT_HOME
GOOGLE_APPLICATION_CREDENTIALS
GOOGLE_ACCOUNT_SERVICE_FILE
NVM_DIR
NVM_CD_FLAGS
NVM_BIN
NVM_INC
pipfile_dir
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH
PIPENV_ACTIVE
VIRTUAL_ENV
PS1
PIP_SHIMS_BASE_MODULE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

PIPENV_ACTIVE: 1

Debug–specific environment variables:

PATH: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4/bin:/home/klaus/bin:/opt/ghc/bin:/home/klaus/.nvm/versions/node/v15.1.0/bin:/home/klaus/bin:/opt/ghc/bin:/home/klaus/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL: /usr/bin/zsh
LANG: en_US.UTF-8
PWD: /home/klaus/src/applied_mapper
VIRTUAL_ENV: /home/klaus/.local/share/virtualenvs/applied_mapper-zaPHu-S4

Contents of Pipfile (‘/home/klaus/src/applied_mapper/Pipfile’):

[[source]]
name = "applied-backend"
url = "https://europe-north1-pypi.pkg.dev/new-server/pypi/simple"
verify_ssl = true

[[source]]
name = "applied"
url = "https://$legacy-pypi.appliedautonomy.no:8443/simple"
verify_ssl = true

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
keyring = "*"
"keyrings.google-artifactregistry-auth" = "*"
pytest = "*"
pytest-flake8 = "*"
pbr = "*"
pylint = "*"
StrEnum = "*"
python-dateutil = "*"
geojson = "*"
aadomain = "*"
applied-core = "*"
applied-domain = "*"

[packages]

[requires]
python_version = "3.9"

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 2
Comments: 33 (15 by maintainers)

Most upvoted comments

Its my dream to have pipenv+artifact-registry work nicely together (leveraging keychain for secure credential management).

Therin lies a world of rainbows and unicorns.

+1 on fixing this problem. I unfortunately lack the time to dig into this but if somebody could gat a fix in they would have my eternal gratitude.

+10

dhendry on Nov 11, 2021

Just to make it easier for future readers. This kind of a Pipfile works for me on a private Google artifact repository. You also need to have your authentication set up correctly.

[[source]]
url = "https://oauth2accesstoken@us-central1-python.pkg.dev/PROJECT_ID/REPOSITORY_NAME/simple"
verify_ssl = true
name = "private-artifacts"

[packages]
PRIVATE_LIBRARY_NAME = {version="==1.0.0", index="python-artifacts"}

[pipenv]
disable_pip_input = false

So basically two features are needed:

The private URL needs to contain oauth2accesstoken@
disable_pip_input = false must be present

I have pipenv version 2023.4.20.

kristjanveeve on May 26, 2023

From your comment @matteius, it sounds like the intended behavior and usage pattern is that the keyring packages be installed inside of the project venv, and pipenv leverages those within-venv packages to perform installs of other packages?

That is the only way Pip, and thus Pipenv, was able to use keyring. This is no longer the only way.

You can install keyring anywhere
Make sure it is found on the PATH
Make sure virtualenv seeds new venvs with Pip 23.1 or higher. (run virtualenv --upgrade-embed-wheels if that is not the case, or look at the output of virtualenv --help if that doesn’t cause the version to change.)
Run pip config set --global global.keyring-provider subprocess (or use --user if you prefer)
Make sure the index has a username that is correct for the keyring backend
- oauth2accesstoken for keyrings.google-artifactregistry-auth
- VssSessionToken for artifacts-keyring

I wonder if it makes sense to have something like an install virtualenv, akin to how python packaging has the “build” environment, where packages that pipenv relies on to execute the installation process itself can be installed and walled off from the actual project virtualenv.

With the above you configure Pip. Pipenv then requires the right index urls.

So improving the documentation seems like the better way to go about this. If you disagree, let me introduce you to virtualenv’s seeder concept. My keyring-subprocess package provides a seeder which seeds itself into new virtual environments.

PS. I contributed the --keyring-provider flag to Pip, eventually new Python versions ship with a wheel of Pip greater than or equal to 23.1 in their ensurepip module. Then Python’s venv module will also create virtual environments with a Pip version >=32.1…

Darsstar on Sep 15, 2023

I’m trying to get a better understanding of how pipenv integrates with the keyring - I’ve got it working a little bit similarly to how kristjanveeve has it, but I’ve noticed some weird behavior:

My setup:

I use pyenv and pipx.
I installed pipenv using pipx install pipenv.

I have a project with a Pipfile that looks like this:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://oauth2accesstoken@mygar/index/simple/"
verify_ssl = true
name = "gar"

[pipenv]
disable_pip_input = false

[packages]
...

[dev-packages]
...

[requires]
python_version = "3.11.3"

A few things I’ve noticed:

pipenv lock seems to work for me, but only when I have keyring and keyrings.google-artifactregistry-auth installed in the project virtualenv. I have tried these other combinations:
- keyring pkgs installed in the pipenv virtualenv (managed by pipx) but not the project virtualenv.
- keyring pkgs installed in neither virtualenv
- keyring pkgs installed in both the pipenv and project virtualenvs.

The two configurations where the keyring packages aren’t installed in the project virtualenv result in the following error when I attempt to run pipenv lock: CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement MY_INTERNAL_LIB (from versions: none). I find the fact that it’s using the project’s virtualenv quite surprising, and I think clearly means that there’s something I don’t understand about how pipenv attempts package resolution…

Additionally, I’ve tested this on the following pipenv versions:

2023.3.20
2023.4.20
2023.4.29
2023.5.19

When I run on 2023.5.19, I start getting the following error from the google artifact keyring ERROR:keyring.backend:Error initializing plugin EntryPoint(name='Google Auth', value='keyrings.gauth', group='keyring.backends'). ... ModuleNotFoundError: No module named 'requests.packages.urllib3'; 'requests.packages' is not a package.

I’m not totally sure if this is a pipenv bug, but I’m a little bit confused about where to start understanding this, since this is the keyring library version in my project virtualenv.

Lastly, I’m not entirely sure what Darsstar meant above, but

[pipenv]
disable_pip_input = false

does seem to still be necessary for me, in the configurations that work at all.

I think my overall question is: is there an “expected” usage pattern for keyring integration with pipenv for private package registries? e.g. should the keyring packages be installed adjacent to pipenv, or inside of the project venv? Should disable_pip_input be used or not?

I think with a better understanding of how I “should” use these, I can better hone in on what’s actually going wrong in my environment.

Thanks!

justin-yan on May 25, 2023

pipenv 2023.4.20 vendors pip 23.1. pip 23.1 contains https://github.com/pypa/pip/pull/11698.

While disable_pip_input = false no longer works, it doesn’t need to. https://pip.pypa.io/en/stable/topics/authentication/#using-keyring-as-a-command-line-application contains instructions for how to set up keyring so it can be used by pip 23.1. The source url MUST contain a username. (https://dummyusername@example.com/pypi/simple for example) Azure DevOps/artifacts-keyring users, you MUST use VssSessionToken as the username. I suspect Google Artifact users can use any value for the username you wish.

I hope this is helpfull to people.

Darsstar on Apr 20, 2023

I experienced the same issue and noticed that pipenv is actually able to download packages from Google Artifact Registry, and only failing when updating the lockfile. You can work around this by editing the lockfile file manually, or by doing something like this:

$ pip download --extra-index-url <URL_TO_ARTIFACT_REG> <YOUR_PACKAGE>
$ pipenv install *.whl

This will update the lockfile, but also modify your Pipfile to point towards local archives. Undo the changes to the Pipfile and it will work as expected.

I’m trying to figure out how to properly fix this, I’ll reach out if I find anything.

romhml on Jun 23, 2021

Thanks @justin-yan your comment was really insightful:

pipenv lock seems to work for me, but only when I have keyring and keyrings.google-artifactregistry-auth installed in the project virtualenv.

That was a missing piece for me to get up an running to debug these issues fruther.

pipenv run pip install keyring
pipenv run pip install keyrings.google-artifactregistry-auth

It seems the next issue is that in 2023.5.19 I had to reintroduce this: https://github.com/pypa/pipenv/blob/main/pipenv/__init__.py#L24

Which has the effect of causing the pip requests to be the first one that is imported, and it does not have a requests.packages.urllib3 – when I change these lines locally to .append() instead of insert in the front, it has the effect of allowing the requests installed in the virtualenv to be found first, which has the required imports for keyring – keyrings.google-artifactregistry-auth. The disadvantage to making that change is something in the virtualenv that could be older than whats in the vendor packages could take precedence and cause a different side effect in pipenv. I’ll have to think more about how to move forward with this.

matteius on Jun 11, 2023

@Darsstar’s comment nudged me in the right direction (thanks !). As I mentionned in this other issue I face the same situation with Google Artifact and the username must be set to oauth2accesstoken.

just-maiyak on May 5, 2023

After some trial and error, I found that you can specify dependencies packaged through setup.cfg in your Pipfile as well. That would allow you to specify the index for those dependencies. For example:

setup.cfg:

install_requires =
  my-library = x.y.z

Pipfile:

[packages]
my-library = {index = "my-pypi"}
my-application = ""

pipenv install then installs my-library at the pinned version, i.e. x.y.z. In addition, building and installing the my-application package installs my-library as intended.

savitha-atg on Apr 21, 2022

@kylebluenote There is no current way to specify the index in the setup.cfg or pyproject.toml or setup.py – as far as I know, it is only a pip and pipenv feature today to specify the indexes. For your case the recommendation would be to mirror the pypi packages to your local pypi server and have that be the default index. All packages that are unspecified try to resolve only to the default index. We are working on a feature that would allow installation from multiple sources again as it was before since they have the hashes properly sources in the lock file already, but for locking resolution to work it needs to not guess what index to pull the package from.

matteius on Apr 20, 2022

@matteius That worked!

Would you be able to release this fix today? I’d rather install an official pipenv. Would really appreciate it! Thank you!

savitha-atg on Apr 20, 2022

@Darsstar Feel free to open a PR about it, I am admittedly a bit rusty on how the pip arguments get supported in pipenv, and I seem recall discussions on old issues about it being challenging to support all the arguments. Perhaps there is a way though with environment variables to specify pip arguments such that we don’t need to add an explicit flag within pipenv for it? Sorry, I am a bit tied up at the moment to really research this further today but definitely see what you can figure out and it will get us thinking more about it.

matteius on Apr 20, 2022