pipenv: Private packages not updated unless environment is removed

Issue description

Private packages defined with an object like mylib = {ref = "dev" git = "ssh://git.mycompany.com/my-lib.git"} are not being updated on my virtual env, even after the pipenv update creates a new Pipfile.lock with the correct commit id.

Expected result

After calling pipenv update, I expect the Pipfile.lock ref to change and also the code in my environment to change, such that I have access to the new code within my project.

Actual result

I see the Pipfile.lock ref change, but the actual code for my dependency does not change.

If I do pipenv --rm to remove the environment and then re-create the environment, I can get the actual code change to replicate on disk.

Steps to replicate

Update my-lib with some code change, commit and push. On my project, do pipenv update and wait for the process to finish.

Check that the Pipfile.lock has changed and has the correct ref for the new commit in my-lib.

Check that the code for my-lib hasnt actually changed within my original projects virtual env.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 24 (12 by maintainers)

Most upvoted comments

@joaomcarlos 2023.8.19 has been released šŸ¤ž

@myii some combination of 1&3 sounds good to me!

@matteius OK, I’ll share my findings here first, then I’ll submit a PR depending on your feedback.


TL;DR – A quick workaround that I’ve been using

In case @joaomcarlos and/or @AndreGraca98 are still affected by this bug and want to try out a workaround, without trawling through the extended discussion below.

Based on the latest release (v2024.0.1):

https://github.com/pypa/pipenv/blob/7493d181b2ede1bf5820189c128020d156cb51f2/pipenv/environment.py#L731-L735

Add the following two lines:

         if match is not None:
+            if (req.link and req.link.is_vcs):
+                return False
             if req.specifier is not None:
                 return SpecifierSet(str(req.specifier)).contains(
                     match.version, prereleases=True
                 )

Identifying the regression

v2023.9.8 was the last working version for me, so I used that to help narrow down my git bisect range. I found that the regression was introduced in:

  • Commit 85c1ba5234ff7228cc64cb18c08f8b8fd9fd3e23 from PR #5951.

Running interactive debugging, I could see that my VCS-based entries were entering this if clause that had been moved to the top and were now returning True:

https://github.com/pypa/pipenv/blob/85c1ba5234ff7228cc64cb18c08f8b8fd9fd3e23/pipenv/environment.py#L772-L775

I.e. These entries were no longer getting to the specific conditional they were supposed to reach.


Hitting a traceback with the first attempted workaround

The obvious workaround to attempt first was to bring the specific conditional back to the top. So based on the current latest commit in the main branch, I’m referring to this block:

https://github.com/pypa/pipenv/blob/b99ea6bcf8149312b1ac43de6679d4c801299029/pipenv/environment.py#L746-L750

However, this led to a traceback due to the match.has_metadata("direct_url.json") clause:

$ pipenv sync
Loading .env environment variables...
Installing dependencies from Pipfile.lock (89ef39)...
Traceback (most recent call last):
  File ".../.local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File ".../pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File ".../pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File ".../pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File ".../pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ".../pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ".../pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File ".../pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File ".../pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File ".../pipenv/vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File ".../pipenv/cli/command.py", line 651, in sync
    retcode = do_sync(
  File ".../pipenv/routines/install.py", line 339, in do_sync
    do_init(
  File ".../pipenv/routines/install.py", line 680, in do_init
    do_install_dependencies(
  File ".../pipenv/routines/install.py", line 438, in do_install_dependencies
    batch_install(
  File ".../pipenv/routines/install.py", line 503, in batch_install
    deps_to_install = [
  File ".../pipenv/routines/install.py", line 506, in <listcomp>
    if not project.environment.is_satisfied(dep)
  File ".../pipenv/environment.py", line 734, in is_satisfied
    if match.has_metadata("direct_url.json") or (req.link and req.link.is_vcs):
AttributeError: 'PathDistribution' object has no attribute 'has_metadata'

With a bit more digging, I found that this was something that slipped through during the pkg_resources => importlib-metadata conversion in b1c14a2e18e90b3b77467d0fb19b70ce410be396.


Finalising the workaround

Not to be presumptuous about the correct fix for this, I expunged the clause mercilessly! This led to the fuller form of the workaround, which may be the basis of a potential fix:

         if match is not None:
+            if (req.link and req.link.is_vcs):
+                # VCS installs we assume are not satisfied since due to
+                # skip-lock we may be installing from Pipfile we have insufficient
+                # information to determine if a branch or ref has actually changed.
+                return False
             if req.specifier is not None:
                 return SpecifierSet(str(req.specifier)).contains(
                     match.version, prereleases=True
                 )
             if req.link is None:
                 return True
             elif req.editable and req.link.is_file:
                 requested_path = req.link.file_path
                 if os.path.exists(requested_path):
                     local_path = requested_path
                 else:
                     parsed_url = urlparse(requested_path)
                     local_path = parsed_url.path
                 return requested_path and os.path.samefile(local_path, match.location)
-            elif match.has_metadata("direct_url.json") or (req.link and req.link.is_vcs):
-                # Direct URL installs and VCS installs we assume are not satisfied
-                # since due to skip-lock we may be installing from Pipfile we have insufficient
-                # information to determine if a branch or ref has actually changed.
-                return False
             return True
         return False

@matteius all is gut šŸ˜„ No rush. I hope your holiday was good!

I saw, just don’t have time at the moment–getting back to work after two weeks off šŸ˜…

Right, we tested it and we saw the same behavior.

Lets re-cap:

We ran, on me and Andres machine, the following two commands:

pip install -U git+https://github.com/pypa/pipenv.git@1afd341a9318f011ac31db3c5367058d65aa4db
cd [into our project]
pipenv --version 
pipenv, version 2023.9.8
pipenv install git+https://github.com/pypa/pipenv.git@1afd341a9318f011ac31db3c5367058d65aa4db
pipenv --version
pipenv, version 2023.9.9.dev0

At this point, we established a baseline for pipenv.

We commit a ā€œpotatoes 2ā€ print to our package on the dev branch and run pipenv update, we see that the commit ref has been updated successfully but no change in the files under our virtual env specifically (/Users/joao.carlos/.local/share/virtualenvs/package-5xV1os6e/src/package-name/module/file.py

We tried a bunch of variations and none worked.

We did however manage to get it updating by adding the editable=true to the Pipfile.

With editable=true, we see both the change in ref and the change in the file.

Does this help with debugging the issue?

I saw your reply, but I cannot reproduce it, so leaving it open for now in case someone can help you.

Hi,

This is actually still happening. The refs get updated on the lock file but the code does not change until you --rm it