pip: Infinite loop on pip when lockfile can't acquire a lock

Deleting ~/.cache/pip resolved this issue for me. I had previously tried several different versions/venvs of python3.[4,5,6] and all blocked on mkdirlock.

The bug I think is rather that some earlier usage of pip leaves a lock-file in place? (pip exits uncleanly elsewhere?)

@MinchinWeb Does adding --no-cache-dir to pip list -o --no-cache-dir avoid the issue ?

We’ve fixed this via #4766.

Happened to me just now with 9.0.1 on Mac OS X.

Not sure where the lock file is though to remove it. 😦 (Edit: Removing ~/Library/Caches/pip worked.)

Running with verbose mode on (in a new virtualenv) provides the following:

> pip install sphinx -vvv

Collecting sphinx
  1 location(s) to search for versions of sphinx:
  * https://pypi.python.org/simple/sphinx/
  Getting page https://pypi.python.org/simple/sphinx/
  Looking up "https://pypi.python.org/simple/sphinx/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/sphinx/ HTTP/1.1" 200 5450
  Updating cache with response from "https://pypi.python.org/simple/sphinx/"
  Caching b/c date exists and max-age > 0

[ it stalls here, so I end it with CTRL+C ]

Cleaning up...

Operation cancelled by user

Exception information:
Traceback (most recent call last):
  File "c:\tmp\env\lib\site-packages\pip\_vendor\lockfile\mkdirlockfile.py", line 40, in acquire
    os.mkdir(self.lock_file)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'c:\\users\\william\\appdata\\local\\pip\\cache\\http\\6\\3\\c\\4\\2\\63c426d150091b9663b3d4923999b229a56ede976ab93677382d9562.lock'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\tmp\env\lib\site-packages\pip\basecommand.py", line 209, in main
    status = self.run(options, args)
  File "c:\tmp\env\lib\site-packages\pip\commands\install.py", line 310, in run
    wb.build(autobuilding=True)
  File "c:\tmp\env\lib\site-packages\pip\wheel.py", line 747, in build
    self.requirement_set.prepare_files(self.finder)
  File "c:\tmp\env\lib\site-packages\pip\req\req_set.py", line 359, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "c:\tmp\env\lib\site-packages\pip\req\req_set.py", line 511, in _prepare_file
    finder, self.upgrade, require_hashes)
  File "c:\tmp\env\lib\site-packages\pip\req\req_install.py", line 277, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "c:\tmp\env\lib\site-packages\pip\index.py", line 436, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "c:\tmp\env\lib\site-packages\pip\index.py", line 394, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "c:\tmp\env\lib\site-packages\pip\index.py", line 539, in _get_pages
    page = self._get_page(location)
  File "c:\tmp\env\lib\site-packages\pip\index.py", line 642, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "c:\tmp\env\lib\site-packages\pip\index.py", line 751, in get_page
    "Cache-Control": "max-age=600",
  File "c:\tmp\env\lib\site-packages\pip\_vendor\requests\sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "c:\tmp\env\lib\site-packages\pip\download.py", line 377, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "c:\tmp\env\lib\site-packages\pip\_vendor\requests\sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\tmp\env\lib\site-packages\pip\_vendor\requests\sessions.py", line 608, in send
    r.content
  File "c:\tmp\env\lib\site-packages\pip\_vendor\requests\models.py", line 737, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "c:\tmp\env\lib\site-packages\pip\_vendor\requests\models.py", line 660, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "c:\tmp\env\lib\site-packages\pip\_vendor\requests\packages\urllib3\response.py", line 344, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "c:\tmp\env\lib\site-packages\pip\_vendor\requests\packages\urllib3\response.py", line 301, in read
    data = self._fp.read(amt)
  File "c:\tmp\env\lib\site-packages\pip\_vendor\cachecontrol\filewrapper.py", line 54, in read
    self.__callback(self.__buf.getvalue())
  File "c:\tmp\env\lib\site-packages\pip\_vendor\cachecontrol\controller.py", line 297, in cache_response
    self.serializer.dumps(request, response, body=body),
  File "c:\tmp\env\lib\site-packages\pip\download.py", line 280, in set
    return super(SafeFileCache, self).set(*args, **kwargs)
  File "c:\tmp\env\lib\site-packages\pip\_vendor\cachecontrol\caches\file_cache.py", line 99, in set
    with self.lock_class(name) as lock:
  File "c:\tmp\env\lib\site-packages\pip\_vendor\lockfile\__init__.py", line 197, in __enter__
    self.acquire()
  File "c:\tmp\env\lib\site-packages\pip\_vendor\lockfile\mkdirlockfile.py", line 57, in acquire
    time.sleep(wait)
KeyboardInterrupt

So if I read that right, it can’t create the lock file because one already exists, so it’s waiting for the lock folder to disappear (probably to try again). My guess is this is the infinite loop it’s stuck in.

Deleting the lock folder in question doesn’t seem to help running processes, but does allow a new process to install Sphinx.

Update: Turns out there was another rogue lock folder (f9daaa0b90d3f26ba99d3a4afed1be31abd9eddd0acd969de27db871.lock). Deleting that allows running processes to keep going and finish as expected.

Does there need to be a limit applied to how long it will wait? Should some procedure be put in place to clear out old lock folders? Or at least tell the user which lock folder is holding up pip? As a minimum, if you break out of the pip (without using the verbose options), it should tell you what lock folder it’s hung up on.

It’s a little bit amazing to see the winding, 3 year road from bug report to fix. Keep up the good work!

I had the same issue so to bypass it I had to delete the cache .lock of pip as @MinchinWeb has said or I had to put a return after line 56 in mkdirlockfile.py to avoid the time.sleep function