pip: pip 10: "pip list" warns "Cache entry deserialization failed, entry ignored"

It seems like you have messed up your pip cache. It’s stored in ~/.cache/pip.

If you delete that directory, this issue should be fixed.

I suspect that the problem is that cache entries created in Python 2 cannot be deserialized in Python 3. If I use the --cache-dir option to force different caches for Python 2 and Python 3, the problem goes away.

The error message is generated from C:\Python36\Lib\site-packages\pip_vendor\cachecontrol\controller.py line 139. The proximate cause is a failure of Serializer.loads() in C:\Python36\Lib\site-packages\pip_vendor\cachecontrol\serialize.py line 75.

I have also noted the same problem for the 64-bit versions of Windows 7 and 10 Professional. I see the problem only for Python 3.5 and not Python 2.7 or Python 3.6 (I have separate installations of all 3 versions). Adding --format legacy or --no-cache-dir makes the problem go away.

pip will absorb that change when we update our vendored dependencies in preparation for the next release. 😃

@pradyunsg: The problem is not fixed by deleting the cache dir (see the comment of @japagetw).

AFAIU, using --no-cache-dir on Python 3 is only a temporary workaround for this (i.e., it is a bug).

Can we make the shared cache dir for Python 2 and 3 work again (or is there a reason why the same version of pip >= 10 needs to use a different cache format on PY2 vs. PY3)?

https://github.com/pypa/pip/issues/5250#issuecomment-394706450 elaborates on why it failed.

In essence, the error occurs due to a mismatch when converting an http request into a string (serialization). This error/warning shows up when trying to convert that string into an http request again (deserialization).

@HakShak The fact stderr is used is a good thing. POSIX specification documents and requires that all logging messages would go to stderr and not stdout, including errors, warnings, info, debug. Only the normal program output should go to stdin. This usually mean, for most tools: computer parsable output. For example on a package manager stdout should contain only installed/removed/upgraded packages, without the boilerplate which should go to stderr. I remember fixing few non compliant tools over the years. The output stream is not the issue here.

On the other hand, I just observed that this bug was added to the 18.0 milestone, not sure how to take this, as current version is 10.0. Is like a very long time in the future, or maybe I don’t know about a plan to boost pip version to match current year?

Thank you. This phenomenon reproduces. I have confirmed that pip3 list --no-cache-dir --outdated works without warnings.

Temporary Fix upgrade pip3 using.

curl https://bootstrap.pypa.io/get-pip.py | python3

Instead of pip install -U pip

For pip2 pip2 install --upgrade pip

Easy to resolve this problem, run app or terminal as administrator

Upstream PR https://github.com/ionrock/cachecontrol/pull/190 is merged and released in CacheControl 0.12.5.

Upstream PR: ionrock/cachecontrol#190

I found that on Python 2, pip is able to use a cache populated by a Python 3 run but not vice versa.

A response cached on Py2 results in the cache storing a header that loads on Py3 as bytes instead of str. This causes the deserialization to fail. Working on a PR to fix it upstream.

Is like a very long time in the future, or maybe I don’t know about a plan to boost pip version to match current year?

We switched to CalVer (#5324), 18.0 is going to be the next release of pip, likely next month.

Is anyone planning to do something about this newly introduced bug? pip should be updated ASAP to avoid clashing incompatible caches.

The workarounds mentioned here are not really applicable in all cases and are likely to cause more problems than sorting. Adding a versioning part in the cache entry naming would address this issue once for all, even if the cache format would change again.

Potentially every python package developer would encounter this issue if they use tox to test with multiple python versions. Please don’t ask them to isolate the caches for each tox target, this defeats the purpose of having a cache.