airflow: PIP 20.3 might break Airflow installation

UPDATE 15.12.2020 6pm CET:

After releasing PIP 20.3.3 today we were able to make 2.0 compatible with the new PIP and 1.10.14 almost works (papermill extra is problematic when installing airflow using the new PIP). We will try to address it in case we release 1.10.15 but if you want to install papermill extra, please downgrade pip or use legacy resolver.

While with 2.0 it seems that airflow can be installed with new PIP following our recommended practice, in case you see any installation problem please report them as issues and downgrade to pip 20.2.4 as a workaround.

Thanks again to the PyPI team for the fast resolution (just in time for the 2.0 release).

We leave the issue open for a while but we updated the description and lowered the priority. We will close it once we have observed installations from our users after 2.0 is released and confirm that the problem is solved for our users.

---

UPDATE 15.12.2020 11am CET:

Seems that with the latest 20.3.3 release and fishing pyarrow dependency we are back in business with 2.0.0rc3.

Once we confirm it and verify 1.10.14 we will be able to close that one!

Thanks to the PYPI team for quick solving it.

---

I am adding this issue to keep track of the on-going problems with new PIP 20.3 released 30th of November.

There are multiple issues with the new PIP that makes it breaks with Airflow’s dependency set.

The first blocking issue is https://github.com/pypa/pip/issues/9203 and https://github.com/pypa/pip/issues/9232.

The latest version of PIP @master is still not usable with Airflow:

• https://github.com/pypa/pip/issues/9297
• https://github.com/pypa/pip/issues/9298
• https://github.com/pypa/pip/issues/9299

Even when those is solved we already know we are affected by a few other problems:

• https://github.com/pypa/pip/issues/9217
• https://github.com/pypa/pip/issues/9215
• https://github.com/pypa/pip/issues/9187
• https://github.com/pypa/pip/issues/9223

We’ve raised the issue to the PIP team and they struggle with fixing a number of other teething problems.

We keep fingers crossed that they will manage to fix the issues promptly and that they will not be overwhelmed with putting out the fire.

There is no resolution yet, so for the time being downgrading PIP to 20.2.4 version is the best thing you can do.

pip install --upgrade pip==20.2.4

We raised the issue https://github.com/pypa/pip/issues/9231 with the proposal of change to PYPI to add an exclusion list to PyPI and we are waiting for their response.

UPDATE! Tested the current master version of PIP (which has been yesterday announced as candidate to 20.3.2) but it still does not solve installation problems with airflow:

Three new issues created:

• https://github.com/pypa/pip/issues/9297
• https://github.com/pypa/pip/issues/9298
• https://github.com/pypa/pip/issues/9299