airflow: Airflow installation fails when with latest `setuptools`

Apache Airflow version

2.1.3

Operating System

Ubuntu 20.04

Versions of Apache Airflow Providers

NA

Deployment

Other Docker-based deployment

Deployment details

Ubuntu Docker image, Airflow installed inside a virtualenv in the image.

What happened

Trying to pip-install Airflow in a virtualenv where setuptools is >=58.0.2 fails:

Orchestrator 'airflow' could not be installed: failed to install plugin 'airflow'.
    ERROR: Command errored out with exit status 1:
     command: /project/.meltano/orchestrators/airflow/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-g8rc5i_t/flask-openid_b02636d256a1457daa8e5d35116652ca/setup.py'"'"'; __file__='"'"'/tmp/pip-install-g8rc5i_t/flask-openid_b02636d256a1457daa8e5d35116652ca/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-r8lhwbha
         cwd: /tmp/pip-install-g8rc5i_t/flask-openid_b02636d256a1457daa8e5d35116652ca/
    Complete output (1 lines):
    error in Flask-OpenID setup command: use_2to3 is invalid.

The setuptools team deprecated use_2to3 (see https://github.com/pypa/setuptools/issues/2086) recently and this is breaking installation of packages which rely on that legacy key. In the case of Airflow, Flask-OpenID, required by Flask-AppBuilder, is the offending dependency. See https://github.com/mitsuhiko/flask-openid/blob/v1.2.5/setup.py#L32.

This should probably be fixed by that project’s maintainers, but we (Meltano) thought it was a good idea to file this issue here in case Airflow users run into the problem. Just removing use_2to3 seems to work 😃.

What you expected to happen

Airflow’s dependencies should be prepared for new versions of setuptools and Python (3.10).

How to reproduce

$ python -m venv venv
$ source venv/bin/activate
(venv) $ pip install "setuptools==58.0.2"
(venv) $ pip install apache-airflow
...
Collecting Flask-OpenID<2,>=1.2.5
  Using cached Flask-OpenID-1.2.5.tar.gz (43 kB)
    ERROR: Command errored out with exit status 1:
     command: python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/8b/z9sdckf92_g3hgnh6nv0yhbh0000gn/T/pip-install-wazcqi5t/flask-openid_d71a670325d74cf6b46f7347ab3e06f4/setup.py'"
'"'; __file__='"'"'/private/var/folders/8b/z9sdckf92_g3hgnh6nv0yhbh0000gn/T/pip-install-wazcqi5t/flask-openid_d71a670325d74cf6b46f7347ab3e06f4/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringI
O('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/8b/z9sdckf92_g3hgnh6nv0yhbh0000gn/T/pip-pip-egg-inf
o-on8iyf31
         cwd: /private/var/folders/8b/z9sdckf92_g3hgnh6nv0yhbh0000gn/T/pip-install-wazcqi5t/flask-openid_d71a670325d74cf6b46f7347ab3e06f4/
    Complete output (1 lines):
    error in Flask-OpenID setup command: use_2to3 is invalid

Anything else

The current workaround it to make sure a version of setuptools prior to 58.0.0 is installed in the active Python environment.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project’s Code of Conduct