airflow: ImportError: cannot import name 'bytes_type' from 'oauthlib.common'

Apache Airflow version: 1.10.10 Environment: Airflow ECS

Cloud provider or hardware configuration: Running on Docker
OS (e.g. from /etc/os-release): AWS Linux 2
Kernel (e.g. uname -a): Linux
Install tools:
Others:

I have made some change in Dockerfile and added these lines

ARG WEBSERVER_CONFIG="default_webserver_config.py"
ENV WEBSERVER_CONFIG=${WEBSERVER_CONFIG}

COPY --chown=airflow:airflow "./airflow/config_templates/${WEBSERVER_CONFIG}" ${AIRFLOW_HOME}/webserver_config.py

ARG AIRFLOW_CFG="default_airflow.cfg"
ENV AIRFLOW_CFG=${AIRFLOW_CFG}

COPY --chown=airflow:airflow "./airflow/config_templates/${AIRFLOW_CFG}" ${AIRFLOW_HOME}/airflow.cfg

What happened I am updating airflow from 1.10.5 to 1.10.10. I am getting the following error in webserver when using google OAuth.

Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 37, in <module>
    args.func(args)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 75, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/bin/cli.py", line 900, in webserver
    app = cached_app_rbac(None) if settings.RBAC else cached_app(None)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 295, in cached_app
    app, _ = create_app(config, session, testing)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 108, in create_app
    update_perms=conf.getboolean('webserver', 'UPDATE_FAB_PERMS'))
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/base.py", line 148, in __init__
    self.init_app(app, session)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/base.py", line 202, in init_app
    self.sm = self.security_manager_class(self)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/security.py", line 177, in __init__
    super(AirflowSecurityManager, self).__init__(appbuilder)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/security/sqla/manager.py", line 51, in __init__
    super(SecurityManager, self).__init__(appbuilder)
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_appbuilder/security/manager.py", line 249, in __init__
    from flask_oauthlib.client import OAuth
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_oauthlib/client.py", line 20, in <module>
    from .utils import to_bytes
  File "/home/airflow/.local/lib/python3.7/site-packages/flask_oauthlib/utils.py", line 5, in <module>
    from oauthlib.common import to_unicode, bytes_type
ImportError: cannot import name 'bytes_type' from 'oauthlib.common' (/home/airflow/.local/lib/python3.7/site-packages/oauthlib/common.py)

This error is because of the dependency mismatch. Airflow 1.10.10 is installing oauthlib==3.1.0 and Flask-OAuthlib==0.9.5 and in 3.1.0 version of oauthlib, there is no bytes_type while Flash-OAuthlib is expecting it to be there.

Considering that lepture has archived flask-oauthlib and created authlib, I see a PR in airflow from lepture about this change https://github.com/apache/airflow/pull/6140 but it is not merged and marked stale by the bot

Has anyone faced this issue in Airflow 1.10.10? I have fixed this issue by changing the oauthlib version to 2.1.0 in requirement.txt

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 16
Comments: 24 (7 by maintainers)

Commits related to this issue

Fix flask_oautlib throws error. Fixes #8467 Fixes dependency issue that causes an error when users try to create a new user using the oauth type. — committed to dkinzer/airflow by dkinzer 3 years ago

Most upvoted comments

hey @namiops what version of airflow are you using? and python version?

hi @snigdhasaikam30, I’m using Airfllow 1.10.15 with python 3.8.

namiops on Apr 3, 2021

@snigdhasaikam30 try using --constraint=https://raw.githubusercontent.com/apache/airflow/cc382b75783d76888fcd5b0e38228f08ee83dbd5/constraints-3.6.txt when running the pip install command.

That constraint is the one that is working for me that I have in #14562. I’m using the same python version and airflow version that you are.

This is what worked for us. changed the user to “airflow” instead of root

USER airflow RUN pip install --user -I flask_oauthlib==0.9.6 --constraint=https://raw.githubusercontent.com/apache/airflow/cc382b75783d76888fcd5b0e38228f08ee83dbd5/constraints-3.6.txt

airflow:1.10.15-python3.6

snigdhasaikam30 on Mar 19, 2021

@potiuk, we are seeing the same issue as @weldpua2008 using airflow 1.10.14 and applying the 1.10.14/constraints-3.6.txt.

I’ve determined that the constraints do not specify a version for flask_oauthlib even though they make a direct dependency via the flask_appbuilder package and this is likely at the root of the problem:

Flask-AppBuilder==2.3.4 has a dep to flask_oauthlib but the constraint do not specify this package https://github.com/dpgaspar/Flask-AppBuilder/blob/v2.3.4/flask_appbuilder/security/manager.py#L249

I’ve deleted my local copy of flask_oauthlib and tried to reinstall it using the constraint and I get the following error:

pip install flask_oauthlib --constraint=https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt
flask-oauthlib 0.9.6 has requirement oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2, but you'll have oauthlib 3.1.0 which is incompatible.
flask-oauthlib 0.9.6 has requirement requests-oauthlib<1.2.0,>=0.6.2, but you'll have requests-oauthlib 1.3.0 which is incompatible.

So I think this incompatibility is at the heart of the issue.

dkinzer on Mar 2, 2021