airflow: Import error when using custom backend and sql_alchemy_conn_secret

Apache Airflow version: 2.0.0

Environment:

• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release): custom Docker image (FROM python:3.6) and macOS Big Sur (11.0.1)
• Kernel (e.g. uname -a):
  • Linux xxx 4.14.174+ #1 SMP x86_64 GNU/Linux
  • Darwin xxx 20.1.0 Darwin Kernel Version 20.1.0 rRELEASE_X86_64 x86_64
• Install tools:
• Others:

What happened:

I may have mixed 2 different issues here, but this is what happened to me.

I’m trying to use Airflow with the airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend and a sql_alchemy_conn_secret too, however, I have a NameError exception when attempting to run either airflow scheduler or airflow webserver:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/airflow/__init__.py", line 34, in <module>
    from airflow import settings
  File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 35, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 786, in <module>
    conf.read(AIRFLOW_CONFIG)
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 447, in read
    self._validate()
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 196, in _validate
    self._validate_config_dependencies()
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 224, in _validate_config_dependencies
    is_sqlite = "sqlite" in self.get('core', 'sql_alchemy_conn')
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 324, in get
    option = self._get_option_from_secrets(deprecated_key, deprecated_section, key, section)
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 342, in _get_option_from_secrets
    option = self._get_secret_option(section, key)
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 303, in _get_secret_option
    return _get_config_value_from_secret_backend(secrets_path)
NameError: name '_get_config_value_from_secret_backend' is not defined

What you expected to happen:

A proper import and configuration creation.

How to reproduce it:

airflow.cfg:

[core]

# ...

sql_alchemy_conn_secret = some-key

# ...

[secrets]

backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend

backend_kwargs = { ... }

# ...

Anything else we need to know:

Here is the workaround I have for the moment, not sure it works all the way, and probably doesn’t cover all edge cases, tho it kinda works for my setup:

Move get_custom_secret_backend before (for me it’s actually below _get_config_value_from_secret_backend): https://github.com/apache/airflow/blob/cc87caa0ce0b31aa29df7bbe90bdcc2426d80ff1/airflow/configuration.py#L794

Then comment: https://github.com/apache/airflow/blob/cc87caa0ce0b31aa29df7bbe90bdcc2426d80ff1/airflow/configuration.py#L232-L236