meltano: bug: Airflow 1.10.x and 2.1.x broken when upgrading to Meltano 2.0+

I’m having issues deploying to production my 2.0.2 version of meltano regarding airflow.

My current version of airflow is 1.10.15 (not 2.X) and everything works locally, but when I push to production I get this error message:

15/06/2022, 11:32:18	[2m2022-06-15T15:32:18.287237Z[0m [[32m[1mdebug [0m] [1mDeleted configuration at /project/.meltano/run/airflow/airflow.cfg[0m	airflow-scheduler
15/06/2022, 11:32:18	[2m2022-06-15T15:32:18.287889Z[0m [[31m[1merror [0m] [1mTraceback (most recent call last):	airflow-scheduler
15/06/2022, 11:32:18	File "/project/.meltano/orchestrators/airflow/venv/bin/airflow", line 25, in <module>	airflow-scheduler
15/06/2022, 11:32:18	from airflow.configuration import conf	airflow-scheduler
15/06/2022, 11:32:18	File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/__init__.py", line 31, in <module>	airflow-scheduler
15/06/2022, 11:32:18	from airflow.utils.log.logging_mixin import LoggingMixin	airflow-scheduler
15/06/2022, 11:32:18	File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/utils/__init__.py", line 24, in <module>	airflow-scheduler
15/06/2022, 11:32:18	from .decorators import apply_defaults as _apply_defaults	airflow-scheduler
15/06/2022, 11:32:18	File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/utils/decorators.py", line 36, in <module>	airflow-scheduler
15/06/2022, 11:32:18	from airflow import settings	airflow-scheduler
15/06/2022, 11:32:18	File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/settings.py", line 38, in <module>	airflow-scheduler
15/06/2022, 11:32:18	from airflow.configuration import conf, AIRFLOW_HOME, WEBSERVER_CONFIG # NOQA F401	airflow-scheduler
15/06/2022, 11:32:18	File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 737, in <module>	airflow-scheduler
15/06/2022, 11:32:18	conf.read(AIRFLOW_CONFIG)	airflow-scheduler
15/06/2022, 11:32:18	File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 425, in read	airflow-scheduler
15/06/2022, 11:32:18	self._validate()	airflow-scheduler
15/06/2022, 11:32:18	File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 217, in _validate	airflow-scheduler
15/06/2022, 11:32:18	self._validate_config_dependencies()	airflow-scheduler
15/06/2022, 11:32:18	File "/project/.meltano/orchestrators/airflow/venv/lib/python3.8/site-packages/airflow/configuration.py", line 249, in _validate_config_dependencies	airflow-scheduler
15/06/2022, 11:32:18	raise AirflowConfigException(	airflow-scheduler
15/06/2022, 11:32:18	airflow.exceptions.AirflowConfigException: error: cannot use sqlite with the LocalExecutor	airflow-scheduler
15/06/2022, 11:32:18	[0m	airflow-scheduler
15/06/2022, 11:32:18	[2m2022-06-15T15:32:18.289815Z[0m [[32m[1mdebug [0m] [1mCommand `airflow --help` failed[0m	airflow-scheduler
15/06/2022, 11:32:18	Traceback (most recent call last):	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/cli/__init__.py", line 55, in _run_cli	airflow-scheduler
15/06/2022, 11:32:18	cli(obj={"project": None})	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__	airflow-scheduler
15/06/2022, 11:32:18	return self.main(*args, **kwargs)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main	airflow-scheduler
15/06/2022, 11:32:18	rv = self.invoke(ctx)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke	airflow-scheduler
15/06/2022, 11:32:18	return _process_result(sub_ctx.command.invoke(sub_ctx))	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke	airflow-scheduler
15/06/2022, 11:32:18	return ctx.invoke(self.callback, **ctx.params)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke	airflow-scheduler
15/06/2022, 11:32:18	return callback(*args, **kwargs)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/cli/params.py", line 23, in decorate	airflow-scheduler
15/06/2022, 11:32:18	return func(*args, **kwargs)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/cli/params.py", line 56, in decorate	airflow-scheduler
15/06/2022, 11:32:18	func(project, *args, **kwargs)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/cli/invoke.py", line 139, in invoke	airflow-scheduler
15/06/2022, 11:32:18	raise invoke_err	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/cli/invoke.py", line 123, in invoke	airflow-scheduler
15/06/2022, 11:32:18	exit_code = run_async(	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/core/utils/__init__.py", line 52, in run_async	airflow-scheduler
15/06/2022, 11:32:18	loop.run_until_complete(future)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete	airflow-scheduler
15/06/2022, 11:32:18	return future.result()	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/cli/invoke.py", line 164, in _invoke	airflow-scheduler
15/06/2022, 11:32:18	async with invoker.prepared(session):	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/async_generator/_util.py", line 34, in __aenter__	airflow-scheduler
15/06/2022, 11:32:18	return await self._agen.asend(None)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/core/plugin_invoker.py", line 248, in prepared	airflow-scheduler
15/06/2022, 11:32:18	await self.prepare(session)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/core/plugin_invoker.py", line 223, in prepare	airflow-scheduler
15/06/2022, 11:32:18	async with self.plugin.trigger_hooks("configure", self, session):	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/async_generator/_util.py", line 34, in __aenter__	airflow-scheduler
15/06/2022, 11:32:18	return await self._agen.asend(None)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/core/behavior/hookable.py", line 87, in trigger_hooks	airflow-scheduler
15/06/2022, 11:32:18	await self.__class__.trigger(self, f"before_{hook_name}", *args, **kwargs)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/core/behavior/hookable.py", line 115, in trigger	airflow-scheduler
15/06/2022, 11:32:18	raise err	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/core/behavior/hookable.py", line 107, in trigger	airflow-scheduler
15/06/2022, 11:32:18	await hook_func(target, *args, **kwargs)	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/core/plugin/airflow.py", line 121, in before_configure	airflow-scheduler
15/06/2022, 11:32:18	raise AsyncSubprocessError(	airflow-scheduler
15/06/2022, 11:32:18	meltano.core.error.AsyncSubprocessError: Command `airflow --help` failed	airflow-scheduler
15/06/2022, 11:32:18	The above exception was the direct cause of the following exception:	airflow-scheduler
15/06/2022, 11:32:18	Traceback (most recent call last):	airflow-scheduler
15/06/2022, 11:32:18	File "/usr/local/lib/python3.8/site-packages/meltano/cli/__init__.py", line 63, in _run_cli	airflow-scheduler
15/06/2022, 11:32:18	raise CliError(str(err)) from err	airflow-scheduler
15/06/2022, 11:32:18	meltano.cli.utils.CliError: Command `airflow --help` failed	airflow-scheduler
15/06/2022, 11:32:18	Command `airflow --help` failed

Here is a link to the complete slack conversation: https://meltano.slack.com/archives/C01TCRBBJD7/p1655303081378359

Has anyone ever seen this error? My first guess is that it’s linked to not being on airflow 2.X, but would appreciate feedback

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (6 by maintainers)

Most upvoted comments

@nickhamlin thanks for the write-up! We do call out the new user creation in https://hub.meltano.com/orchestrators/airflow#next-steps and we have a command specified for it in the form of meltano invoke airflow:create-admin which basically does what you say. Agreed though that the migration story isn’t there yet.

+2
tayloramurphy on Jul 7, 2022

Thanks everyone! Circling back with a final recap from my end now that the dust has started to settle in hopes of leaving some clues for future searchers who might encounter the same issue.

We’ve got prod up and running again by following a version of @SBurwash 's recommendation and quickly migrating to Airflow 2.3.2. This process requires starting from a blank underlying airflow DB since migrating the old one doesn’t seem to work. There wasn’t really much value in retaining the old airflow data, so starting fresh worked fine (once I figured out the right environment variable names to use described in my comment above).

One other gotcha to be aware of in this process is that Airflow 2 requires a manual CLI command to be run to initialize the first admin user. Airflow 1.X didn’t require this step before allowing access to the GUI, which threw me for a loop. Fortunately, it’s straightforward to fix via a command like meltano invoke airflow users create --role Admin --username admin --email admin --firstname admin --lastname admin --password admin. Once that runs, this user can then be used to log into the airflow GUI and additional users can be easily managed from there (including removing this temp admin user for clear security reasons).

That’s all the loose ends I can think of, but I’ll come back and edit this comment if I come across others.

+1
nickhamlin on Jul 7, 2022

@nickhamlin For each of the following scenarios, can you please share: 1. (the airflow parts of) meltano.yml (especially pip_url and config), 2. any environment variables you’re setting, 3. the result of meltano invoke --dump=config airflow (with creds edited out), and 4. the exact error message when you run meltano invoke airflow webserver?

  1. Airflow 2.1.x with AIRFLOW__CORE__SQL_ALCHEMY_CONN
  2. Airflow 2.3.x with AIRFLOW__CORE__SQL_ALCHEMY_CONN
  3. Airflow 2.3.x withAIRFLOW__DATABASE__SQL_ALCHEMY_CONN
+1
DouweM on Jul 6, 2022
Read more comments on GitHub
← melt-ui: Tooltip doesn't work on iPhone
meltano: bug: Disabling telemetry from the UI has no effect →