meltano: Intermittent errors when doing meltano config airflow

Can’t seem to consistently reproduce this.

❯ meltano config airflow set webserver web_server_port 8081
2022-06-09T02:07:03.866027Z [info     ] Environment 'dev' is active
[Errno 2] No such file or directory: '/Users/taylormurphy/Documents/Projects/dev/demoday/prepped/.meltano/run/airflow/airflow.cfg'

but if I run meltano invoke airflow:ui and then kill the process once it’s trying to connect I can successfully run

❯ meltano config airflow set webserver web_server_port 8081
2022-06-09T02:09:10.771279Z [info     ] Environment 'dev' is active
Orchestrator 'airflow' setting 'webserver.web_server_port' was set in the active environment in `meltano.yml`: 8081

so clearly something about the airflow.cfg being present or not.

I would expect to be able to set the config in my meltano.yml even if the airflow.cfg is not present.

It seems like it’s something with invoke because this is what happens if I try to change the port to 8082, then do an invoke, and then reset the port again

❯ meltano config airflow set webserver web_server_port 8082
2022-06-09T02:10:32.255003Z [info     ] Environment 'dev' is active
Orchestrator 'airflow' setting 'webserver.web_server_port' was set in the active environment in `meltano.yml`: 8082
❯ meltano invoke airflow version
2022-06-09T02:10:47.148948Z [info     ] Environment 'dev' is active
2.1.2
❯ meltano config airflow set webserver web_server_port 8082
2022-06-09T02:11:00.416107Z [info     ] Environment 'dev' is active
[Errno 2] No such file or directory: '/Users/taylormurphy/Documents/Projects/dev/demoday/prepped/.meltano/run/airflow/airflow.cfg'

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (12 by maintainers)

Most upvoted comments

@aaronsteers So, yea, this might actually be a really quick fix. I was worried I had added this intentionally for a reason I can’t recall, but I think this just got introduced accidentally when we picked up the community contrib:

https://gitlab.com/meltano/meltano/-/merge_requests/2179/diffs?commit_id=7bad854e3d7ee4c4b2a0ebede24e1ee41797d3b1#bd50d05c414b847ffb1be734807753e08662c778_52_58

And then I missed the implications of that addition being in that top-level click group, and just blindly made it async aware to resolve the merge conflict from having just landed that large async refactor:

https://gitlab.com/meltano/meltano/-/merge_requests/2179/diffs?commit_id=a029c46acb157b10870e62ff2d5bb3ea2822b34e

So as @DouweM alluded, I think the fix is just dropping that run_async(invoker.prepare(session)) statement, and double-checking that meltano config <plugin> test which actually relies on that invoker still works as expected.

pandemicsyn on Jun 17, 2022

@pandemicsyn OK, I think we can at least fix the meltano config issue then by losing that superfluous invoke call. If it’s never happened on meltano invoke we don’t really need to dive deeper into that part of it.

I’ll put this in the project/board, you can figure out with AJ if it makes sense for you to pick it up or someone else should!

DouweM on Jun 9, 2022