airflow: Bad terminal handling of `airflow db shell` command with Postgres DB

Apache Airflow version

main (development)

What happened

This happens in both iTerm 2 and the built-in Terminal.app in macOS when running ~breeze start-airflow~. airflow db shell when using postgres as a backend.

Within psql (run via airflow db shell):

  • Scrollback editing (eg: hitting up arrow to edit a previous query) is messed up for multi-line queries. I don’t have a better description. I recorded a video of what happens.
  • Some things require an additional keypress (eg: quitting the pager)
    • Some things, like ctrl+P to scrollback needing an additional keypress happen in a normal Docker environment. This is different and distinct from that.
  • SIGINT isn’t registered when trying to cancel editing a query and reset the SQL prompt

I suspect that there’s a lot of interaction happening between iTerm/Terminal.app, ZSH, Python/Breeze, Bash, Docker, tmux, Python/Airflow CLI, and psql, and some keypresses are being filtered or queued along the way. So there are multiple fun rabbit holes to chase down to solve this.

What you think should happen instead

  • Scrollback editing should work. Your text cursor should be in the correct place and the text should be rendered correctly.
  • An additional keypress to exit the pager should not be necessary
  • SIGINT should cancel editing the query and reset the edit line (eg: it should return to an empty SQL prompt: airflow=# instead of airflow-#)

How to reproduce

  • Scrollback editing is messed up - indentation and text wrapping are off/wrong

  • Some things require an additional keypress Example:

    • Run a SQL query:
      airflow=# SELECT * FROM dag_schedule_dataset_reference
      
    • it displays in the pager
       dataset_id |        dag_id         |          created_at           |          updated_at
      ------------+-----------------------+-------------------------------+-------------------------------
                3 | consumer_dag_2        | 2022-09-06 20:49:23.719275+00 | 2022-09-06 20:49:23.719295+00
                2 | consumer_dag_2        | 2022-09-06 20:49:23.7193+00   | 2022-09-06 20:49:23.719302+00
                4 | taskflow_consumer_dag | 2022-09-06 20:49:23.719305+00 | 2022-09-06 20:49:23.719308+00
                5 | taskflow_consumer_dag | 2022-09-06 20:49:23.719311+00 | 2022-09-06 20:49:23.719313+00
                6 | taskflow_consumer_dag | 2022-09-06 20:49:23.719316+00 | 2022-09-06 20:49:23.719319+00
                1 | consumer_dag_1        | 2022-09-06 20:49:23.719322+00 | 2022-09-06 20:49:23.719324+00
      (6 rows)
      
      (END)
      
    • Usually you just hit q and it exits immediately, but the pager sticks around:
       dataset_id |        dag_id         |          created_at           |          updated_at
      ------------+-----------------------+-------------------------------+-------------------------------
                3 | consumer_dag_2        | 2022-09-06 20:49:23.719275+00 | 2022-09-06 20:49:23.719295+00
                2 | consumer_dag_2        | 2022-09-06 20:49:23.7193+00   | 2022-09-06 20:49:23.719302+00
                4 | taskflow_consumer_dag | 2022-09-06 20:49:23.719305+00 | 2022-09-06 20:49:23.719308+00
                5 | taskflow_consumer_dag | 2022-09-06 20:49:23.719311+00 | 2022-09-06 20:49:23.719313+00
                6 | taskflow_consumer_dag | 2022-09-06 20:49:23.719316+00 | 2022-09-06 20:49:23.719319+00
                1 | consumer_dag_1        | 2022-09-06 20:49:23.719322+00 | 2022-09-06 20:49:23.719324+00
      (6 rows)
      
      (END)
      
    • The q keypress is registered upon your next keypress, whatever that is (in this case, 6). And the second keypress is echoed in the SQL editor:
      airflow=# SELECT * FROM dag_schedule_dataset_reference;
      airflow=# 6
      
  • SIGINT isn’t registered to cancel editing a query Example:

    • Write a query:
      airflow=# SELECT * FROM dag_messedupspelling
      
    • Then try to cancel/reset it with Ctrl+C:
      airflow=# SELECT * FROM dag_messedupspelling
      airflow-#
      
      (nothing happens) In normal psql this will reset the query editor (you can tell because the airflow-# prompt changes back to airflow=#).

Operating System

macOS 11.6.8 Big Sur

Versions of Apache Airflow Providers

N/A

Deployment

Other Docker-based deployment

Deployment details

Breeze

Anything else

This happens every time.

Apologies if this isn’t the place to report Breeze issues.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 15 (14 by maintainers)

Most upvoted comments

I updated the description to clarify it. Fancy to attempt to fix it @blag ? I think it is the matter of how subprocess is used to run psql