airflow: PythonVirtualOperator fails silently when virtualenv is not installed.

Apache Airflow version

2.9.2

If “Other Airflow 2 version” selected, which one?

No response

What happened?

When using PythonVirtualOperator the venv is created by airflow as passed in the dag, in a tmp directory a standard python -m venv /tmp/.... is run. Later pip install -r ... is also run.

However, when virtualenv is not installed the failed command is not properly detected.

Later an uncaught OSError in subprocess.Popen (https://github.com/apache/airflow/blob/c310159bc2363c12110b11febd5febaab8670210/airflow/utils/process_utils.py#L184)is raised by the absent.venv/bin/pipcausing the exit logic not to fire and theTempDirectory.exit` then deletes the evidence.

See: subprocess Exceptions Additionally stderr is not logged.

What you think should happen instead?

I see in the project file there is a core-all listing including virtualenv.

Shoudn’t virtualenv be a dependecy of the pip install apache-airflow? I haven’t used hatch

If it shouldn’t I think there should be better error handing to explain the issue.

How to reproduce

Run a PythonVirtualOperator with airflow installed in a venv that has no virtualenv.

Operating System

ws2 ubuntu 2404

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (9 by maintainers)

Commits related to this issue

Most upvoted comments

Hi @pateash , I really appreciate your concern regarding this issue but I think I will just continue to work on this. It’s just been busy weeks for a bit. I will be opening a PR within this week