airflow: PythonVirtualOperator fails silently when virtualenv is not installed.
Apache Airflow version
2.9.2
If “Other Airflow 2 version” selected, which one?
No response
What happened?
When using PythonVirtualOperator the venv is created by airflow as passed in the dag, in a tmp directory a standard python -m venv /tmp/.... is run. Later pip install -r ... is also run.
However, when virtualenv is not installed the failed command is not properly detected.
Later an uncaught OSError in subprocess.Popen (https://github.com/apache/airflow/blob/c310159bc2363c12110b11febd5febaab8670210/airflow/utils/process_utils.py#L184)is raised by the absent.venv/bin/pipcausing the exit logic not to fire and theTempDirectory.exit` then deletes the evidence.
See: subprocess Exceptions
Additionally stderr is not logged.
What you think should happen instead?
I see in the project file there is a core-all listing including virtualenv.
Shoudn’t virtualenv be a dependecy of the pip install apache-airflow? I haven’t used hatch
If it shouldn’t I think there should be better error handing to explain the issue.
How to reproduce
Run a PythonVirtualOperator with airflow installed in a venv that has no virtualenv.
Operating System
ws2 ubuntu 2404
Versions of Apache Airflow Providers
No response
Deployment
Virtualenv installation
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (9 by maintainers)
Commits related to this issue
- Switch PythonVirtualenvOperator to venv from virtualenv package The PythonVirtualenvOperator has been using virtualenv in order to support Python 2.7 and pre Python 3.5, however we do not need those ... — committed to potiuk/airflow by potiuk 2 years ago
- Switch PythonVirtualenvOperator to venv from virtualenv package The PythonVirtualenvOperator has been using virtualenv in order to support Python 2.7 and pre Python 3.5, however we do not need those ... — committed to potiuk/airflow by potiuk 2 years ago
- Switch PythonVirtualenvOperator to venv from virtualenv package The PythonVirtualenvOperator has been using virtualenv in order to support Python 2.7 and pre Python 3.5, however we do not need those ... — committed to potiuk/airflow by potiuk 2 years ago
- Switch PythonVirtualenvOperator to venv from virtualenv package The PythonVirtualenvOperator has been using virtualenv in order to support Python 2.7 and pre Python 3.5, however we do not need those ... — committed to potiuk/airflow by potiuk 2 years ago
- Switch PythonVirtualenvOperator to venv from virtualenv package The PythonVirtualenvOperator has been using virtualenv in order to support Python 2.7 and pre Python 3.5, however we do not need those ... — committed to potiuk/airflow by potiuk 2 years ago
- Switch PythonVirtualenvOperator to venv from virtualenv package The PythonVirtualenvOperator has been using virtualenv in order to support Python 2.7 and pre Python 3.5, however we do not need those ... — committed to potiuk/airflow by potiuk 2 years ago
- Switch PythonVirtualenvOperator to venv from virtualenv package The PythonVirtualenvOperator has been using virtualenv in order to support Python 2.7 and pre Python 3.5, however we do not need those ... — committed to potiuk/airflow by potiuk 2 years ago
- Switch PythonVirtualenvOperator to venv from virtualenv package The PythonVirtualenvOperator has been using virtualenv in order to support Python 2.7 and pre Python 3.5, however we do not need those ... — committed to potiuk/airflow by potiuk 2 years ago
- Switch PythonVirtualenvOperator to venv from virtualenv package (#43568) The PythonVirtualenvOperator has been using virtualenv in order to support Python 2.7 and pre Python 3.5, however we do not n... — committed to apache/airflow by potiuk 2 years ago
- Switch PythonVirtualenvOperator to venv from virtualenv package (#43568) The PythonVirtualenvOperator has been using virtualenv in order to support Python 2.7 and pre Python 3.5, however we do not n... — committed to ellisms/airflow by potiuk 2 years ago
Hi @pateash , I really appreciate your concern regarding this issue but I think I will just continue to work on this. It’s just been busy weeks for a bit. I will be opening a PR within this week