airflow: Airflow Scheduler on WSL fails to execute Windows EXE

Apache Airflow version: 1.10.11

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A

Environment:

• Cloud provider or hardware configuration: Local dev machine. Issue is unrelated to hardware.
• OS (e.g. from /etc/os-release): Ubuntu 20.04.1 LTS (Focal Fossa), running within WSL 2 on Windows 10 Pro 20H2, OS Build 19042.685 and Windows Feature Experience Pack 120.2212.551.0
• Kernel (e.g. uname -a): Linux LifeMapPC 4.19.128-microsoft-standard #1 SMP Tue Jun 23 12:58:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

What happened:

I have a BashOperator task which invokes an .EXE on Windows (via /mnt/c/... or via symlink). The task fails. Log shows:

[2020-12-16 18:34:11,833] {bash_operator.py:134} INFO - Temporary script location: /tmp/airflowtmp2gz6d79p/download.legacyFilesnihvszli
[2020-12-16 18:34:11,833] {bash_operator.py:146} INFO - Running command: /mnt/c/Windows/py.exe
[2020-12-16 18:34:11,836] {bash_operator.py:153} INFO - Output:
[2020-12-16 18:34:11,840] {bash_operator.py:159} INFO - Command exited with return code 1
[2020-12-16 18:34:11,843] {taskinstance.py:1150} ERROR - Bash command failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.8/dist-packages/airflow/operators/bash_operator.py", line 165, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
[2020-12-16 18:34:11,844] {taskinstance.py:1187} INFO - Marking task as FAILED. dag_id=test-dag, task_id=download.files, execution_date=20201216T043701, start_date=20201216T073411, end_date=20201216T073411

And that’s it. Return code 1 with no further useful info.

Running the very same EXE via bash works perfectly, with no error (I also tried it on my own program which emits something to the console - in bash it emits just fine, but via airflow scheduler it’s the same error 1).

What you expected to happen:

Expected the Windows executable to run successfully via airflow scheduler, same as when I run it directly in Bash. That is: emit any output to the console and return success (error 0).

Alternatively, happy to learn a way to get more insight into the log produced by the airflow scheduler run. i.e. to see “what happened” that makes it return error 1 on certain commands.

How to reproduce it:

I do not know the circumstances / environment “fault” that is causing it so can’t supply reproduction steps.

Anything else we need to know:

Some more data and things I’ve done to rule out any other issue:

• airflow scheduler runs as root. I also confirmed it’s running in a root context by putting an whoami command in my BashOperator, which indeed emitted root (I should also note that all native Linux programs run just fine! only the Windows programs don’t.)
• The Windows EXE I’m trying to execute and its directory have full ‘Everyone’ permissions (on my own program of course, wouldn’t dare doing it on my Windows folder - that was just an example.)
• The failure happens both when accessing via /mnt/c as well as via symlink. In the case of a symlink, the symlink has 777 permissions.
• I tried running airflow test on a BashOperator task - it runs perfectly - emits output to the console and returns 0 (success).
• Tried with various EXE files - both “native” (e.g. ones that come with Windows) as well as my C#-made programs. Same behavior in all.
• Didn’t find any similar issue documented in Airflow’s GitHub repo nor here in Stack Overflow.