charts: Importing shared module fails when using gitSync

What is the bug?

I have a DAG in my GitHub repository in examples/hello_world.py that imports a shared module at the root of the same repository. In production we are using Airflow 1.10.14 and the 7.16.0 chart running in a GKE cluster with the gitSync sidecar container and everything works well. We are building our own image based on the official image with some providers and extra Python libraries installed.

I’ve upgraded our identical test environment to the 8.0.2 chart running Airflow 2.0.1 and suddenly the same DAG fails to load, giving this error in the UI:

Broken DAG: [/opt/airflow/dags/repo/examples/hello_world.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/airflow/dags/repo/examples/hello_world.py", line 8, in <module>
    import shared
ModuleNotFoundError: No module named 'shared'

If I open a shell to the scheduler container, I can import all the correct module with no trouble. I can also import the relevant airflow code that parses the DAG and that too works correctly in the interpreter. Finally, I found that if I just run airflow scheduler in the same container, that scheduler is able to load all my DAGs successfully. Similarly, if I kill the scheduler process so that only that container restarts, the scheduler is then able to clear all the errors. However if I restart the pod all the errors come back again.

My only theory is that there is a race condition between the scheduler and the gitSync sidecar at container startup that triggers unexpected import behavior in Python.

What are your Helm values?

Any relevant parts of your custom_values.yaml

airflow:
  image:
    repository: gcr.io/####/####
    tag: ####
    pullPolicy: Always
    pullSecret: ""

  executor: CeleryExecutor

scheduler:
  resources:
    requests:
      cpu: "1000m"
      memory: "1Gi"

dags:
  gitSync:
    enabled: true
    repo: "ssh://git@github.com/####/####.git"
    branch: airflow2-testing
    revision: HEAD
    syncWait: 60
    sshSecret: "airflow-git-keys"
    sshSecretKey: id_rsa

What is your Kubernetes Version?:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.15-gke.800", GitCommit:"fa6f8f9671a3ba4b1790bac677bb72b04204e8fc", GitTreeState:"clean", BuildDate:"2020-12-21T09:19:13Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

What is your Helm version?:

$ helm version
version.BuildInfo{Version:"v3.5.2", GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", GitTreeState:"dirty", GoVersion:"go1.15.7"}