airflow: helm install airflow in namespace get error: File "", line 32, in TimeoutError: There are still unapplied migrations after 60 seconds

Apache Airflow version: master git

Kubernetes version (if you are using kubernetes) (use kubectl version):


Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.17", 

Environment:

  • Cloud provider or hardware configuration: Microk8s
  • OS (e.g. from /etc/os-release): VERSION=“18.04.3 LTS (Bionic Beaver)”

What happened:

git clone https://github.com/apache/airflow.git
cd airflow/chart/
helm dependency update
kubectl create namespace xxxxx
werf helm install --wait --set webserver.defaultUser.password=password,ingress.enabled=true,ingress.hosts[0]=airflow.192.168.22.7.xip.io --namespace xxxxx airflow ./

Log

│ ┌ deploy/airflow-webserver po/airflow-webserver-86857b5969-sqkv6 container/wait-for-airflow-migrations logs
│ │ [2021-04-13 05:57:20,571] {<string>:35} INFO - Waiting for migrations... 60 second(s)
│ │ Traceback (most recent call last):
│ │   File "<string>", line 32, in <module>
│ │ TimeoutError: There are still unapplied migrations after 60 seconds.
│ └ deploy/airflow-webserver po/airflow-webserver-86857b5969-sqkv6 container/wait-for-airflow-migrations logs

Next line log

│ deploy/airflow-scheduler ERROR: po/airflow-scheduler-658d5d4454-r2sgl container/wait-for-airflow-migrations: CrashLoopBackOff: back-off 10s restarting failed container=wait-for-airflow-migrations             ↵
│ pod=airflow-scheduler-658d5d4454-r2sgl_sdpcc(40e85057-2aa5-4e9e-a47d-e91530038c0c)
│ 1/1 allowed errors occurred for deploy/airflow-scheduler: continue tracking

Full log https://gist.github.com/patsevanton/0edd5571cf69aa539edcdb803c288061

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 3
  • Comments: 37 (7 by maintainers)

Most upvoted comments

I’m facing the same issue. I don’t ever get any pod or job containing run-airflow-migrations, and consequently the wait never ends. Is there a solution for this? I’m not running terraform and neither is patsevanto. He is using werf, I’m using flux.

Anyone using ArgoCD to deploy the Airflow Helm chart who reaches this issue: read this piece of documentation.

When installing the chart using ArgoCD, you MUST set the two following values, or your application will not start as the migrations will not be run:

createUserJob.useHelmHooks: false
migrateDatabaseJob.useHelmHooks: false

I came to this issue upgrading to airflow 2.3.0 via helm chart 1.6.0. The issue turned out to be the migration job not being able to schedule due to a too low CPU request limit on the k8s namespace. Just another thing to check if you end up here like I did.

I also ran into this issue. In the interest of saving time for anyone else that stumbles upon this issue, the fix seems to be setting --wait=false on the Helm command, per @LiboShen’s advice.

On Rancher you can un-check “Wait” on the final page before deploying. I’m sure OpenShift and other solutions have similar options for exposing the underlying Helm --wait flag.

I can confirm that this worked on Rancher v2.6.1 installing Airflow to a downstream cluster provisioned by RKE running Kubernetes v1.21.5.

FYI, have you tried set “wait” false? I found this works for me: https://forum.astronomer.io/t/run-airflow-migration-and-wait-for-airflow-migrations/1189/10

@dingobar Thank you for the input. I am using the same versions and having (maybe similar) migration issue. May I ask you to elaborate on the solution?

Run kubectl -n <namespace> get events and see if there are events where some things are not being scheduled. If not then also try a kubectl -n <namespace> describe replicaset <migration job replicaset> and see if there are events there that can give you a clue. In my case, the scheduler could not start the migration pod due to a limit in how much CPU the namespace could request in total. You can see the limits by describing the namespace, kubectl describe namespace <namespace>. Hope that helps.

In my case, just removing the --wait flag was enough, I didn’t have to fiddle with airflow.dbMigrations.runAsJob

@yehoshuadimarsky Did you find the way to do this? I am also trying to implement the exact same thing and have been struggling to get the issue fixed.

Yes! I finally got this to work: put this in your values.yaml override:

 ​        # per https://github.com/apache/airflow/pull/16291 
 ​        # and https://github.com/apache/airflow/pull/16331 
 ​        createUserJob: 
 ​          jobAnnotations: 
 ​            "argocd.argoproj.io/hook": Sync 
 ​            "argocd.argoproj.io/sync-wave": "0" 
 ​            "argocd.argoproj.io/hook-delete-policy": BeforeHookCreation,HookSucceeded 
 ​        migrateDatabaseJob: 
 ​          jobAnnotations: 
 ​            "argocd.argoproj.io/hook": Sync 
 ​            "argocd.argoproj.io/sync-wave": "0" 
 ​            "argocd.argoproj.io/hook-delete-policy": BeforeHookCreation,HookSucceeded

I’m using ArgoCD to deploy the Helm chart, tearing out my hair trying every possible variation, but I’m also not seeing the run-airflow-migrations pod, it doesn’t run or show up. So my webserver and scheduler wait forever for the migrations that never started.

Not sure how to set the --wait=false param using Argo. I tried argocd app set [my-app] --helm-set-string wait=false but doesn’t seem to do anything.

So I’m stuck.

@LiboShen How add wait false to install I install airflow:

git clone https://github.com/apache/airflow.git
cd airflow/chart/
helm dependency update
kubectl create namespace apatsev
werf helm install --wait --set webserver.defaultUser.password=password,ingress.enabled=true,ingress.hosts[0]=airflow.192.168.22.8.sslip.io --namespace apatsev airflow ./

Create file or add option?