airflow: helm install airflow in namespace get error: File "", line 32, in TimeoutError: There are still unapplied migrations after 60 seconds
Apache Airflow version:
master git
Kubernetes version (if you are using kubernetes) (use kubectl version
):
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.17",
Environment:
- Cloud provider or hardware configuration: Microk8s
- OS (e.g. from /etc/os-release): VERSION=“18.04.3 LTS (Bionic Beaver)”
What happened:
git clone https://github.com/apache/airflow.git
cd airflow/chart/
helm dependency update
kubectl create namespace xxxxx
werf helm install --wait --set webserver.defaultUser.password=password,ingress.enabled=true,ingress.hosts[0]=airflow.192.168.22.7.xip.io --namespace xxxxx airflow ./
Log
│ ┌ deploy/airflow-webserver po/airflow-webserver-86857b5969-sqkv6 container/wait-for-airflow-migrations logs
│ │ [2021-04-13 05:57:20,571] {<string>:35} INFO - Waiting for migrations... 60 second(s)
│ │ Traceback (most recent call last):
│ │ File "<string>", line 32, in <module>
│ │ TimeoutError: There are still unapplied migrations after 60 seconds.
│ └ deploy/airflow-webserver po/airflow-webserver-86857b5969-sqkv6 container/wait-for-airflow-migrations logs
Next line log
│ deploy/airflow-scheduler ERROR: po/airflow-scheduler-658d5d4454-r2sgl container/wait-for-airflow-migrations: CrashLoopBackOff: back-off 10s restarting failed container=wait-for-airflow-migrations ↵
│ pod=airflow-scheduler-658d5d4454-r2sgl_sdpcc(40e85057-2aa5-4e9e-a47d-e91530038c0c)
│ 1/1 allowed errors occurred for deploy/airflow-scheduler: continue tracking
Full log https://gist.github.com/patsevanton/0edd5571cf69aa539edcdb803c288061
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 3
- Comments: 37 (7 by maintainers)
I’m facing the same issue. I don’t ever get any pod or job containing run-airflow-migrations, and consequently the wait never ends. Is there a solution for this? I’m not running terraform and neither is patsevanto. He is using werf, I’m using flux.
Anyone using ArgoCD to deploy the Airflow Helm chart who reaches this issue: read this piece of documentation.
I came to this issue upgrading to airflow 2.3.0 via helm chart 1.6.0. The issue turned out to be the migration job not being able to schedule due to a too low CPU request limit on the k8s namespace. Just another thing to check if you end up here like I did.
I also ran into this issue. In the interest of saving time for anyone else that stumbles upon this issue, the fix seems to be setting
--wait=false
on the Helm command, per @LiboShen’s advice.On Rancher you can un-check “Wait” on the final page before deploying. I’m sure OpenShift and other solutions have similar options for exposing the underlying Helm
--wait
flag.I can confirm that this worked on Rancher v2.6.1 installing Airflow to a downstream cluster provisioned by RKE running Kubernetes v1.21.5.
FYI, have you tried set “wait” false? I found this works for me: https://forum.astronomer.io/t/run-airflow-migration-and-wait-for-airflow-migrations/1189/10
Run
kubectl -n <namespace> get events
and see if there are events where some things are not being scheduled. If not then also try akubectl -n <namespace> describe replicaset <migration job replicaset>
and see if there are events there that can give you a clue. In my case, the scheduler could not start the migration pod due to a limit in how much CPU the namespace could request in total. You can see the limits by describing the namespace,kubectl describe namespace <namespace>
. Hope that helps.In my case, just removing the --wait flag was enough, I didn’t have to fiddle with
airflow.dbMigrations.runAsJob
Yes! I finally got this to work: put this in your
values.yaml
override:I’m using ArgoCD to deploy the Helm chart, tearing out my hair trying every possible variation, but I’m also not seeing the
run-airflow-migrations
pod, it doesn’t run or show up. So my webserver and scheduler wait forever for the migrations that never started.Not sure how to set the
--wait=false
param using Argo. I triedargocd app set [my-app] --helm-set-string wait=false
but doesn’t seem to do anything.So I’m stuck.
@LiboShen How add wait false to install I install airflow:
Create file or add option?