airflow: KubePodOperator sometimes fails with `Unable to parse string [failed]`

Apache Airflow version: 2.0

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.15

Environment: Kubernetes, using the KubernetesExecutor as well

  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
  • Kernel (e.g. uname -a): Linux airflow-7f86fbfdc-6s4l6 3.10.0-957.10.1.el7.x86_64
  • Others: I am using the official Airflow Docker Image

What happened:

I was trying to execute a DAG but for some reason it works at certain times but there are times it throws an error. Below I have attached two execution logs (I am executing the DAG twice, one after the other), the first execution throws a pendulum.parsing.exceptions.ParserError: Unable to parse string [failed] error.

Below is the log for the first execution

airflow@airflow-7f86fbfdc-6s4l6:/opt/airflow$ airflow dags test AtinXSxy 2020-10-10
[2020-12-01 01:30:57,766] {dagbag.py:440} INFO - Filling up the DagBag from /opt/airflow/dags
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
[2020-12-01 01:30:58,308] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.warc-to-json 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:03,202] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=warc-to-json
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:03,443] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Pending
[2020-12-01 01:31:03,444] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-4a01397600d247a89026e9dca8b920f6
[2020-12-01 01:31:04,475] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Pending
[2020-12-01 01:31:04,475] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-4a01397600d247a89026e9dca8b920f6
[2020-12-01 01:31:05,533] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Pending
[2020-12-01 01:31:05,534] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-4a01397600d247a89026e9dca8b920f6
[2020-12-01 01:31:06,587] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Running
[2020-12-01 01:31:07,622] {pod_launcher.py:136} INFO - warc_to_json.py starting...
[2020-12-01 01:31:07,622] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/4/8/base.json`
[2020-12-01 01:31:09,628] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Succeeded
[2020-12-01 01:31:09,628] {pod_launcher.py:289} INFO - Event with job id warc-to-json-4a01397600d247a89026e9dca8b920f6 Succeeded
[2020-12-01 01:31:09,680] {pod_launcher.py:176} INFO - Event: warc-to-json-4a01397600d247a89026e9dca8b920f6 had an event of type Succeeded
[2020-12-01 01:31:09,681] {pod_launcher.py:289} INFO - Event with job id warc-to-json-4a01397600d247a89026e9dca8b920f6 Succeeded
[2020-12-01 01:31:09,818] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=warc-to-json, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013109
[2020-12-01 01:31:09,892] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:31:09,956] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 5 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 5
[2020-12-01 01:31:10,013] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-id 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:10,463] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-id
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:10,743] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Pending
[2020-12-01 01:31:10,744] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-34424163821f4c1595936541737eaf1b
[2020-12-01 01:31:11,781] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Pending
[2020-12-01 01:31:11,781] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-34424163821f4c1595936541737eaf1b
[2020-12-01 01:31:12,813] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Pending
[2020-12-01 01:31:12,813] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-34424163821f4c1595936541737eaf1b
[2020-12-01 01:31:13,846] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Running
[2020-12-01 01:31:17,186] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/5/9/id.json`
[2020-12-01 01:31:17,186] {pod_launcher.py:136} INFO - extract_id.py starting...
[2020-12-01 01:31:19,092] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Succeeded
[2020-12-01 01:31:19,093] {pod_launcher.py:289} INFO - Event with job id add-id-34424163821f4c1595936541737eaf1b Succeeded
[2020-12-01 01:31:19,123] {pod_launcher.py:176} INFO - Event: add-id-34424163821f4c1595936541737eaf1b had an event of type Succeeded
[2020-12-01 01:31:19,124] {pod_launcher.py:289} INFO - Event with job id add-id-34424163821f4c1595936541737eaf1b Succeeded
[2020-12-01 01:31:19,226] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-id, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013119
[2020-12-01 01:31:19,277] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:31:19,341] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 4 | succeeded: 2 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 4
[2020-12-01 01:31:19,429] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-username-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:19,801] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-username-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:20,160] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Pending
[2020-12-01 01:31:20,161] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-77d091d0586e4ddda4ac67f1e4064612
[2020-12-01 01:31:21,190] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Pending
[2020-12-01 01:31:21,191] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-77d091d0586e4ddda4ac67f1e4064612
[2020-12-01 01:31:22,440] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Pending
[2020-12-01 01:31:22,440] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-77d091d0586e4ddda4ac67f1e4064612
[2020-12-01 01:31:23,467] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Pending
[2020-12-01 01:31:23,467] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-77d091d0586e4ddda4ac67f1e4064612
[2020-12-01 01:31:24,494] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Pending
[2020-12-01 01:31:24,494] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-77d091d0586e4ddda4ac67f1e4064612
[2020-12-01 01:31:25,521] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Running
[2020-12-01 01:31:27,674] {pod_launcher.py:136} INFO - extract_username.py starting...
[2020-12-01 01:31:27,674] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/6/10/username.json`
[2020-12-01 01:31:29,565] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Succeeded
[2020-12-01 01:31:29,565] {pod_launcher.py:289} INFO - Event with job id add-username-field-77d091d0586e4ddda4ac67f1e4064612 Succeeded
[2020-12-01 01:31:29,594] {pod_launcher.py:176} INFO - Event: add-username-field-77d091d0586e4ddda4ac67f1e4064612 had an event of type Succeeded
[2020-12-01 01:31:29,595] {pod_launcher.py:289} INFO - Event with job id add-username-field-77d091d0586e4ddda4ac67f1e4064612 Succeeded
[2020-12-01 01:31:29,683] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-username-field, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013129
[2020-12-01 01:31:29,719] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:31:29,756] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 3 | succeeded: 3 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 3
[2020-12-01 01:31:29,802] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-timestamp-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:29,958] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-timestamp-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:30,223] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Pending
[2020-12-01 01:31:30,224] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58
[2020-12-01 01:31:31,269] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Pending
[2020-12-01 01:31:31,269] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58
[2020-12-01 01:31:32,297] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Pending
[2020-12-01 01:31:32,298] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58
[2020-12-01 01:31:33,352] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Pending
[2020-12-01 01:31:33,352] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58
[2020-12-01 01:31:34,454] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Running
[2020-12-01 01:31:35,987] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/7/11/timestamp.json`
[2020-12-01 01:31:35,988] {pod_launcher.py:136} INFO - extract_timestamp.py starting...
[2020-12-01 01:31:37,865] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Succeeded
[2020-12-01 01:31:37,866] {pod_launcher.py:289} INFO - Event with job id add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 Succeeded
[2020-12-01 01:31:37,895] {pod_launcher.py:176} INFO - Event: add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 had an event of type Succeeded
[2020-12-01 01:31:37,895] {pod_launcher.py:289} INFO - Event with job id add-timestamp-field-d9a69307d9a84350a4f7dead1771df58 Succeeded
[2020-12-01 01:31:37,994] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-timestamp-field, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013137
[2020-12-01 01:31:38,060] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:31:38,116] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 2 | succeeded: 4 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
[2020-12-01 01:31:38,165] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-hashtags-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:38,363] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-hashtags-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:38,562] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Pending
[2020-12-01 01:31:38,563] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7
[2020-12-01 01:31:39,591] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Pending
[2020-12-01 01:31:39,591] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7
[2020-12-01 01:31:40,622] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Pending
[2020-12-01 01:31:40,622] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7
[2020-12-01 01:31:41,651] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Pending
[2020-12-01 01:31:41,652] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7
[2020-12-01 01:31:42,686] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Pending
[2020-12-01 01:31:42,686] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7
[2020-12-01 01:31:43,725] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Running
[2020-12-01 01:31:46,063] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/8/12/hashtags.json`
[2020-12-01 01:31:46,064] {pod_launcher.py:136} INFO - extract_hashtags.py starting...
[2020-12-01 01:31:47,974] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Succeeded
[2020-12-01 01:31:47,974] {pod_launcher.py:289} INFO - Event with job id add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 Succeeded
[2020-12-01 01:31:48,008] {pod_launcher.py:176} INFO - Event: add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 had an event of type Succeeded
[2020-12-01 01:31:48,008] {pod_launcher.py:289} INFO - Event with job id add-hashtags-field-73170a58a0a5453f86dd0fb93803eba7 Succeeded
[2020-12-01 01:31:48,131] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-hashtags-field, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013148
[2020-12-01 01:31:48,187] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:31:48,250] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 1 | succeeded: 5 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 1
[2020-12-01 01:31:48,321] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-mentions-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:31:48,474] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-mentions-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:31:48,696] {pod_launcher.py:176} INFO - Event: add-mentions-field-faac94be589a41408bc2d3245c182866 had an event of type Pending
[2020-12-01 01:31:48,696] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-faac94be589a41408bc2d3245c182866
[2020-12-01 01:31:49,804] {pod_launcher.py:176} INFO - Event: add-mentions-field-faac94be589a41408bc2d3245c182866 had an event of type Pending
[2020-12-01 01:31:49,804] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-faac94be589a41408bc2d3245c182866
[2020-12-01 01:31:50,836] {pod_launcher.py:176} INFO - Event: add-mentions-field-faac94be589a41408bc2d3245c182866 had an event of type Pending
[2020-12-01 01:31:50,836] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-faac94be589a41408bc2d3245c182866
[2020-12-01 01:31:51,901] {pod_launcher.py:176} INFO - Event: add-mentions-field-faac94be589a41408bc2d3245c182866 had an event of type Running
[2020-12-01 01:31:52,003] {taskinstance.py:1402} ERROR - Unable to parse string [failed]
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1087, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1306, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 340, in execute
    final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 486, in create_new_pod_for_operator
    final_state, result = launcher.monitor_pod(pod=self.pod, get_logs=self.get_logs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/kubernetes/pod_launcher.py", line 135, in monitor_pod
    last_log_time = pendulum.parse(timestamp)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parser.py", line 29, in parse
    return _parse(text, **options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parser.py", line 45, in _parse
    parsed = base_parse(text, **options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parsing/__init__.py", line 74, in parse
    return _normalize(_parse(text, **_options), **_options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parsing/__init__.py", line 128, in _parse
    raise ParserError("Unable to parse string [{}]".format(text))
pendulum.parsing.exceptions.ParserError: Unable to parse string [failed]
[2020-12-01 01:31:52,009] {taskinstance.py:1446} INFO - Marking task as FAILED. dag_id=AtinXSxy, task_id=add-mentions-field, execution_date=20201010T000000, start_date=20201201T013058, end_date=20201201T013152
[2020-12-01 01:31:52,037] {debug_executor.py:83} ERROR - Failed to execute task: Unable to parse string [failed].
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/debug_executor.py", line 78, in _run_task
    ti._run_raw_task(job_id=ti.job_id, **params)  # pylint: disable=protected-access
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 63, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1087, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1306, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 340, in execute
    final_state, _, result = self.create_new_pod_for_operator(labels, launcher)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 486, in create_new_pod_for_operator
    final_state, result = launcher.monitor_pod(pod=self.pod, get_logs=self.get_logs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/kubernetes/pod_launcher.py", line 135, in monitor_pod
    last_log_time = pendulum.parse(timestamp)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parser.py", line 29, in parse
    return _parse(text, **options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parser.py", line 45, in _parse
    parsed = base_parse(text, **options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parsing/__init__.py", line 74, in parse
    return _normalize(_parse(text, **_options), **_options)
  File "/home/airflow/.local/lib/python3.7/site-packages/pendulum/parsing/__init__.py", line 128, in _parse
    raise ParserError("Unable to parse string [{}]".format(text))
pendulum.parsing.exceptions.ParserError: Unable to parse string [failed]
[2020-12-01 01:31:52,064] {backfill_job.py:219} ERROR - Task instance <TaskInstance: AtinXSxy.add-mentions-field 2020-10-10 00:00:00+00:00 [failed]> failed
[2020-12-01 01:31:52,084] {dagrun.py:432} ERROR - Marking run <DagRun AtinXSxy @ 2020-10-10 00:00:00+00:00: backfill__2020-10-10T00:00:00+00:00, externally triggered: False> failed
[2020-12-01 01:31:52,088] {dagrun.py:598} WARNING - Failed to record first_task_scheduling_delay metric:
unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'
[2020-12-01 01:31:52,093] {backfill_job.py:388} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 5 | running: 0 | failed: 1 | skipped: 0 | deadlocked: 0 | not ready: 0
Some task instances failed:
DAG ID    Task ID             Execution date               Try number
--------  ------------------  -------------------------  ------------
AtinXSxy  add-mentions-field  2020-10-10 00:00:00+00:00             1

Below is the log for the second execution

airflow@airflow-7f86fbfdc-6s4l6:/opt/airflow$ airflow dags test AtinXSxy 2020-10-10
[2020-12-01 01:41:56,401] {dagbag.py:440} INFO - Filling up the DagBag from /opt/airflow/dags
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
[2020-12-01 01:41:56,917] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.warc-to-json 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:01,773] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=warc-to-json
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:02,051] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Pending
[2020-12-01 01:42:02,052] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7
[2020-12-01 01:42:03,090] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Pending
[2020-12-01 01:42:03,091] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7
[2020-12-01 01:42:04,191] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Pending
[2020-12-01 01:42:04,191] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7
[2020-12-01 01:42:05,234] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Pending
[2020-12-01 01:42:05,234] {pod_launcher.py:113} WARNING - Pod not yet started: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7
[2020-12-01 01:42:06,314] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Running
[2020-12-01 01:42:06,848] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/4/8/base.json`
[2020-12-01 01:42:06,849] {pod_launcher.py:136} INFO - warc_to_json.py starting...
[2020-12-01 01:42:08,736] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Succeeded
[2020-12-01 01:42:08,736] {pod_launcher.py:289} INFO - Event with job id warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 Succeeded
[2020-12-01 01:42:08,766] {pod_launcher.py:176} INFO - Event: warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 had an event of type Succeeded
[2020-12-01 01:42:08,766] {pod_launcher.py:289} INFO - Event with job id warc-to-json-a9fc7f5a9c5b4d9281bf924f2579cae7 Succeeded
[2020-12-01 01:42:08,894] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=warc-to-json, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014208
[2020-12-01 01:42:08,959] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:09,039] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 5 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 5
[2020-12-01 01:42:09,124] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-id 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:09,396] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-id
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:09,687] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Pending
[2020-12-01 01:42:09,688] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-f429c835d2f340dd8566b1789bd538ab
[2020-12-01 01:42:10,864] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Pending
[2020-12-01 01:42:10,864] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-f429c835d2f340dd8566b1789bd538ab
[2020-12-01 01:42:11,895] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Pending
[2020-12-01 01:42:11,895] {pod_launcher.py:113} WARNING - Pod not yet started: add-id-f429c835d2f340dd8566b1789bd538ab
[2020-12-01 01:42:12,925] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Running
[2020-12-01 01:42:16,396] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/5/9/id.json`
[2020-12-01 01:42:16,397] {pod_launcher.py:136} INFO - extract_id.py starting...
[2020-12-01 01:42:18,277] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Succeeded
[2020-12-01 01:42:18,278] {pod_launcher.py:289} INFO - Event with job id add-id-f429c835d2f340dd8566b1789bd538ab Succeeded
[2020-12-01 01:42:18,330] {pod_launcher.py:176} INFO - Event: add-id-f429c835d2f340dd8566b1789bd538ab had an event of type Succeeded
[2020-12-01 01:42:18,330] {pod_launcher.py:289} INFO - Event with job id add-id-f429c835d2f340dd8566b1789bd538ab Succeeded
[2020-12-01 01:42:18,420] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-id, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014218
[2020-12-01 01:42:18,486] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:18,552] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 4 | succeeded: 2 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 4
[2020-12-01 01:42:18,634] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-username-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:18,948] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-username-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:19,174] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Pending
[2020-12-01 01:42:19,175] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-27348aafaaba44b68474700497b94be2
[2020-12-01 01:42:20,205] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Pending
[2020-12-01 01:42:20,206] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-27348aafaaba44b68474700497b94be2
[2020-12-01 01:42:21,254] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Pending
[2020-12-01 01:42:21,254] {pod_launcher.py:113} WARNING - Pod not yet started: add-username-field-27348aafaaba44b68474700497b94be2
[2020-12-01 01:42:22,285] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Running
[2020-12-01 01:42:24,051] {pod_launcher.py:136} INFO - extract_username.py starting...
[2020-12-01 01:42:24,051] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/6/10/username.json`
[2020-12-01 01:42:25,987] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Succeeded
[2020-12-01 01:42:25,987] {pod_launcher.py:289} INFO - Event with job id add-username-field-27348aafaaba44b68474700497b94be2 Succeeded
[2020-12-01 01:42:26,018] {pod_launcher.py:176} INFO - Event: add-username-field-27348aafaaba44b68474700497b94be2 had an event of type Succeeded
[2020-12-01 01:42:26,019] {pod_launcher.py:289} INFO - Event with job id add-username-field-27348aafaaba44b68474700497b94be2 Succeeded
[2020-12-01 01:42:26,162] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-username-field, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014226
[2020-12-01 01:42:26,202] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:26,250] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 3 | succeeded: 3 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 3
[2020-12-01 01:42:26,306] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-timestamp-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:26,492] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-timestamp-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:26,776] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Pending
[2020-12-01 01:42:26,777] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0
[2020-12-01 01:42:27,811] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Pending
[2020-12-01 01:42:27,817] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0
[2020-12-01 01:42:28,854] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Pending
[2020-12-01 01:42:28,855] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0
[2020-12-01 01:42:29,892] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Pending
[2020-12-01 01:42:29,893] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0
[2020-12-01 01:42:30,931] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Pending
[2020-12-01 01:42:30,935] {pod_launcher.py:113} WARNING - Pod not yet started: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0
[2020-12-01 01:42:32,061] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Running
[2020-12-01 01:42:33,223] {pod_launcher.py:136} INFO - extract_timestamp.py starting...
[2020-12-01 01:42:33,223] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/7/11/timestamp.json`
[2020-12-01 01:42:35,133] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Succeeded
[2020-12-01 01:42:35,134] {pod_launcher.py:289} INFO - Event with job id add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 Succeeded
[2020-12-01 01:42:35,179] {pod_launcher.py:176} INFO - Event: add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 had an event of type Succeeded
[2020-12-01 01:42:35,181] {pod_launcher.py:289} INFO - Event with job id add-timestamp-field-8c2eab9f51524b169b6ba1bcc361d4a0 Succeeded
[2020-12-01 01:42:35,304] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-timestamp-field, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014235
[2020-12-01 01:42:35,355] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:35,388] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 2 | succeeded: 4 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
[2020-12-01 01:42:35,433] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-hashtags-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:35,558] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-hashtags-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:35,891] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Pending
[2020-12-01 01:42:35,892] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-d47b7007849246eeb851cb55030c9a18
[2020-12-01 01:42:36,988] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Pending
[2020-12-01 01:42:36,989] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-d47b7007849246eeb851cb55030c9a18
[2020-12-01 01:42:38,024] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Pending
[2020-12-01 01:42:38,024] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-d47b7007849246eeb851cb55030c9a18
[2020-12-01 01:42:39,098] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Pending
[2020-12-01 01:42:39,099] {pod_launcher.py:113} WARNING - Pod not yet started: add-hashtags-field-d47b7007849246eeb851cb55030c9a18
[2020-12-01 01:42:40,131] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Running
[2020-12-01 01:42:42,831] {pod_launcher.py:136} INFO - extract_hashtags.py starting...
[2020-12-01 01:42:42,832] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/8/12/hashtags.json`
[2020-12-01 01:42:44,725] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Succeeded
[2020-12-01 01:42:44,725] {pod_launcher.py:289} INFO - Event with job id add-hashtags-field-d47b7007849246eeb851cb55030c9a18 Succeeded
[2020-12-01 01:42:44,771] {pod_launcher.py:176} INFO - Event: add-hashtags-field-d47b7007849246eeb851cb55030c9a18 had an event of type Succeeded
[2020-12-01 01:42:44,771] {pod_launcher.py:289} INFO - Event with job id add-hashtags-field-d47b7007849246eeb851cb55030c9a18 Succeeded
[2020-12-01 01:42:44,849] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-hashtags-field, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014244
[2020-12-01 01:42:44,903] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:44,968] {backfill_job.py:388} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 1 | succeeded: 5 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 1
[2020-12-01 01:42:45,030] {base_executor.py:79} INFO - Adding to queue: ['<TaskInstance: AtinXSxy.add-mentions-field 2020-10-10 00:00:00+00:00 [queued]>']
[2020-12-01 01:42:45,134] {taskinstance.py:1232} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=AtinXSxy
AIRFLOW_CTX_TASK_ID=add-mentions-field
AIRFLOW_CTX_EXECUTION_DATE=2020-10-10T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-10-10T00:00:00+00:00
[2020-12-01 01:42:45,340] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Pending
[2020-12-01 01:42:45,340] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b
[2020-12-01 01:42:46,373] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Pending
[2020-12-01 01:42:46,373] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b
[2020-12-01 01:42:47,405] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Pending
[2020-12-01 01:42:47,406] {pod_launcher.py:113} WARNING - Pod not yet started: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b
[2020-12-01 01:42:48,438] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Running
[2020-12-01 01:42:51,813] {pod_launcher.py:136} INFO - extract_mentions.py starting...
[2020-12-01 01:42:51,814] {pod_launcher.py:136} INFO - Warning: will overwrite file `/mnt/camelot-cs5604/workflows/AtinXSxy/9/13/mentions.json`
[2020-12-01 01:42:53,699] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Succeeded
[2020-12-01 01:42:53,699] {pod_launcher.py:289} INFO - Event with job id add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b Succeeded
[2020-12-01 01:42:53,732] {pod_launcher.py:176} INFO - Event: add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b had an event of type Succeeded
[2020-12-01 01:42:53,732] {pod_launcher.py:289} INFO - Event with job id add-mentions-field-cd44809b4d6b4e1c8249981d9cf7622b Succeeded
[2020-12-01 01:42:53,852] {taskinstance.py:1143} INFO - Marking task as SUCCESS. dag_id=AtinXSxy, task_id=add-mentions-field, execution_date=20201010T000000, start_date=20201201T014156, end_date=20201201T014253
[2020-12-01 01:42:53,889] {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2020-12-01 01:42:53,930] {dagrun.py:447} INFO - Marking run <DagRun AtinXSxy @ 2020-10-10 00:00:00+00:00: backfill__2020-10-10T00:00:00+00:00, externally triggered: False> successful
[2020-12-01 01:42:53,931] {dagrun.py:598} WARNING - Failed to record first_task_scheduling_delay metric:
unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'
[2020-12-01 01:42:53,937] {backfill_job.py:388} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 6 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
[2020-12-01 01:42:53,962] {backfill_job.py:830} INFO - Backfill done. Exiting.

What you expected to happen: I expect the DAG to run always just like in the case of the second execution

Anything else we need to know: The error occurs quite randomly, in my testing I found out that the error comes once in every 4-5 DAG runs

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 20 (12 by maintainers)

Most upvoted comments

Today!

+3
potiuk on Jun 14, 2021

@bmfisher thanks for those logs - unfortunately I need the pod logs to look at what the problematic line is, the airflow task logs just tell us where in the Airflow code the error is occurring, but I need to see the input line that is causing it.

+1
ashb on May 3, 2021

@bmfisher If you happen to still have those pod logs in your ES, could you go and search through them and find where the log line starts with “failed” or “Unable”? i.e. I’d love to be able to see the line the errors and a few lines (10?) either side of it.

Same to anyone who has this problem: if you are able to get the “raw” pod logs via kubectl ... get logs <podname> --timestamps this would be very helpful in debugging this issue. You probably need to set is_delete_operator_pod=False in the DAG to not have the pod be automatically failed.

+1
ashb on Apr 27, 2021
Read more comments on GitHub
← airflow: Jinja templating doesn't work with container_resources when using dymanic task mapping with Kubernetes Pod Operator
airflow: Kubernetes Executor Config Volumes Break Airflow UI →