airflow: Error on logging empty line to Cloudwatch
Apache Airflow version: 2.0.1
Environment:
- Cloud provider or hardware configuration: AWS
What happened:
I have Airflow with Cloudwatch-based remote logging running. I also have BashOperator
that does, for example, rsync
with invalid parameters, for example rsync -av test test
. The output of the rsync
error is formatted and contains empty line. Once that empty line is logged to the Cloudwatch, i receive an error:
2021-04-06 19:29:22,318] /home/airflow/.local/lib/python3.6/site-packages/watchtower/__init__.py:154 WatchtowerWarning: Failed to deliver logs: Parameter validation failed:
Invalid length for parameter logEvents[5].message, value: 0, valid range: 1-inf
[2021-04-06 19:29:22,320] /home/airflow/.local/lib/python3.6/site-packages/watchtower/__init__.py:158 WatchtowerWarning: Failed to deliver logs: None
So basically empty lines can’t be submitted to the Cloudwatch and as result the whole output of the process doesn’t appear in logs.
What you expected to happen:
I expect to have an output of the bash command in logs. Empty lines can be skipped or replaced with something.
How to reproduce it:
For example: run BashOperator
with rsync
command that fails on Airflow with Cloudwatch-based remote logging. It could be any other command that produces empty line in the output.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 20 (12 by maintainers)
Commits related to this issue
- Move to watchtower 2.0.1 - This version of watchtower contains patches that fixes #15279 where empty log lines would crash Watchtower. — committed to o-nikolas/airflow by o-nikolas 3 years ago
- Move to watchtower 2.0.1 (#19907) - This version of watchtower contains patches that fixes #15279 where empty log lines would crash Watchtower. — committed to apache/airflow by o-nikolas 3 years ago
- Move to watchtower 2.0.1 (#19907) - This version of watchtower contains patches that fixes #15279 where empty log lines would crash Watchtower. (cherry picked from commit 2539cb44b47d78e81a88fde51... — committed to apache/airflow by o-nikolas 3 years ago
- Move to watchtower 2.0.1 (#19907) - This version of watchtower contains patches that fixes #15279 where empty log lines would crash Watchtower. (cherry picked from commit 2539cb44b47d78e81a88fde51... — committed to apache/airflow by o-nikolas 3 years ago
I cut an issue to the watchtower repo (since the other was closed) with some possible fixes I had tested. But it seems like there is already a fix that just got released into Watchtower 2.0.
I have yet to do some testing with the new release, but once I get around to (and assuming it works as expected) it should be an easy fix to bump the watchtower version in Airflow to 2.0.
I’m happy to be assigned this issue if no one else is working on it.
I just saw that a new version of apache-airflow-providers-amazon was released (2.0.0) with an updated watchtower, i’ll test it later 😄
Hey y’all, i’m having the same issue now, but i found a workaround, you can use this awk command to delete the newlines in the Bash command:
my_cool_command | awk NF
In case you have the same problem as i did (In my case it happens with running dbt, which puts empty newlines for styling between it’s messages)
Have a great day!