airflow: Remote logging broken (Airflow 2.0.0 on GCS)
Apache Airflow version: 2.0.0
Kubernetes version (if you are using kubernetes) (use kubectl version):
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.14-gke.1600", GitCommit:"7c407f5cc8632f9af5a2657f220963aa7f1c46e7", GitTreeState:"clean", BuildDate:"2020-12-07T09:22:27Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}
Environment:
- Cloud provider or hardware configuration: GKE
- OS (e.g. from /etc/os-release):
- Kernel (e.g.
uname -a): - Install tools:
- Others:
What happened:
Remote logging is configured as follows:
export AIRFLOW__LOGGING__REMOTE_LOGGING=True
export AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=gs://your-bucket-name"
export AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=google_cloud_default
It worked flawlessly before the upgrade to 2.0.0. Now its utterly broken, and returns weird one-line logs, all prefixed with the same error message.
All logs are broken in the same way:
*** Reading remote log from gs://<your-bucket-name>/<dag_id>/<task_id>/2020-01-24T00:00:00+00:00/1.log.
b'*** Previous log discarded: 404 GET https://storage.googleapis.com/download/storage/v1/b/<redacted>?alt=media: No such object: <redacted>/2020-01-24T00:00:00+00:00/1.log: (\'Request failed with status code\', 404, \'Expected one of\', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)\n\n[2020-12-28 14:57:51,263] {taskinstance.py:826} INFO - Dependencies all met for <TaskInstance: <redacted> 2020-01-24T00:00:00+00:00 [queued]>\n[2020-12-28 14:57:51,281] {taskinstance.py:826} INFO - Dependencies all met for <TaskInstance: <redacted> 2020-01-24T00:00:00+00:00 [queued]>\n[2020-12-28 14:57:51,281] {taskinstance.py:1017} INFO - \n--------------------------------------------------------------------------------\n[2020-12-28 14:57:51,281] {taskinstance.py:1018} INFO - Starting attempt 1 of 1\n[2020-12-28 14:57:51,281] {taskinstance.py:1019} INFO - \n--------------------------------------------------------------------------------\n[2020-12-28 14:57:51,305] {taskinstance.py:1038} <<<SNIP>>>\n'
As you can see the, the actual log is there, however scrambled. At first I thought the log could not be written. But when I manually check the bucket, the log is actually there!
I suspect the GCSTaskHandler / remote logging to be broken on 2.0.0.
What you expected to happen:
Remote logging to GCS works as advertised.
How to reproduce it:
Get a GCS bucket.
Configure remote logging to GCS.
export AIRFLOW__LOGGING__REMOTE_LOGGING=True
export AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=gs://your-bucket-name"
export AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=google_cloud_default
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 20 (10 by maintainers)
@boring-cyborg who put you in charge? I followed the issue template!
@potiuk I still have the same problem. That
-was just a typo on my issue, the problem still persists. I have updated my issue accordingly.I do not have a permission problem, the logs get written too! I would appreciate some helpers in how to debug/better the problem instead of a blunt rejection of the problem at hand.
Here is the log file written to the bucket:
Here is the same log output from Airflow, just scrambled:
Here is a screenshot showing the Airflow service account has “Storage Admin” on the same bucket:
Like I said, I am very willing to help and debug the problem, please just don’t close the issue…
Milestone is target for fix, so hopefully we will be able to investigate and fix it and release it in 2.0.1
You can refer to #13115 for the one-line logs issue on GCS logging.
off topic:
@potiuk I get it. Sorry if I put out some bad vibes here, no intent, I was just surprised. Also thanks for having a second look.