tfx: TFX components in GCP does not display component logs in GCP Vertex AI

If the bug is related to a specific library below, please raise an issue in the respective repo directly: TFX

TensorFlow Data Validation Repo

TensorFlow Model Analysis Repo

TensorFlow Transform Repo

TensorFlow Serving Repo

System information

  • Have I specified the code to reproduce the issue (Yes, No):
  • Environment in which the code is executed (e.g., Local(Linux/MacOS/Windows), Interactive Notebook, Google Cloud, etc):
  • TensorFlow version: 2.13.1
  • TFX Version: TFX version: 1.14.0
  • KFP version: 1.8.22
  • Python version:3.8.18
  • Python dependencies (from pip freeze output):
  • google-api-core 2.15.0 google-api-python-client 1.12.11 google-apitools 0.5.31 google-auth 2.25.2 google-auth-httplib2 0.1.1 google-auth-oauthlib 1.0.0 google-cloud-aiplatform 1.37.0 google-cloud-appengine-logging 1.4.0 google-cloud-audit-log 0.2.5 google-cloud-bigquery 2.34.4 google-cloud-bigquery-storage 2.23.0 google-cloud-bigtable 2.21.0 google-cloud-core 2.4.1 google-cloud-datastore 2.18.0 google-cloud-dlp 3.14.0 google-cloud-language 2.12.0 google-cloud-logging 3.9.0 google-cloud-pubsub 2.19.0 google-cloud-pubsublite 1.8.3 google-cloud-recommendations-ai 0.10.6 google-cloud-resource-manager 1.11.0 google-cloud-spanner 3.40.1 google-cloud-storage 2.13.0 google-cloud-videointelligence 2.12.0 google-cloud-vision 3.5.0 google-crc32c 1.5.0 google-pasta 0.2.0 google-resumable-media 2.6.0 googleapis-common-protos 1.62.0 grpc-google-iam-v1 0.13.0

Describe the current behavior

I am running in GCP Vertex AI Kubeflow pipelines with TFX components. The problem is that no component logs are displayed in the Vertex interface (neither main job nor pipeline job) while in the Logs Explorer only framework messages are displayed. This is irrespective of the component type (ExamplesGen, Trainer, Transform, etc) and leads to very difficult blindly debugging of TFX components. I submit the pipelines using a service account which has Logs Writer/Reader privileges.

image

Describe the expected behavior Be able to view the component logs for code debugging.

Standalone code to reproduce the issue

Providing a bare minimum test case or step(s) to reproduce the problem will greatly help us to debug the issue. If possible, please share a link to Colab/Jupyter/any notebook.

Name of your Organization (Optional)

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 21 (11 by maintainers)

Most upvoted comments

@lego0901 Hi, thank you for pursuing this. I see you cannot reproduce with the Penguin Example. I will try to reproduce with a simple pipeline to further aid problem determination…

Hello, we also ran several VAI pipelines with our hands but we were able to see the component logs, regardless if a component run failed or not. This is very weird and I want to check if “all” components logs are not displayed regardless if it failed or not, @crbl1122.

But, I can give you a general way to debug.

  1. We usually can’t see the component logs if the orchestrator fails to launch a component.

  2. If that’s the case, we have to see the orchestrator’s log and you can find this in Error Reporting. So please visit there and see if there is a relevant error.

  3. Otherwise, you can follow Get Support to raise an issue.

+1. Logs are also not displayed when using PyTorch + Kubeflow pipelines. Please fix it, this seems to be a general issue. Not only makes debugging tricky but I also can’t get information if the specified GPU and memory is utilized when training.