pipelines: Unable to use tensorboard viewer using S3 path
What steps did you take
To view the output in tensorboard we have created the mlpipeline-ui-metadata.json file with following content (Followed the instructions from - https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/#tensorboard )
{"outputs": [{"type": "tensorboard", "source": "s3://<S3_path>"}]}
to view the training output in tensorboard.
What happened:
In the visualizations tab, Start Tensorboard button is appearing as expected.
Once Start Tensorboard is clicked, Open Tensorboard is appearing as expected.
But when the Start Tensorboard is clicked, an empty dashboard is shown.
What did you expect to happen:
Tensorboard should be opened as below
Environment:
- KFP version: 1.4 To find the version number, See version number shows on bottom of KFP UI left sidenav. -->
- KFP SDK version: 0.5.2
Anything else you would like to add:
To test the above behavious in local machine, we tried running
tensorboard --logdir s3://<S3_path>
Tensorboard is showing an empty dashboard.
However, the dashboard is working fine with the db argument as below
tensorboard --db s3://<S3_path>
From the https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/#tensorboard i couldnt figure out a way to change from logdir to db. Or is there any other way to achieve this in kubeflow mlpipeline-ui-metadata.json ?
Labels
/area frontend
Impacted by this bug? Give it a đź‘Ť. We prioritise the issues with the most đź‘Ť.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 20 (19 by maintainers)
Already occupied with deliverables. I can pick it up after 2 weeks. I will make a note of it.
@Bobgy It will be more helpful if the documentation is updated about the supported systems - https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/#tensorboard . What do you think ?
@Bobgy We have identified the solution. Tensorboard with S3 buckets works based on region. The bucket which we used was in different region. Once we updated the AWS_REGION variable in cluster, we are able to see the tensorboard UI
@lechwolowski it’s not currently, but I just filed https://github.com/kubeflow/pipelines/issues/5471 which tries to support customizing tensorboard image
@Bobgy Thanks for the reply.
We were able to log the tensorboard events to S3. Only the Dashboard was having issues when opened with S3 Path. I was looking into couple of blog posts related to the tensorboard S3 integration. One of them suggested to have the paramter as
dbinstead oflogdir(i will try to see if i can get the link). However, i couldnt do it in KFP, as there is no option to set thedbparameter.I have quick question before the suggestion. Does KFP support Tensorboard with S3 urls as backend?.
It would be helpful to have an extra parameter in
mlpipeline-ui-metadata.jsonfile. For Example:{"outputs": [{"type": "tensorboard", "source": "s3://<S3_path>", "type" : "db"}]}