polyaxon: Logs are not displayed correctly in terminal

Describe the bug

Unable to see the logs correctly. Unfortunately the only things visible within in terminal are callback errors:

$ polyaxon experiment -xp X logs
building -- 
scheduled -- 
starting -- 
running -- 
error from callback <function SocketTransportMixin.socket.<locals>.<lambda> at 0x7fd723146400>: the JSON object must be str, not 'bytes'
error from callback <function SocketTransportMixin.socket.<locals>.<lambda> at 0x7fd723146400>: the JSON object must be str, not 'bytes'
error from callback <function SocketTransportMixin.socket.<locals>.<lambda> at 0x7fd723146400>: the JSON object must be str, not 'bytes'
error from callback <function SocketTransportMixin.socket.<locals>.<lambda> at 0x7fd723146400>: the JSON object must be str, not 'bytes'
error from callback <function SocketTransportMixin.socket.<locals>.<lambda> at 0x7fd723146400>: the JSON object must be str, not 'bytes'
...
error from callback <bound method SocketTransportMixin._on_close of <polyaxon_client.transport.Transport object at 0x7fd723190978>>: _on_close() missing 1 required positional argument: 'ws'

To Reproduce

Started experiment with polyaxon run -u and then started the logs-view polyaxon experiment -xp X logs

Experiment:

https://github.com/polyaxon/polyaxon-examples/tree/master/tensorflow/cifare10/polyaxonfile.yml

Expected behavior

Building -- creating image -
  master.1 -- INFO:tensorflow:Using config: {'_model_dir': '/outputs/root/cifar10/experiments/1', '_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_session_config': gpu_options {
  master.1 --   force_gpu_compatible: true
  master.1 -- }

Environment

Local

polyaxon is running within a virtualenv using python3.

Cluster

OS: Ubuntu 18.04 Kubernetes: 1.12.1

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 19 (19 by maintainers)

Most upvoted comments

@vfdev-5 resources are not saved after an experiment is finished, there’s already a ticket to persist the tracked resources, but it’s not done yet.