polyaxon: Logs are not displayed correctly in terminal
Describe the bug
Unable to see the logs correctly. Unfortunately the only things visible within in terminal are callback errors:
$ polyaxon experiment -xp X logs
building --
scheduled --
starting --
running --
error from callback <function SocketTransportMixin.socket.<locals>.<lambda> at 0x7fd723146400>: the JSON object must be str, not 'bytes'
error from callback <function SocketTransportMixin.socket.<locals>.<lambda> at 0x7fd723146400>: the JSON object must be str, not 'bytes'
error from callback <function SocketTransportMixin.socket.<locals>.<lambda> at 0x7fd723146400>: the JSON object must be str, not 'bytes'
error from callback <function SocketTransportMixin.socket.<locals>.<lambda> at 0x7fd723146400>: the JSON object must be str, not 'bytes'
error from callback <function SocketTransportMixin.socket.<locals>.<lambda> at 0x7fd723146400>: the JSON object must be str, not 'bytes'
...
error from callback <bound method SocketTransportMixin._on_close of <polyaxon_client.transport.Transport object at 0x7fd723190978>>: _on_close() missing 1 required positional argument: 'ws'
To Reproduce
Started experiment with polyaxon run -u and then started the logs-view polyaxon experiment -xp X logs
Experiment:
https://github.com/polyaxon/polyaxon-examples/tree/master/tensorflow/cifare10/polyaxonfile.yml
Expected behavior
Building -- creating image -
master.1 -- INFO:tensorflow:Using config: {'_model_dir': '/outputs/root/cifar10/experiments/1', '_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_session_config': gpu_options {
master.1 -- force_gpu_compatible: true
master.1 -- }
Environment
Local
polyaxon is running within a virtualenv using python3.
Cluster
OS: Ubuntu 18.04 Kubernetes: 1.12.1
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 19 (19 by maintainers)
@vfdev-5 resources are not saved after an experiment is finished, there’s already a ticket to persist the tracked resources, but it’s not done yet.