ray: [Dashboard] Broken on multi node.

What is the problem?

Ray version and other system information (Python version, TensorFlow version, OS):

Reproduction (REQUIRED)

Please provide a script that can be run to reproduce the issue. The script should have no external library dependencies (i.e., use fake or mock data / environments):

> ray start --head # start a cluster
> ray start --address auto # start the worker

go to the dashboard:

SyntaxError: Unexpected token I in JSON at position 4
GET http://localhost:8265/api/raylet_info 500 (Internal Server Error)

Dashboard log

Error handling request
Traceback (most recent call last):
  File "/Users/simonmo/miniconda3/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/Users/simonmo/miniconda3/lib/python3.6/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/Users/simonmo/Desktop/ray/ray/python/ray/dashboard/dashboard.py", line 264, in raylet_info
    result = self.dashboard_controller.get_raylet_info()
  File "/Users/simonmo/Desktop/ray/ray/python/ray/dashboard/dashboard.py", line 158, in get_raylet_info
    return self._construct_raylet_info()
  File "/Users/simonmo/Desktop/ray/ray/python/ray/dashboard/dashboard.py", line 96, in _construct_raylet_info
    workers_info_by_node, infeasible_tasks, ready_tasks)
  File "/Users/simonmo/Desktop/ray/ray/python/ray/dashboard/node_stats.py", line 162, in get_actors
    for worker_info in workers_info:
TypeError: 'NoneType' object is not iterable

If we cannot run your script, we cannot fix your issue.

I have verified my script runs in a clean environment and reproduces the issue.
I have verified the issue also occurs with the latest wheels.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 19 (8 by maintainers)

Most upvoted comments

works for me now with new 1.0.1 release.

utke1 on Nov 11, 2020

For me the issue is not in first tab (which worked OK) but instead in the “TUNE” tab when I set the directory it gives the json error. I got the latest(?) wheel from https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-1.1.0.dev0-cp36-cp36m-manylinux1_x86_64.whl following the instructions on https://docs.ray.io/en/master/installation.html The ray install-nightly pointed to a non-existent ````https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-1.0.0-cp36-cp36m-manylinux1_x86_64.whl``` so I grabbed the wheel one from the table. In any case, with that wheel I am now getting different errors showing in the log

2020-10-19 16:40:12,385	WARNING dashboard.py:220 -- The dashboard on node vap0846 failed with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ray/new_dashboard/dashboard.py", line 207, in <module>
    log_dir=log_dir)
  File "/usr/local/lib/python3.6/dist-packages/ray/new_dashboard/dashboard.py", line 78, in __init__
    build_dir = setup_static_dir()
  File "/usr/local/lib/python3.6/dist-packages/ray/new_dashboard/dashboard.py", line 42, in setup_static_dir
    "&& npm run build)", build_dir)
FileNotFoundError: [Errno 2] Dashboard build directory not found. If installing from source, please follow the additional steps required to build the dashboard(cd python/ray/new_dashboard/client && npm install && npm ci && npm run build): '/usr/local/lib/python3.6/dist-packages/ray/new_dashboard/client/build'

so I guess something in the pip update went wrong…

utke1 on Oct 19, 2020

Hi, we are using 1.1.0 and still got this error.

trunghlt on Jan 27, 2021