ray: [dashboard] FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ray/session_2022-07-06_16-46-30_058837_12938/logs/dashboard.log'

What happened + What you expected to happen

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ray/session_2022-07-06_16-46-30_058837_12938/logs/dashboard.log'

2022-07-06 16:46:31,076	DEBUG gcs_utils.py:224 -- internal_kv_put b'CLUSTER_METADATA' b'{"ray_version": "1.13.0", "python_version": "3.9.13"}' True b'cluster'
2022-07-06 16:46:31,096	DEBUG gcs_utils.py:207 -- internal_kv_get b'dashboard' b'dashboard'
2022-07-06 16:46:31,202	DEBUG gcs_utils.py:207 -- internal_kv_get b'dashboard' b'dashboard'
2022-07-06 16:46:31,307	DEBUG gcs_utils.py:207 -- internal_kv_get b'dashboard' b'dashboard'
2022-07-06 16:46:31,413	DEBUG gcs_utils.py:207 -- internal_kv_get b'dashboard' b'dashboard'
2022-07-06 16:46:31,519	DEBUG gcs_utils.py:207 -- internal_kv_get b'dashboard' b'dashboard'
2022-07-06 16:46:31,520	ERROR services.py:1488 -- Failed to start the dashboard: Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2022-07-06_16-46-30_058837_12938/logs/dashboard.log'
2022-07-06 16:46:31,520	ERROR services.py:1489 -- Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2022-07-06_16-46-30_058837_12938/logs/dashboard.log'

Versions / Dependencies

1.13

Reproduction script


import ray
ray.init()

@ray.remote
def f(x):
	return x * x

futures = [f.remote(i) for i in range(4)]
print(ray.get(futers))  # [0, 1, 4, 9]

macOS, full trace back:

2022-07-06 17:00:23,648	ERROR services.py:1488 -- Failed to start the dashboard: Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2022-07-06_17-00-22_187900_14307/logs/dashboard.log'
2022-07-06 17:00:23,648	ERROR services.py:1489 -- Failed to start the dashboard, return code 1
Failed to read dashboard log: [Errno 2] No such file or directory: '/tmp/ray/session_2022-07-06_17-00-22_187900_14307/logs/dashboard.log'
Traceback (most recent call last):
    with open(dashboard_log, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ray/session_2022-07-06_17-00-22_187900_14307/logs/dashboard.log'

Issue Severity

High: It blocks me from completing my task.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 25 (11 by maintainers)

Most upvoted comments

Ah, got it. pip uninstall grpcio and then conda install grpcio worked for me.

Can you make sure to download grpcio <= 1.49.1?

pip install grpcio == 1.49.1 pip uninstall -y ray pip install -U “ray[default]”

Appears so far to have solved all of the problems I was observing. However, the errors have a very flaky nature. I’ll reply to this thread if I’ll observe the same problems with additional use. Thanks for the help!

@deepwonwoo I think your issue is a separate issue from the dashboard (it is a log monitor). Do you mind creating a separate issue for that?