ray: Cannot Initialize Ray
What happened + What you expected to happen
Hi,
I got the following error when calling ray.init(). The code worked before, and it is likely related to pydantic releasing a new version (2.0) today.
2023-06-30 23:34:20,508 ERROR services.py:1207 -- Failed to start the dashboard , return code 1
2023-06-30 23:34:20,508 ERROR services.py:1232 -- Error should be written to 'dashboard.log' or 'dashboard.err'. We are printing the last 20 lines for you. See 'https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure' to find where the log file is.
2023-06-30 23:34:20,532 ERROR services.py:1276 --
The last 20 lines of /tmp/ray/session_2023-06-30_23-34-17_565773_79053/logs/dashboard.log (it contains the error message from the dashboard):
File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/site-packages/ray/dashboard/modules/log/log_manager.py", line 8, in <module>
from ray.util.state.common import (
File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/site-packages/ray/util/state/__init__.py", line 1, in <module>
from ray.util.state.api import (
File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/site-packages/ray/util/state/api.py", line 17, in <module>
from ray.util.state.common import (
File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/site-packages/ray/util/state/common.py", line 120, in <module>
@dataclass(init=True)
File "/Users/kaiyuy/miniconda3/envs/lean-dojo/lib/python3.9/site-packages/pydantic/dataclasses.py", line 139, in dataclass
assert init is False, 'pydantic.dataclasses.dataclass only supports init=False'
AssertionError: pydantic.dataclasses.dataclass only supports init=False
2023-06-30 23:34:21,181 INFO worker.py:1636 -- Started a local Ray instance.
[2023-06-30 23:34:22,874 E 79053 5475804] core_worker.cc:193: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory
Versions / Dependencies
# packages in environment at /Users/kaiyuy/miniconda3/envs/lean-dojo:
#
# Name Version Build Channel
aiohttp 3.8.4 pypi_0 pypi
aiohttp-cors 0.7.0 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
annotated-types 0.5.0 pypi_0 pypi
async-timeout 4.0.2 pypi_0 pypi
attrs 23.1.0 pypi_0 pypi
blessed 1.20.0 pypi_0 pypi
ca-certificates 2023.05.30 hecd8cb5_0
cachetools 5.3.1 pypi_0 pypi
certifi 2023.5.7 pypi_0 pypi
charset-normalizer 3.1.0 pypi_0 pypi
click 8.1.3 pypi_0 pypi
colorful 0.5.5 pypi_0 pypi
distlib 0.3.6 pypi_0 pypi
filelock 3.12.2 pypi_0 pypi
frozenlist 1.3.3 pypi_0 pypi
google-api-core 2.11.1 pypi_0 pypi
google-auth 2.21.0 pypi_0 pypi
googleapis-common-protos 1.59.1 pypi_0 pypi
gpustat 1.1 pypi_0 pypi
grpcio 1.49.1 pypi_0 pypi
idna 3.4 pypi_0 pypi
jsonschema 4.17.3 pypi_0 pypi
libcxx 14.0.6 h9765a3e_0
libffi 3.4.4 hecd8cb5_0
msgpack 1.0.5 pypi_0 pypi
multidict 6.0.4 pypi_0 pypi
ncurses 6.4 hcec6c5f_0
numpy 1.25.0 pypi_0 pypi
nvidia-ml-py 11.525.131 pypi_0 pypi
opencensus 0.11.2 pypi_0 pypi
opencensus-context 0.1.3 pypi_0 pypi
openssl 3.0.9 hca72f7f_0
packaging 23.1 pypi_0 pypi
pip 23.1.2 py39hecd8cb5_0
platformdirs 3.8.0 pypi_0 pypi
prometheus-client 0.17.0 pypi_0 pypi
protobuf 4.23.3 pypi_0 pypi
psutil 5.9.5 pypi_0 pypi
py-spy 0.3.14 pypi_0 pypi
pyasn1 0.5.0 pypi_0 pypi
pyasn1-modules 0.3.0 pypi_0 pypi
pydantic 2.0 pypi_0 pypi
pydantic-core 2.0.1 pypi_0 pypi
pyrsistent 0.19.3 pypi_0 pypi
python 3.9.16 h5ee71fb_3
ray 2.5.1 pypi_0 pypi
readline 8.2 hca72f7f_0
requests 2.31.0 pypi_0 pypi
rsa 4.9 pypi_0 pypi
setuptools 67.8.0 py39hecd8cb5_0
six 1.16.0 pypi_0 pypi
smart-open 6.3.0 pypi_0 pypi
sqlite 3.41.2 h6c40b1e_0
tk 8.6.12 h5d9f67b_0
typing-extensions 4.7.0 pypi_0 pypi
tzdata 2023c h04d1e81_0
urllib3 1.26.16 pypi_0 pypi
virtualenv 20.21.0 pypi_0 pypi
wcwidth 0.2.6 pypi_0 pypi
wheel 0.38.4 py39hecd8cb5_0
xz 5.4.2 h6c40b1e_0
yarl 1.9.2 pypi_0 pypi
zlib 1.2.13 h4dc903c_0
Reproduction script
import ray
ray.init()
Issue Severity
High: It blocks me from completing my task.
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 3
- Comments: 16 (11 by maintainers)
Commits related to this issue
- pydantic > 2 conflicts with ray (#938) Set "pydantic < 2", https://github.com/ray-project/ray/issues/37019. --------- Co-authored-by: Gaurav Tarlok Kakkar <gaurav21776@gmail.com> — committed to georgia-tech-db/evadb by xzdandy 10 months ago
- [CHORE] drop ray default dep to make room for Pydantic > 2.0 (#1618) * Drop ray default feature due to the fact that pyiceberg requires pydantic > 2.0 * for context: https://github.com/ray-project/... — committed to Eventual-Inc/Daft by samster25 7 months ago
@martalist we’ve run into a number of issues related to serialization in order to support Pydantic 2 in Ray Serve. We are currently targeting the 2.8 release to address these.
We’ve merged the changes to make Ray compatible with Pydantic 2.5+. You can start using Pydantic 2.5+ with Ray 2.9, which should be out at the end of December.
These changes should also be in the Ray nightly, so feel free to try them out!
@alanwguo it’s now supported, as of FastAPI v0.100.0. Pydantic 2 seems to have significant performance improvements. Do you know if Ray Serve will be updated anytime soon?
For everyone affected by this issue, can you try to use pydantic 1.x?
pip install "pydantic<2"?Thanks for the report! This is indeed very likely related to the pydantic 2 release, see https://github.com/ray-project/ray/issues/36990
As a workaround, try to run
pip install "pydantic<2"– does that work for you?We already have a fix https://github.com/ray-project/ray/pull/37000 that will be released in Ray 2.6