ray: [Core] ray.exceptions.LocalRayletDiedError: The task's local raylet died. Check raylet.out for more information.

What happened + What you expected to happen

I was using ray as a dependency to run the Modin library in Win 11 however I am getting Raylet error

Versions / Dependencies

OS: Windows 11 Python: 3.7 Ray Version: 1.12.0

Ray agent failed dashboard.log dashboard_agent.log

Reproduction script

df.head(20):

Unnamed: 0                       game score home_odds draw_odds away_odds country                 league             datetime
0            0  Sport Recife - Imperatriz   2:2      1.36      4.31      7.66  Brazil  Copa do Nordeste 2020  2020-02-07 00:00:00
1            1           ABC - America RN   2:1      2.62      3.30      2.48  Brazil  Copa do Nordeste 2020  2020-02-02 22:00:00
2            2  Frei Paulistano - Nautico   0:2      5.19      3.58      1.62  Brazil  Copa do Nordeste 2020  2020-02-02 00:00:00
3            3    Botafogo PB - Confianca   1:1      2.06      3.16       3.5  Brazil  Copa do Nordeste 2020  2020-02-02 22:00:00
4            4          Fortaleza - Ceara   1:1      2.19      2.98      3.38  Brazil  Copa do Nordeste 2020  2020-02-02 22:00:00
5            5             River-PI - CSA   3:1      3.41      3.07       2.1  Brazil  Copa do Nordeste 2020  2020-02-02 20:00:00
6            6     Sport Recife - Vitoria   1:1      1.92      3.20      3.93  Brazil  Copa do Nordeste 2020  2020-02-02 20:00:00
7            7           CRB - Santa Cruz   1:0      2.22      3.06      3.18  Brazil  Copa do Nordeste 2020  2020-01-29 23:30:00
8            8         Bahia - Imperatriz   2:0      1.31      4.75      8.55  Brazil  Copa do Nordeste 2020  2020-01-29 00:00:00
9            9    Ceara - Frei Paulistano   2:2       1.2      5.43     12.94  Brazil  Copa do Nordeste 2020  2020-01-26 22:00:00
10          10            Confianca - ABC   1:0      2.09      3.01      3.58  Brazil  Copa do Nordeste 2020  2020-01-26 00:00:00
11          11   America RN - Botafogo PB   0:0      2.14      3.30      3.11  Brazil  Copa do Nordeste 2020  2020-01-26 22:00:00
12          12         CSA - Sport Recife   0:1      2.63      3.04      2.62  Brazil  Copa do Nordeste 2020  2020-01-26 22:00:00
13          13         Santa Cruz - Bahia   0:0      3.72      3.22      1.96  Brazil  Copa do Nordeste 2020  2020-01-26 20:00:00
14          14        Vitoria - Fortaleza   0:0      3.13      3.02      2.27  Brazil  Copa do Nordeste 2020  2020-01-26 20:00:00
15          15         Nautico - River-PI   1:1      1.44      4.06      6.19  Brazil  Copa do Nordeste 2020  2020-01-23 23:30:00
16          16           Imperatriz - CRB   2:1      2.34      2.94      3.08  Brazil  Copa do Nordeste 2020  2020-01-21 23:00:00
17          17       Nautico - Campinense   2:0       1.4      4.18      7.26  Brazil  Copa do Nordeste 2020  2019-05-16 00:30:00
18          18          CRB - Juazeirense   2:1       1.4      4.19      7.32  Brazil  Copa do Nordeste 2020  2019-05-14 23:00:00
19          19             ABC - AE Altos   2:0      1.44      3.83      7.02  Brazil  Copa do Nordeste 2020  2019-05-08 22:30:00
import os
import ray
ray.init()
os.environ["MODIN_ENGINE"] = "ray"
import modin.pandas as pd
df = pd.read_csv(r"data.csv", low_memory=False)
df['game'] = df['game'].astype(str).str.replace('(\(\w+\))', '', regex=True)
df['league'] = df['league'].astype(str).str.replace('(\s\d+\S\d+)$', '', regex=True)
df['league'] = df['league'].astype(str).str.replace('é', 'e', regex=True)
df['game'] = df['game'].astype(str).str.replace('(\s\d+\S\d+)$', '', regex=True)
df[['home_team', 'away_team']] = df['game'].str.split(' - ', expand=True, n=1)
df[['home_score', 'away_score']] = df['score'].str.split(':', expand=True)
df['away_score'] = df['away_score'].astype(str).str.replace('[a-zA-Z\s\D]', '', regex=True)
print (df.head())

Issue Severity

High: It blocks me from completing my task.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 38 (23 by maintainers)

Most upvoted comments

xref #24905, which has a pure-python fix

Update: Observed only with virtualenv.

OK: py 3.7.9, python repro.py OK: py 3.9.7, python repro.py NG: py 3.7.9, .venv\Scripts\python.exe repro.py NG: py 3.9.7, .venv\Scripts\python.exe repro.py NG: py 3.7.9, .venv\Scripts\activate && python repro.py

it may be related to PYTHONPATH etc., as suggested in #23945 .

@cexen When you run inside a virtualenv, are you activating it first?

.venv\Scripts\activate
.venv\Scripts\python.exe repro.py

i see it failing both when using the client directly and when launching via ray submit

@mbehrendt @cardchase the error message refers to /tmp/ray/session_latest/logs/raylet.out. Are you able to fetch that file to see the error?

Finally, one more question for @mbehrendt – if you re-run the script with no workers at all (max_workers=0), what’s the behavior? Does the head’s Raylet crash?