joblib: Parallel crashed on python 3.6 when running in vscode debugger

My environment is python 3.6 and I ran a simple script file like below:

from math import sqrt
from joblib import Parallel, delayed


def main():
    print('simulate workers, testing parallel')
    Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))


if __name__ == '__main__':
    print('simulate workers, testing parallel')
    Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))

And facing a crash:

Below is the log details: E00001.438: Exception escaped from start_client

        Traceback (most recent call last):
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/log.py", line 110, in g
            return f(*args, **kwargs)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/pydevd_hooks.py", line 74, in start_client
            sock, start_session = daemon.start_client((host, port))
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 214, in start_client
            with self.started():
          File "/Users/fans3210/anaconda3/envs/assignQuestion-py36/lib/python3.6/contextlib.py", line 81, in __enter__
            return next(self.gen)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 110, in started
            self.start()
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 145, in start
            raise RuntimeError('already started')
        RuntimeError: already started

/Users/fans3210/anaconda3/envs/assignQuestion-py36/lib/python3.6/site-packages/joblib/externals/loky/backend/semaphore_tracker.py:74: UserWarning: semaphore_tracker: process died unexpectedly, relaunching. Some semaphores might leak. warnings.warn('semaphore_tracker: process died unexpectedly, ’ E00001.467: Exception escaped from start_client

        Traceback (most recent call last):
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/log.py", line 110, in g
            return f(*args, **kwargs)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/pydevd_hooks.py", line 74, in start_client
            sock, start_session = daemon.start_client((host, port))
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 214, in start_client
            with self.started():
          File "/Users/fans3210/anaconda3/envs/assignQuestion-py36/lib/python3.6/contextlib.py", line 81, in __enter__
            return next(self.gen)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 110, in started
            self.start()
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 145, in start
            raise RuntimeError('already started')
        RuntimeError: already started

E00001.481: Exception escaped from start_client

        Traceback (most recent call last):
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/log.py", line 110, in g
            return f(*args, **kwargs)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/pydevd_hooks.py", line 74, in start_client
            sock, start_session = daemon.start_client((host, port))
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 214, in start_client
            with self.started():
          File "/Users/fans3210/anaconda3/envs/assignQuestion-py36/lib/python3.6/contextlib.py", line 81, in __enter__
            return next(self.gen)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 110, in started
            self.start()
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 145, in start
            raise RuntimeError('already started')
        RuntimeError: already started

E00001.515: Exception escaped from start_client

        Traceback (most recent call last):
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/log.py", line 110, in g
            return f(*args, **kwargs)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/pydevd_hooks.py", line 74, in start_client
            sock, start_session = daemon.start_client((host, port))
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 214, in start_client
            with self.started():
          File "/Users/fans3210/anaconda3/envs/assignQuestion-py36/lib/python3.6/contextlib.py", line 81, in __enter__
            return next(self.gen)
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 110, in started
            self.start()
          File "/Users/fans3210/.vscode/extensions/ms-python.python-2019.3.6215/pythonFiles/lib/python/ptvsd/daemon.py", line 145, in start
            raise RuntimeError('already started')
        RuntimeError: already started

The only solution I can use is to set" “prefer = ‘threads’” in the function call. Appreciated if someone could help!

About this issue

Original URL
State: open
Created 5 years ago
Reactions: 13
Comments: 19 (1 by maintainers)

Most upvoted comments

Hey, I saw a solution by amittleider and it worked for me. Paste it here:

On Ubuntu, putting this at the top of the main entrypoint file worked for me:

import multiprocessing
multiprocessing.set_start_method('spawn', True)

+26

Eralien on Jun 12, 2019

Do the following to avoid the semaphore error:

Write the following piece of code in the beginning of the .py file you are debugging. import multiprocessing multiprocessing.set_start_method('spawn', True)
In the debug configurations file - launch.json, add the following: "subProcess": true, and for Pytorch DataLoader function, set num_worker threads as ‘0’.

Hope this helps you! It worked for me.

anerisheth19 on Dec 9, 2019

Doesn’t work for pytorch.

As far as I remember I had to import a second package, but for me the easy workaround was to set num_workers=0 in the Dataloader constructor.

johannespitz on Nov 28, 2019

Hi all, can people still reproduce this issue with at least the latest supported version of python?

👍-react if you can
👎-react if you can’t

jjerphan on Feb 15, 2023

@anerisheth19 That worked to debug my celery application, thanks.

sergioamr on Dec 17, 2019