nbdev: `nbdev_prepare` fails with `concurrent.futures.process.BrokenProcessPool` exception

nbdev_prepare is giving me a concurrent.futures.process.BrokenProcessPool on a new project.

Environment

Python: 3.8.13
nbdev: 2.0.1
OS: macOS Monterey 12.4
Processor: Apple M1

Steps to produce this error

gh repo create # then followed prompts
cd nbdev_test # the name of the repo I made
nbev_new
nbdev_install_hooks
nbdev_prepare

Error message

objc[25641]: +[NSMutableString initialize] may have been in progress in another thread when fork() was called.
objc[25641]: +[NSMutableString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
Traceback (most recent call last):
  File "/Users/michael.fishman/miniconda3/envs/ml8/bin/nbdev_prepare", line 8, in <module>
    sys.exit(prepare())
  File "/Users/michael.fishman/miniconda3/envs/ml8/lib/python3.8/site-packages/nbdev/shortcuts.py", line 98, in prepare
    _c(nbdev_test)
  File "/Users/michael.fishman/miniconda3/envs/ml8/lib/python3.8/site-packages/nbdev/shortcuts.py", line 20, in _c
    def _c(f, *args, **kwargs): return f.__wrapped__(*args, **kwargs)
  File "/Users/michael.fishman/miniconda3/envs/ml8/lib/python3.8/site-packages/nbdev/test.py", line 86, in nbdev_test
    results = parallel(test_nb, files, skip_flags=skip_flags, force_flags=force_flags, n_workers=n_workers, pause=pause, do_print=do_print)
  File "/Users/michael.fishman/miniconda3/envs/ml8/lib/python3.8/site-packages/fastcore/parallel.py", line 117, in parallel
    return L(r)
  File "/Users/michael.fishman/miniconda3/envs/ml8/lib/python3.8/site-packages/fastcore/foundation.py", line 98, in __call__
    return super().__call__(x, *args, **kwargs)
  File "/Users/michael.fishman/miniconda3/envs/ml8/lib/python3.8/site-packages/fastcore/foundation.py", line 106, in __init__
    items = listify(items, *rest, use_list=use_list, match=match)
  File "/Users/michael.fishman/miniconda3/envs/ml8/lib/python3.8/site-packages/fastcore/basics.py", line 63, in listify
    elif is_iter(o): res = list(o)
  File "/Users/michael.fishman/miniconda3/envs/ml8/lib/python3.8/concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/Users/michael.fishman/miniconda3/envs/ml8/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/Users/michael.fishman/miniconda3/envs/ml8/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/Users/michael.fishman/miniconda3/envs/ml8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 24 (20 by maintainers)

Most upvoted comments

After the recent fix you can add back:

#|export
from __future__ import annotations
import warnings

from fastcore.net import * <-------- this import

from nbdev.read import *
from nbdev.sync import *
from nbdev.process import *
from nbdev.processors import *
from nbdev.doclinks import *

from execnb.nbio import *
from fastcore.utils import *
from fastcore.script import call_parse
from fastcore import shutil

from urllib.error import HTTPError
from contextlib import redirect_stdout
import os, tarfile, subprocess, sys

to 10_cli.ipynb, then export it, and run prepare, you should get the error.

(Strangely adding from fastcore.parallel import * doesn’t trigger the issue… which was my original hypothesis that .net was importing .parallel which was the root cause)

Btw my Python version is 3.9.13…