spack: MacOS Mojave, Python 3.8: "AttributeError: Can't pickle local object 'fork..child_process'"
I tried to install a package on MacOS Mojave using Python 3.8 installed from MacPorts. I got an error related to "pickle"ing an object.
The issue does not reproduce with Python 3.7. It reproduces on the latest commit of develop.
Steps to reproduce the issue
$ sudo port select --set python3 python38
$ spack install <package>
Error Message
$ spack -d install googletest%clang
==> [2019-12-10-16:40:53.367136] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/modules.yaml
==> [2019-12-10-16:40:53.375856] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/darwin/modules.yaml
==> [2019-12-10-16:40:53.378937] Reading config file /Users/agaspar/.spack/darwin/modules.yaml
==> [2019-12-10-16:40:53.382033] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/config.yaml
==> [2019-12-10-16:40:53.402445] Imported install from built-in commands
==> [2019-12-10-16:40:53.403393] Imported install from built-in commands
==> [2019-12-10-16:40:53.423052] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/repos.yaml
==> [2019-12-10-16:40:53.426926] Reading config file /Users/agaspar/.spack/darwin/repos.yaml
==> [2019-12-10-16:40:53.590667] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/packages.yaml
==> [2019-12-10-16:40:53.602863] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/darwin/packages.yaml
==> [2019-12-10-16:40:53.609408] Reading config file /Users/agaspar/.spack/packages.yaml
==> [2019-12-10-16:40:53.632981] WRITE LOCK: /Users/agaspar/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:53.633665] WRITE LOCK: /Users/agaspar/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.187669] WRITE LOCK: /Users/agaspar/.spack/cache/providers/.builtin-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.191428] WRITE LOCK: /Users/agaspar/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.192032] WRITE LOCK: /Users/agaspar/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.320890] WRITE LOCK: /Users/agaspar/.spack/cache/tags/.builtin-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.325895] WRITE LOCK: /Users/agaspar/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.326356] WRITE LOCK: /Users/agaspar/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.913941] WRITE LOCK: /Users/agaspar/.spack/cache/patches/.builtin-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.920255] READ LOCK: /Users/agaspar/.spack/cache/providers/.eap-cpp-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.920748] READ LOCK: /Users/agaspar/.spack/cache/providers/.eap-cpp-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.921247] READ LOCK: /Users/agaspar/.spack/cache/providers/.eap-cpp-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.921705] READ LOCK: /Users/agaspar/.spack/cache/tags/.eap-cpp-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.921978] READ LOCK: /Users/agaspar/.spack/cache/tags/.eap-cpp-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.922417] READ LOCK: /Users/agaspar/.spack/cache/tags/.eap-cpp-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.922960] READ LOCK: /Users/agaspar/.spack/cache/patches/.eap-cpp-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.923237] READ LOCK: /Users/agaspar/.spack/cache/patches/.eap-cpp-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.931963] READ LOCK: /Users/agaspar/.spack/cache/patches/.eap-cpp-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.933605] READ LOCK: /Users/agaspar/.spack/cache/providers/.agaspar-spack-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.934507] READ LOCK: /Users/agaspar/.spack/cache/providers/.agaspar-spack-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.935006] READ LOCK: /Users/agaspar/.spack/cache/providers/.agaspar-spack-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.935556] READ LOCK: /Users/agaspar/.spack/cache/tags/.agaspar-spack-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.935896] READ LOCK: /Users/agaspar/.spack/cache/tags/.agaspar-spack-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.936364] READ LOCK: /Users/agaspar/.spack/cache/tags/.agaspar-spack-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.936862] READ LOCK: /Users/agaspar/.spack/cache/patches/.agaspar-spack-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.937520] READ LOCK: /Users/agaspar/.spack/cache/patches/.agaspar-spack-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.937955] READ LOCK: /Users/agaspar/.spack/cache/patches/.agaspar-spack-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.938516] READ LOCK: /Users/agaspar/.spack/cache/providers/.sds-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.938796] READ LOCK: /Users/agaspar/.spack/cache/providers/.sds-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.939228] READ LOCK: /Users/agaspar/.spack/cache/providers/.sds-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.939772] READ LOCK: /Users/agaspar/.spack/cache/tags/.sds-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.940071] READ LOCK: /Users/agaspar/.spack/cache/tags/.sds-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.940524] READ LOCK: /Users/agaspar/.spack/cache/tags/.sds-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.941433] READ LOCK: /Users/agaspar/.spack/cache/patches/.sds-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.941899] READ LOCK: /Users/agaspar/.spack/cache/patches/.sds-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.942576] READ LOCK: /Users/agaspar/.spack/cache/patches/.sds-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.947954] Reading config file /Users/agaspar/.spack/darwin/compilers.yaml
==> [2019-12-10-16:41:00.015384] Warning: clang@11.0.0-apple cannot build optimized binaries for "skylake". Using best target possible: "x86_64"
==> [2019-12-10-16:41:00.066119] DATABASE LOCK TIMEOUT: 120s
==> [2019-12-10-16:41:00.066246] PACKAGE LOCK TIMEOUT: No timeout
==> [2019-12-10-16:41:00.069298] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.070192] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.116237] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.118529] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.119082] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.165906] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.167462] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.167934] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.218965] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.219271] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.219655] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.266983] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.267310] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.267728] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.477182] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.484131] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquiring]
==> [2019-12-10-16:41:00.485073] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquired]
==> [2019-12-10-16:41:00.485185] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.485539] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.538750] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.539044] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.539420] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.585088] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.588744] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Released]
==> [2019-12-10-16:41:00.598045] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquiring]
==> [2019-12-10-16:41:00.598586] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquired]
==> [2019-12-10-16:41:00.598719] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.599106] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.645810] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.646311] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Released]
==> [2019-12-10-16:41:00.646450] Installing googletest dependencies
==> [2019-12-10-16:41:00.646715] cmake@3.15.1 : externally installed in /opt/local
==> [2019-12-10-16:41:00.646790] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.647079] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.693385] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.693685] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.694108] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.740314] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.740884] cmake@3.15.1 : generating module file
==> [2019-12-10-16:41:00.740948] SKIP: shebang filtering [external package]
==> [2019-12-10-16:41:00.741077] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.741426] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.787969] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.788322] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.788725] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.834796] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.842742] WRITE: cmake@3.15.1%clang@11.0.0-apple~doc+ncurses+openssl+ownlibs~qt arch=darwin-mojave-x86_64/6u2g2jh [/Users/agaspar/Code/spack/share/spack/modules/darwin-mojave-x86_64/cmake-3.15.1-clang-11.0.0-apple-6u2g2jh]
==> [2019-12-10-16:41:00.864711] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.865189] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.933812] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:01.114793] cmake@3.15.1 : registering into DB
==> [2019-12-10-16:41:01.114944] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:01.115530] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:01.164461] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:01.214115] Bootstrapping clang@11.0.0-apple compiler for googletest
==> [2019-12-10-16:41:01.214736] Installing googletest
==> [2019-12-10-16:41:01.214778] Searching for binary cache of googletest
==> [2019-12-10-16:41:01.214992] No Spack mirrors are currently configured
==> [2019-12-10-16:41:01.215405] No binary for googletest found: installing from source
==> [2019-12-10-16:41:01.215535] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:01.215997] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:01.262544] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:01.280533] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:01.282336] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:01.533037] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
Traceback (most recent call last):
File "/Users/agaspar/Code/spack/bin/spack", line 64, in <module>
sys.exit(spack.main.main())
File "/Users/agaspar/Code/spack/lib/spack/spack/main.py", line 715, in main
return _invoke_command(command, parser, args, unknown)
File "/Users/agaspar/Code/spack/lib/spack/spack/main.py", line 457, in _invoke_command
return_val = command(parser, args)
File "/Users/agaspar/Code/spack/lib/spack/spack/cmd/install.py", line 331, in install
install_spec(args, kwargs, abstract, concrete)
File "/Users/agaspar/Code/spack/lib/spack/spack/cmd/install.py", line 209, in install_spec
spec.package.do_install(**kwargs)
File "/Users/agaspar/Code/spack/lib/spack/spack/package.py", line 1761, in do_install
PackageBase._verbose = spack.build_environment.fork(
File "/Users/agaspar/Code/spack/lib/spack/spack/build_environment.py", line 860, in fork
p.start()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'fork.<locals>.child_process'
Information on your system
This includes:
- MacOS Mojave
- Python 3.8 from MacPorts
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 3
- Comments: 25 (25 by maintainers)
Commits related to this issue
- MacOS nightly builds: use Python 3.7 in CI Nightly builds with MacOS started failing again due to an upgrade of the default virtual environment that now uses Python 3.8 This makes us hit #14102 and ... — committed to alalazo/spack by alalazo 6 years ago
- MacOS nightly builds: use Python 3.7 in CI Nightly builds with MacOS started failing again due to an upgrade of the default virtual environment that now uses Python 3.8 This makes us hit #14102 and ... — committed to spack/spack by alalazo 6 years ago
- MacOS nightly builds: use Python 3.7 in CI Nightly builds with MacOS started failing again due to an upgrade of the default virtual environment that now uses Python 3.8 This makes us hit #14102 and ... — committed to spack/spack by alalazo 6 years ago
- MacOS nightly builds: use Python 3.7 in CI Nightly builds with MacOS started failing again due to an upgrade of the default virtual environment that now uses Python 3.8 This makes us hit #14102 and ... — committed to spack/spack by alalazo 6 years ago
- MacOS nightly builds: use Python 3.7 in CI Nightly builds with MacOS started failing again due to an upgrade of the default virtual environment that now uses Python 3.8 This makes us hit #14102 and ... — committed to spack/spack by alalazo 6 years ago
- macos: update build process to use spawn instead of fork (#18205) Spack creates a separate process to do package installation. Different operating systems and Python versions use different methods t... — committed to spack/spack by scheibelp 6 years ago
- Fix copy manager functionality so it works under Python >= 3.8 on OS X. In Python 3.8, local objects can't be pickled anymore which means we need to move that class definition to a top module level s... — committed to scalyr/scalyr-agent-2 by tomaz-s1 5 years ago
- Fix copy manager functionality so it works under Python >= 3.8 on OS X (#818) * Fix copy manager functionality so it works under Python >= 3.8 on OS X. In Python 3.8, local objects can't be pickle... — committed to scalyr/scalyr-agent-2 by Kami 5 years ago
Spack should work fine with Python 3.8 in Linux, but not on Mac OS, because Python 3.8 multiprocessing behavior diverged for Mac OS and Linux.
I’m starting to experience this issue as well.
This is about to get bigger, since macOS Big Sur by Apple now ships with Python 3.8 by default, and most brew formulae has updated to Python 3.8 Space is still broken for me 😕
Update: I’ve resolved some additional issues but more remain. I will be continuing with this in January.
The sooner this gets resolved, the sooner I can start recommending spack to mac people again…
I can replicate this using a Spack-installed Python 3.8.0 on Mac OS. I think the problem is that the implementation of Spack install wasn’t following the requirements for pickling: in particular, there are locally-defined that are passed to
multiprocessing.Process.I can fix the initial error with the changes at https://github.com/spack/spack/pull/14187 but that does not fully resolve this issue so I will need to work on it more over the next day or two.
See also:
@sethrj This solution doesn’t seem ideal, if I remove /usr/local/bin from my path, then I can’t access my Homebrew tools
Just hit this with a fresh Catalina install using homebrew. Worked around by removing
/usr/local/binfrom my PATH:@adamjstewart I am taking a look at it myself, I am going to try to fix it, btw homebrew already switched.
@scheibelp we should bump this up in priority. Spack, MacPorts, and Anaconda all offer Python 3.8 installations. From what I can tell, the only reason Anaconda doesn’t default to Python 3.8 is that a few packages still don’t support it (TensorFlow being the big one), so as soon as that changes, I think the default version of Python any Anaconda user will have will be Python 3.8. I was also planning on switching Spack’s default Python version to 3.8 once Anaconda makes the switch. Homebrew will likely follow.
I figured I’ll just add a (not very helpful) “I’m having this problem, too” comment here, with current
masterand also python 3.8.