spack: MacOS Mojave, Python 3.8: "AttributeError: Can't pickle local object 'fork..child_process'"

I tried to install a package on MacOS Mojave using Python 3.8 installed from MacPorts. I got an error related to "pickle"ing an object.

The issue does not reproduce with Python 3.7. It reproduces on the latest commit of develop.

Steps to reproduce the issue

$ sudo port select --set python3 python38
$ spack install <package>

Error Message

$ spack -d install googletest%clang
==> [2019-12-10-16:40:53.367136] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/modules.yaml
==> [2019-12-10-16:40:53.375856] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/darwin/modules.yaml
==> [2019-12-10-16:40:53.378937] Reading config file /Users/agaspar/.spack/darwin/modules.yaml
==> [2019-12-10-16:40:53.382033] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/config.yaml
==> [2019-12-10-16:40:53.402445] Imported install from built-in commands
==> [2019-12-10-16:40:53.403393] Imported install from built-in commands
==> [2019-12-10-16:40:53.423052] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/repos.yaml
==> [2019-12-10-16:40:53.426926] Reading config file /Users/agaspar/.spack/darwin/repos.yaml
==> [2019-12-10-16:40:53.590667] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/packages.yaml
==> [2019-12-10-16:40:53.602863] Reading config file /Users/agaspar/Code/spack/etc/spack/defaults/darwin/packages.yaml
==> [2019-12-10-16:40:53.609408] Reading config file /Users/agaspar/.spack/packages.yaml
==> [2019-12-10-16:40:53.632981] WRITE LOCK: /Users/agaspar/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:53.633665] WRITE LOCK: /Users/agaspar/.spack/cache/providers/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.187669] WRITE LOCK: /Users/agaspar/.spack/cache/providers/.builtin-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.191428] WRITE LOCK: /Users/agaspar/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.192032] WRITE LOCK: /Users/agaspar/.spack/cache/tags/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.320890] WRITE LOCK: /Users/agaspar/.spack/cache/tags/.builtin-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.325895] WRITE LOCK: /Users/agaspar/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.326356] WRITE LOCK: /Users/agaspar/.spack/cache/patches/.builtin-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.913941] WRITE LOCK: /Users/agaspar/.spack/cache/patches/.builtin-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.920255] READ LOCK: /Users/agaspar/.spack/cache/providers/.eap-cpp-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.920748] READ LOCK: /Users/agaspar/.spack/cache/providers/.eap-cpp-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.921247] READ LOCK: /Users/agaspar/.spack/cache/providers/.eap-cpp-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.921705] READ LOCK: /Users/agaspar/.spack/cache/tags/.eap-cpp-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.921978] READ LOCK: /Users/agaspar/.spack/cache/tags/.eap-cpp-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.922417] READ LOCK: /Users/agaspar/.spack/cache/tags/.eap-cpp-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.922960] READ LOCK: /Users/agaspar/.spack/cache/patches/.eap-cpp-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.923237] READ LOCK: /Users/agaspar/.spack/cache/patches/.eap-cpp-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.931963] READ LOCK: /Users/agaspar/.spack/cache/patches/.eap-cpp-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.933605] READ LOCK: /Users/agaspar/.spack/cache/providers/.agaspar-spack-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.934507] READ LOCK: /Users/agaspar/.spack/cache/providers/.agaspar-spack-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.935006] READ LOCK: /Users/agaspar/.spack/cache/providers/.agaspar-spack-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.935556] READ LOCK: /Users/agaspar/.spack/cache/tags/.agaspar-spack-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.935896] READ LOCK: /Users/agaspar/.spack/cache/tags/.agaspar-spack-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.936364] READ LOCK: /Users/agaspar/.spack/cache/tags/.agaspar-spack-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.936862] READ LOCK: /Users/agaspar/.spack/cache/patches/.agaspar-spack-repo-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.937520] READ LOCK: /Users/agaspar/.spack/cache/patches/.agaspar-spack-repo-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.937955] READ LOCK: /Users/agaspar/.spack/cache/patches/.agaspar-spack-repo-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.938516] READ LOCK: /Users/agaspar/.spack/cache/providers/.sds-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.938796] READ LOCK: /Users/agaspar/.spack/cache/providers/.sds-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.939228] READ LOCK: /Users/agaspar/.spack/cache/providers/.sds-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.939772] READ LOCK: /Users/agaspar/.spack/cache/tags/.sds-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.940071] READ LOCK: /Users/agaspar/.spack/cache/tags/.sds-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.940524] READ LOCK: /Users/agaspar/.spack/cache/tags/.sds-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.941433] READ LOCK: /Users/agaspar/.spack/cache/patches/.sds-index.json.lock[0:0] [Acquiring]
==> [2019-12-10-16:40:59.941899] READ LOCK: /Users/agaspar/.spack/cache/patches/.sds-index.json.lock[0:0] [Acquired]
==> [2019-12-10-16:40:59.942576] READ LOCK: /Users/agaspar/.spack/cache/patches/.sds-index.json.lock[0:0] [Released]
==> [2019-12-10-16:40:59.947954] Reading config file /Users/agaspar/.spack/darwin/compilers.yaml
==> [2019-12-10-16:41:00.015384] Warning: clang@11.0.0-apple cannot build optimized binaries for "skylake". Using best target possible: "x86_64"
==> [2019-12-10-16:41:00.066119] DATABASE LOCK TIMEOUT: 120s
==> [2019-12-10-16:41:00.066246] PACKAGE LOCK TIMEOUT: No timeout
==> [2019-12-10-16:41:00.069298] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.070192] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.116237] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.118529] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.119082] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.165906] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.167462] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.167934] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.218965] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.219271] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.219655] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.266983] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.267310] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.267728] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.477182] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.484131] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquiring]
==> [2019-12-10-16:41:00.485073] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquired]
==> [2019-12-10-16:41:00.485185] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.485539] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.538750] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.539044] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.539420] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.585088] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.588744] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Released]
==> [2019-12-10-16:41:00.598045] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquiring]
==> [2019-12-10-16:41:00.598586] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Acquired]
==> [2019-12-10-16:41:00.598719] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.599106] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.645810] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.646311] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/prefix_lock[5820083178631369124:1] [Released]
==> [2019-12-10-16:41:00.646450] Installing googletest dependencies
==> [2019-12-10-16:41:00.646715] cmake@3.15.1 : externally installed in /opt/local
==> [2019-12-10-16:41:00.646790] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.647079] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.693385] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.693685] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.694108] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.740314] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.740884] cmake@3.15.1 : generating module file
==> [2019-12-10-16:41:00.740948] SKIP: shebang filtering [external package]
==> [2019-12-10-16:41:00.741077] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.741426] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.787969] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.788322] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.788725] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.834796] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:00.842742]        WRITE: cmake@3.15.1%clang@11.0.0-apple~doc+ncurses+openssl+ownlibs~qt arch=darwin-mojave-x86_64/6u2g2jh [/Users/agaspar/Code/spack/share/spack/modules/darwin-mojave-x86_64/cmake-3.15.1-clang-11.0.0-apple-6u2g2jh]
==> [2019-12-10-16:41:00.864711] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:00.865189] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:00.933812] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:01.114793] cmake@3.15.1 : registering into DB
==> [2019-12-10-16:41:01.114944] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:01.115530] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:01.164461] WRITE LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:01.214115] Bootstrapping clang@11.0.0-apple compiler for googletest
==> [2019-12-10-16:41:01.214736] Installing googletest
==> [2019-12-10-16:41:01.214778] Searching for binary cache of googletest
==> [2019-12-10-16:41:01.214992] No Spack mirrors are currently configured
==> [2019-12-10-16:41:01.215405] No binary for googletest found: installing from source
==> [2019-12-10-16:41:01.215535] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:01.215997] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:01.262544] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
==> [2019-12-10-16:41:01.280533] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquiring]
==> [2019-12-10-16:41:01.282336] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Acquired]
==> [2019-12-10-16:41:01.533037] READ LOCK: /Users/agaspar/Code/spack/opt/spack/.spack-db/lock[0:0] [Released]
Traceback (most recent call last):
  File "/Users/agaspar/Code/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/Users/agaspar/Code/spack/lib/spack/spack/main.py", line 715, in main
    return _invoke_command(command, parser, args, unknown)
  File "/Users/agaspar/Code/spack/lib/spack/spack/main.py", line 457, in _invoke_command
    return_val = command(parser, args)
  File "/Users/agaspar/Code/spack/lib/spack/spack/cmd/install.py", line 331, in install
    install_spec(args, kwargs, abstract, concrete)
  File "/Users/agaspar/Code/spack/lib/spack/spack/cmd/install.py", line 209, in install_spec
    spec.package.do_install(**kwargs)
  File "/Users/agaspar/Code/spack/lib/spack/spack/package.py", line 1761, in do_install
    PackageBase._verbose = spack.build_environment.fork(
  File "/Users/agaspar/Code/spack/lib/spack/spack/build_environment.py", line 860, in fork
    p.start()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'fork.<locals>.child_process'

Information on your system

This includes:

  1. MacOS Mojave
  2. Python 3.8 from MacPorts

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 3
  • Comments: 25 (25 by maintainers)

Commits related to this issue

Most upvoted comments

Oddly enough, I’ve been using Spack on Ubuntu 20.04 (via WSL) which comes with Python 3.8.2 but I haven’t encountered a single problem yet. I already have 160+ packages successfully installed.

Spack should work fine with Python 3.8 in Linux, but not on Mac OS, because Python 3.8 multiprocessing behavior diverged for Mac OS and Linux.

I’m starting to experience this issue as well.

This is about to get bigger, since macOS Big Sur by Apple now ships with Python 3.8 by default, and most brew formulae has updated to Python 3.8 Space is still broken for me 😕

I will need to work on it more over the next day or two.

Update: I’ve resolved some additional issues but more remain. I will be continuing with this in January.

The sooner this gets resolved, the sooner I can start recommending spack to mac people again…

I can replicate this using a Spack-installed Python 3.8.0 on Mac OS. I think the problem is that the implementation of Spack install wasn’t following the requirements for pickling: in particular, there are locally-defined that are passed to multiprocessing.Process.

I can fix the initial error with the changes at https://github.com/spack/spack/pull/14187 but that does not fully resolve this issue so I will need to work on it more over the next day or two.

See also:

@sethrj This solution doesn’t seem ideal, if I remove /usr/local/bin from my path, then I can’t access my Homebrew tools

Just hit this with a fresh Catalina install using homebrew. Worked around by removing /usr/local/bin from my PATH:

export PATH=/usr/bin:/bin:/usr/sbin:/sbin:$PATH

@adamjstewart I am taking a look at it myself, I am going to try to fix it, btw homebrew already switched.

@scheibelp we should bump this up in priority. Spack, MacPorts, and Anaconda all offer Python 3.8 installations. From what I can tell, the only reason Anaconda doesn’t default to Python 3.8 is that a few packages still don’t support it (TensorFlow being the big one), so as soon as that changes, I think the default version of Python any Anaconda user will have will be Python 3.8. I was also planning on switching Spack’s default Python version to 3.8 once Anaconda makes the switch. Homebrew will likely follow.

I figured I’ll just add a (not very helpful) “I’m having this problem, too” comment here, with current master and also python 3.8.