salt: [BUG] AttributeError: 'Process' object has no attribute '_args_for_getstate'

It is not working anymore because python formula has been upgraded from 3.7 to 3.8 …

brew info python
python@3.8: stable 3.8.3 (bottled)
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@3.8/3.8.3_2 (4,229 files, 64.6MB)
  Poured from bottle on 2020-07-14 at 14:43:45
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.8.rb
==> Dependencies
Build: pkg-config ✔
Required: gdbm ✔, openssl@1.1 ✔, readline ✔, sqlite ✔, xz ✔
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python@3.8/libexec/bin

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.8/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Analytics
install: 463,840 (30 days), 1,253,709 (90 days), 1,905,378 (365 days)
install-on-request: 48,437 (30 days), 71,239 (90 days), 96,550 (365 days)
build-error: 0 (30 days)

We need to fix the code for making it to work with python 3.8

Applying the following patch to your salt formula (brew edit salt) and install it building from source will allow you to run salt-minion again.

diff --git a/Formula/salt.rb b/Formula/salt.rb
index d4a87977f..196e27648 100644
--- a/Formula/salt.rb
+++ b/Formula/salt.rb
@@ -17,7 +17,7 @@ class Salt < Formula
   depends_on "libgit2"
   depends_on "libyaml"
   depends_on "openssl@1.1"
-  depends_on "python@3.8"
+  depends_on "python@3.7"
   depends_on "zeromq"

   on_linux do
@@ -61,7 +61,7 @@ class Salt < Formula
     # https://github.com/Homebrew/homebrew-core/pull/52835#issuecomment-617502578
     File.write(buildpath/"pkg/osx/req_pyobjc.txt", "")

-    venv = virtualenv_create(libexec, Formula["python@3.8"].bin/"python3.8")
+    venv = virtualenv_create(libexec, Formula["python@3.7"].bin/"python3.7")
     venv.pip_install resources

     system libexec/"bin/pip", "install", "-v", "--ignore-installed", buildpath

brew install --build-from-source salt

Undo your formula changes after installing it to allow future updates for the salt formula.

I’ll try to at least rebase the PR against the latest master in the next couple of days…

I have copied the official salt formula from homebrew-core and patched it to use python@3.7

You can install salt with python3.7 using my homebrew-tap:

brew install cdalvaro/tap/salt

⚠️ I will keep the formula updated using python 3.7 when the official one is updated until this bug is resolved.

This is a known issue being worked on in https://github.com/saltstack/salt/pull/57028 for magnesium

You can install salt with python3.7 using my homebrew-tap:

Thanks @cdalvaro worked for me on Darwin.

Gonna need to put up the @s0undt3ch bat signal on this one. He’s already working on it here #57028

This is a temporary solution until the bug is fixed.

Edit the Homebrew salt formula (brew edit salt) to apply the following patch:

diff --git a/Formula/salt.rb b/Formula/salt.rb
index d4a87977f..196e27648 100644
--- a/Formula/salt.rb
+++ b/Formula/salt.rb
@@ -17,7 +17,7 @@ class Salt < Formula
   depends_on "libgit2"
   depends_on "libyaml"
   depends_on "openssl@1.1"
-  depends_on "python@3.8"
+  depends_on "python"
   depends_on "zeromq"

   on_linux do
@@ -61,7 +61,7 @@ class Salt < Formula
     # https://github.com/Homebrew/homebrew-core/pull/52835#issuecomment-617502578
     File.write(buildpath/"pkg/osx/req_pyobjc.txt", "")

-    venv = virtualenv_create(libexec, Formula["python@3.8"].bin/"python3.8")
+    venv = virtualenv_create(libexec, "python3")
     venv.pip_install resources

     system libexec/"bin/pip", "install", "-v", "--ignore-installed", buildpath

then:

brew uninstall salt
sudo rm -rf /usr/local/Cellar/salt/3001
brew install --build-from-source salt

May be we could create a PR at homebrew-core to apply this patch to the formula until the next Salt released is ready with the bug fixed.

Ran into this when trying to use salt-ssh from my mac. Doing the 3.7 fix with brew edit worked.

This bug report is about salt-ssh though, no running salt-minion involved.

I have tried adding this changes to salt/utils/process.py:

diff --git a/salt/utils/process.py b/salt/utils/process.py
index 2ac797b5f3..34e22b84cc 100644
--- a/salt/utils/process.py
+++ b/salt/utils/process.py
@@ -744,6 +744,9 @@ class Process(multiprocessing.Process, NewStyleClassMixIn):

     # __setstate__ and __getstate__ are only used on Windows.
     def __setstate__(self, state):
+        if not salt.utils.platform.is_windows():
+            self.__dict__.update(state)
+            return
         args = state["args"]
         kwargs = state["kwargs"]
         # This will invoke __init__ of the most derived class.
@@ -752,6 +755,8 @@ class Process(multiprocessing.Process, NewStyleClassMixIn):
         self._finalize_methods = self._finalize_methods

     def __getstate__(self):
+        if not salt.utils.platform.is_windows():
+            return self.__dict__
         args = self._args_for_getstate
         kwargs = self._kwargs_for_getstate
         if "log_queue" not in kwargs:

but then, the following error is reported:

[tornado.application:353 ][ERROR   ][1761] Future <salt.ext.tornado.concurrent.Future object at 0x1116c9550> exception was never retrieved: Traceback (most recent call last):
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/minion.py", line 1719, in _handle_decoded_payload
    process.start()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 872, in start
    super(SignalHandlingProcess, self).start()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Minion.setup_beacons.<locals>.handle_beacons'
[salt.minion      :505 ][ERROR   ][1761] Exception cannot pickle '_thread._local' object occurred in scheduled job

It looks this is the origin of the issue: https://docs.python.org/3/library/multiprocessing.html

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

Update

Similar issue solved here: https://github.com/horovod/horovod/pull/1904