salt: [BUG] Minions running 3002+ throw a JSONDecodeError when called from master

Description Minions running with Saltstack 3002.6 or 3003 throw an exception when called from master also running on 3002.6 or 3003.

Setup Fresh installation of saltstack 3003 on Ubuntu 20.04

Steps to Reproduce the behavior

sudo salt -t 300 'vm-api-server' test.ping
or
sudo salt -t 300 'vm-api-server' state.apply
vm-api-server:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/salt/minion.py", line 1904, in _thread_return
        return_data = minion_instance._execute_job_function(
      File "/usr/lib/python3/dist-packages/salt/minion.py", line 1861, in _execute_job_function
        return_data = self.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 1235, in __call__
        return self.loader.run(run_func, *args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 2268, in run
        return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 2283, in _run_as
        return _func_or_method(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/executors/sudo.py", line 77, in execute
        cmd_meta = salt.utils.json.loads(cmd_ret["stdout"])["local"]
      File "/usr/lib/python3/dist-packages/salt/utils/json.py", line 97, in loads
        return json_module.loads(s, **kwargs)
      File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
        return _default_decoder.decode(s)
      File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Expected behavior

sudo salt -t 300 'vm-api-server' test.ping
vm-api-server:
    True

Versions Report master:

sudo salt -V
Salt Version:
          Salt: 3003

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: 0.28.3
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: 1.0.3
        Python: 3.8.5 (default, May 27 2021, 13:30:53)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-1047-azure
        system: Linux
       version: Ubuntu 20.04 focal

minion:

Salt Version:
          Salt: 3003

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.5 (default, May 27 2021, 13:30:53)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-1046-azure
        system: Linux
       version: Ubuntu 20.04 focal

Additional context When i downgrade the minion on Ubuntu 20.04 to 3001.7 the call from master works just fine. Also running sudo salt-call state.highstate on the minion itself with 3002.6 or 3003 works fine too.

Also when using Ubuntu 20.04 on the master and using Ubuntu 18.04 as the minion it only works with 3001.7. Using Ubuntu 18.04 with 3002.6 as the master and Ubuntu 18.04 with 3002.6 or 3003 as the minion works too. Using Ubuntu 18.04 with 3002.6 as the master and Ubuntu 20.04 with 3002.6 on the minion doesn’t work. Using Ubuntu 18.04 with 3002.6 as the master and Ubuntu 18.04 with 3002.6 on the minion doesn’t work. I also have some Servers using the same combination without problems.

Is this a locale problem, because the servers it works on are locale: ISO-8859-1while the others are locale: UTF-8? Even though it doesn’t make sense that old version of saltstack on the minion work just fine.

------ Update Doesn’t seem to be the locale.

Working minion:

Salt Version:
          Salt: 3002.6

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: 1.3.10
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.4.7
        pygit2: Not Installed
        Python: 3.6.9 (default, Jan 26 2021, 15:33:00)
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.5

System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: ISO-8859-1
       machine: x86_64
       release: 4.15.0-136-generic
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver

Not working minion:

Salt Version:
          Salt: 3002.6

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.4.7
        pygit2: Not Installed
        Python: 3.6.9 (default, Jan 26 2021, 15:33:00)
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.5

System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: ISO-8859-1
       machine: x86_64
       release: 5.4.0-1048-azure
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 25
Comments: 18 (8 by maintainers)

Most upvoted comments

We have the same issue with Salt Minion 3004 running on Ubuntu 18.04 (AWS).

+21

vladimir-incountry on Dec 1, 2021

Hi, concerning the cmd_ret[“stdout”], this is just empty, as i wrote earlier. And in the json loads function the variable “s” is just an empty string '' pprint just gives “None”

sebastianreloaded on Aug 11, 2021

I’m seeing the same issue with Salt Minion and Master 3003.2 on REHL 8.4. In my setup I’m running the minion as a non-privileged user using the user config setting and setting the sudo_user setting to root. A salt-call pkg.install works fine on the Minion, but calling it with salt from the master fails with the same errors as above. I’ve logged the cmd_ret in the sudo.py file and the stdout is empty as stated, and manually setting use_vt=False fixes the issue.

dorschs57 on Dec 2, 2021