salt: 2018.3 regression - unable to create Caller in a runner

Description of Issue/Question

Runners which create salt.client.Caller throw exception listed below. This stops us from upgrading to 2018.3 release.

Steps to Reproduce Issue

On a CentOS 7.4 minimally installed VM, do the following -

# curl -o bootstrap-salt.sh -L https://bootstrap.saltstack.com
# sudo sh bootstrap-salt.sh -M git 2018.3

Make sure that following returns true -

[root@base2 ~]# salt-call test.ping
local:
    True

Now create a test runner with the following - cat /srv/_runners/test_runner.py

# -*- coding: utf-8 -*-
'''
test_runner.py - implementation of test runner api
Custom salt runner module
'''

import logging
import salt.client

__virtualname__ = 'test_runner'

#pylint: disable=bad-indentation, line-too-long, invalid-name

def __virtual__():
  return __virtualname__

def _get_logger():
  '''
  Get current logger.
  '''
  logger = logging.getLogger(__name__)
  return logger

def ping():
  '''
  Verify presence and installation of the module
  CLI Example:
     salt-run test_runner.ping
  '''
  local = salt.client.Caller()
  _get_logger().info('%s pinged', __name__)
  return True

Running (after configuring runners etc.) the salt-run test_runner.ping returns the following exception -

2018-04-05 12:41:06,259 [tornado.application:611 ][ERROR   ][9823] Exception in callback <functools.partial object at 0x2800ec0>
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/tornado/ioloop.py", line 591, in _run_callback
    ret = callback()
  File "/usr/lib64/python2.7/site-packages/tornado/stack_context.py", line 342, in wrapped
    raise_exc_info(exc)
  File "/usr/lib64/python2.7/site-packages/tornado/stack_context.py", line 313, in wrapped
    ret = fn(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/tornado/ioloop.py", line 597, in <lambda>
    self.add_future(ret, lambda f: f.result())
  File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 230, in wrapper
    yielded = next(result)
  File "/usr/lib/python2.7/site-packages/salt/crypt.py", line 595, in _authenticate
    io_loop=self.io_loop)
  File "/usr/lib/python2.7/site-packages/salt/transport/client.py", line 108, in factory
    return salt.transport.zeromq.AsyncZeroMQReqChannel(opts, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 133, in __new__
    obj.__singleton_init__(opts, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 197, in __singleton_init__
    kwargs={'io_loop': self._io_loop})
  File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 927, in __init__
    super(AsyncReqMessageClientPool, self).__init__(AsyncReqMessageClient, opts, args=args, kwargs=kwargs)
  File "/usr/lib/python2.7/site-packages/salt/transport/__init__.py", line 71, in __init__
    self.message_clients = [tgt(*args, **kwargs) for _ in range(sock_pool_size)]
  File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 974, in __init__
    self._init_socket()
  File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 1028, in _init_socket
    self.stream = zmq.eventloop.zmqstream.ZMQStream(self.socket, io_loop=self.io_loop)
  File "/usr/lib64/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 114, in __init__
    self._init_io_state()
  File "/usr/lib64/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 535, in _init_io_state
    self.io_loop.add_handler(self.socket, self._handle_events, self._state)
  File "/usr/lib64/python2.7/site-packages/tornado/ioloop.py", line 703, in add_handler
    self._impl.register(fd, events | self.ERROR)
TypeError: argument must be an int, or have a fileno() method.

If I manually update the pyzmq using pip to 17.0.0 version, this exception disappears.

Observation - By default repo.saltstack.com is supplying python-zmq version 15.3, should this be updated?

Versions Report

Salt Version:
           Salt: 2018.3.0-n/a-a064a3e

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  4 2017, 00:39:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.4.1708 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-693.17.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.4.1708 Core

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 34 (30 by maintainers)

Most upvoted comments

@akbhat - for what it’s worth, I used pip to upgrade pyzmq to version 17 (as recommended by @isbm on April 12), and found this to be a suitable workaround for the time being:

# pip install --upgrade pyzmq
pyzmq (17.0.0)         - Python bindings for 0MQ
INSTALLED: 15.3.0 <- Version we were on
LATEST:    17.0.0 <- Version we upgraded to via pip

Yuck… Sorry for this, guys. It was pretty heavily tested with 14 and then 17. Part of this because at SUSE we had 14 and then upgraded it to the latest in upcoming SLE 15 and Leap. And so I had naїve trust to ZMQ’s claim it is different only after 16, but till 16 it is same as 13. But seems I have shouldn’t trust that… Also strange that this popped up only now. No Beta testers around?

If you are unable to wait, just upgrade your PyZMQ to 17 and libzmq (or upgrade to Leap over CentOS, then it will work flawlessly out of the box — Salt is default there now, hehe 😛 ). It has anyway quite dramatic bugfixes to ZMQ you want to have.

Jokes aside, in the meanwhile I will try to fix this for you.