salt: [BUG] Salt master-initiated jobs no longer work after master upgraded to Fedora 39/Python 3.12

Description Running Fedora 38 master with a Fedora 39 minion and a Debian 11 minion. Everything worked perfectly until I upgraded the master to Fedora 39. No sls or config changes took place.

Setup fails even for test.ping

Please be as specific as possible and give set-up details.

on-prem machine
VM (Virtualbox, KVM, etc. please specify)
VM running on a cloud service, please be explicit and add details
container (Kubernetes, Docker, containerd, etc. please specify)
or a combination, please be explicit
jails if it is FreeBSD
classic packaging
onedir packaging
used bootstrap to install

Steps to Reproduce the behavior Run state.apply or test.ping from the master against any minion

Expected behavior The action succeeds.

Screenshots N/A

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.3
 
Python Version:
        Python: 3.12.0 (main, Oct  2 2023, 00:00:00) [GCC 13.2.1 20230918 (Red Hat 13.2.1-3)]
 
Dependency Versions:
          cffi: 1.15.1
      cherrypy: 18.8.0
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: 4.0.10
     gitpython: 3.1.32
        Jinja2: 3.1.2
       libgit2: 1.7.1
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: 1.2.3
       msgpack: 1.0.5
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.20
      pycrypto: 2.6.1
  pycryptodome: 3.19.0
        pygit2: 1.13.0
  python-gnupg: 0.5.0
        PyYAML: 6.0.1
         PyZMQ: 25.1.0
        relenv: Not Installed
         smmap: 5.0.0
       timelib: Not Installed
       Tornado: 6.3.3
           ZMQ: 4.3.4
 
System Versions:
          dist: fedora 39 
        locale: utf-8
       machine: x86_64
       release: 6.5.5-300.fc39.x86_64
        system: Linux
       version: Fedora Linux 39

Additional context I’ve checked firewall ports, tcp connectivity, DNS, and rekeyed the master and minions. Nothing helps.

About this issue

Original URL
State: open
Created 9 months ago
Reactions: 1
Comments: 29 (11 by maintainers)

Most upvoted comments

It is merged into the master branch, so it should appear in 3007 RC1 which is planned to be released before the end of the month.

dmurphy18 on Dec 7, 2023

Just in case someone stumbles upon this issue and thinks changing the Fedora 39 salt package’s USE_VENDORED_TORNADO = True to USE_VENDORED_TORNADO = False is the obvious workaround: That did not work for me.

ndim on Nov 16, 2023

Ok, good. I’d love a patch that fixed this for 3006.3, but given that timeline, if it is indeed in 3007, I won’t lose too much sleep over it.

limburgher on Oct 18, 2023

An error stacktrace for the curious

Traceback (most recent call last):
  File "/usr/bin/salt-call", line 33, in <module>
    sys.exit(load_entry_point('salt==3006.3', 'console_scripts', 'salt-call')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/salt/scripts.py", line 437, in salt_call
    import salt.cli.call
  File "/usr/lib/python3.12/site-packages/salt/cli/call.py", line 3, in <module>
    import salt.cli.caller
  File "/usr/lib/python3.12/site-packages/salt/cli/caller.py", line 12, in <module>
    import salt.channel.client
  File "/usr/lib/python3.12/site-packages/salt/channel/client.py", line 13, in <module>
    import salt.crypt
  File "/usr/lib/python3.12/site-packages/salt/crypt.py", line 26, in <module>
    import salt.payload
  File "/usr/lib/python3.12/site-packages/salt/payload.py", line 12, in <module>
    import salt.loader.context
  File "/usr/lib/python3.12/site-packages/salt/loader/__init__.py", line 23, in <module>
    import salt.utils.event
  File "/usr/lib/python3.12/site-packages/salt/utils/event.py", line 67, in <module>
    import salt.ext.tornado.iostream
  File "/usr/lib/python3.12/site-packages/salt/ext/tornado/iostream.py", line 41, in <module>
    from salt.ext.tornado.netutil import ssl_wrap_socket, ssl_match_hostname, SSLCertificateErr
or, _client_ssl_defaults, _server_ssl_defaults
  File "/usr/lib/python3.12/site-packages/salt/ext/tornado/netutil.py", line 57, in <module>
    import backports.ssl_match_hostname

Note well how using upstream tornado would fix this.

Vaelatern on Oct 18, 2023