salt: [BUG] /usr/bin/salt-call state.apply hangs at end on Xubuntu 20.04 salt 3001

Description Running /usr/bin/salt-call state.apply on Xubuntu 20.04 minion finishes but never completes.

Summary for local
--------------
Succeeded: 153
Failed:      0
--------------
Total states run:     153
Total run time:    16.030 s

But command never returns back to terminal.

ps -ef |grep salt-call
root     2928868 2914767  0 08:12 pts/28   00:00:00 sudo /usr/bin/salt-call state.apply
root     2928869 2928868  0 08:12 pts/28   00:00:05 /usr/bin/python3 /usr/bin/salt-call state.apply
root     3026694 3026554  0 09:37 pts/27   00:00:00 grep --color=auto salt-call

Notice the command has been running over an hour but only took 16 seconds for the states to be applied.

Setup

Steps to Reproduce the behavior Xubuntu 20.04 install with ZFS root

more /etc/apt/sources.list.d/saltstack.list
deb [arch=amd64] http://repo.saltstack.com/py3/ubuntu/20.04/amd64/latest focal main

Salt master is Ubuntu 18.04 version 3001 recently upgraded from nigrogen. All other minions on network are 18.04 and working as expected.

Expected behavior salt-call state.apply to finish and return to terminal.

Screenshots

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Master:
salt --versions-report
Salt Version:
           Salt: 3001
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
         Jinja2: 2.10
        libgit2: 0.26.0
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: 3.4.7
         pygit2: 0.26.2
         Python: 3.6.9 (default, Apr 18 2020, 01:56:04)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 17.1.2
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: ubuntu 18.04 Bionic Beaver
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-101-generic
         system: Linux
        version: Ubuntu 18.04 Bionic Beaver

Minion With issue:
Salt Version:
           Salt: 3001
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: 4.1.0
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: 1.1.0
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.6.1
         pygit2: Not Installed
         Python: 3.8.2 (default, Apr 27 2020, 15:53:34)
   python-gnupg: 0.4.5
         PyYAML: 5.3.1
          PyZMQ: 18.1.1
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2
 
System Versions:
           dist: ubuntu 20.04 focal
         locale: utf-8
        machine: x86_64
        release: 5.4.0-39-generic
         system: Linux
        version: Ubuntu 20.04 focal


Working Minon:

salt-call --versions-report
Salt Version:
           Salt: 2017.7.4
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.9 (default, Apr 18 2020, 01:56:04)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-101-generic
         system: Linux
        version: Ubuntu 18.04 bionic

Additional context Add any other context about the problem here.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 37 (21 by maintainers)

Most upvoted comments

@ocfmatt we are working on a long term solution, but since we do not maintain this library it is taking some time. It was one suggestion to try to downgrade, but we realize that isn’t really a good idea, either, but looking for a real fix to this, still.

sagetherage on Aug 13, 2020

I’m hesitant on #57356 as well as I keep most of my states without jinja references and filter via grains in the top.sls file.

It also fails if I run just a single state file. For example, my vscode state:

salt-call state.apply vscode

cat vscode/init.sls 
vscode_repo:
  pkgrepo.managed:
    - name: deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main
    - file: /etc/apt/sources.list.d/vscode.list
    - key_url: https://packages.microsoft.com/keys/microsoft.asc
    - refresh_db: true

vscode:
  pkg.installed:
    - name: code
    - require:
        - pkgrepo: vscode_repo

Output is similar to other fails:

local:
----------
          ID: vscode_repo
    Function: pkgrepo.managed
        Name: deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main
      Result: True
     Comment: Package repo 'deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main' already configured
     Started: 13:02:08.425043
    Duration: 107.504 ms
     Changes:   
----------
          ID: vscode
    Function: pkg.installed
        Name: code
      Result: True
     Comment: All specified packages are already installed
     Started: 13:02:09.400992
    Duration: 73.534 ms
     Changes:   

Summary for local
------------
Succeeded: 2
Failed:    0
------------
Total states run:     2
Total run time: 181.038 ms
^C

CTRL-C after waiting a couple of minutes

mgdotson on Jul 3, 2020

this means salt isn’t usable for ubuntu 20.04? As you cannot run mixed clients tcp and not tcp you cannot even upgrade your infra

disaster123 on Aug 11, 2020

yes, returns instantly with transport set to tcp.

mgdotson on Jul 23, 2020

@JaseFace yep and I’m working in this direction. Thank you for analyze.

DmitryKuzmenko on Jul 20, 2020