salt: 2019.2.1/2019.2.0 pip failures even when not using pip

While testing my setup with Debian Buster and “stable” Saltstack version (2019.2.0) on a vagrant instance I run into weird problems… Especially i didn’t use pip and most of the other modules notified in the failure messages I was search for over half hour for the cause:

# salt-call state.apply
[ERROR   ] Failed to import states pip_state, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pip_state.py", line 58, in <module>
    del pip
NameError: name 'pip' is not defined
[ERROR   ] Failed to import states pkg, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pkg.py", line 79, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states pkgbuild, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pkgbuild.py", line 48, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states pkgrepo, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/pkgrepo.py", line 93, in <module>
    from salt.exceptions import CommandExecutionError, SaltInvocationError
  File "/usr/lib/python2.7/dist-packages/salt/exceptions.py", line 9, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.portage_config. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.portage_config.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'portage_config', please fix this.
[ERROR   ] Failed to import states ports, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/ports.py", line 21, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_cluster. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_cluster.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'postgres_cluster', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_database. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_database.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'postgres_database', please fix this.
[ERROR   ] Failed to import states postgres_extension, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/postgres_extension.py", line 19, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states postgres_group, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/postgres_group.py", line 18, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_initdb. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_initdb.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'postgres_initdb', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_language. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_language.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'postgres_language', please fix this.
[ERROR   ] Exception raised when processing __virtual__ function for salt.loaded.int.states.postgres_privileges. Module will not be loaded: 'NoneType' object is not callable
[WARNING ] salt.loaded.int.states.postgres_privileges.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'postgres_privileges', please fix this.
[ERROR   ] Failed to import states postgres_schema, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/postgres_schema.py", line 16, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states postgres_tablespace, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/postgres_tablespace.py", line 23, in <module>
    import salt.utils.dictupdate as dictupdate
  File "/usr/lib/python2.7/dist-packages/salt/utils/__init__.py", line 18, in <module>
    from salt.ext import six
  File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 728, in <module>
    print_ = getattr(moves.builtins, "print", None)
  File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 99, in __get__
    result = self._resolve()
  File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 122, in _resolve
    return _import_module(self.mod)
  File "/usr/lib/python2.7/dist-packages/salt/ext/six.py", line 90, in _import_module
    return sys.modules[name]
AttributeError: 'module' object has no attribute 'modules'
[ERROR   ] Failed to import states postgres_user, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1567, in _load_module
    mod = imp.load_module(mod_namespace, fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/states/postgres_user.py", line 17, in <module>
    import logging
  File "/usr/lib/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref, collections
  File "/usr/lib/python2.7/os.py", line 119, in <module>
    sys.modules['os.path'] = path
AttributeError: 'module' object has no attribute 'modules'

Passed invalid arguments: 'NoneType' object is not callable.

Usage:

    .. versionadded:: 2015.5.0

    This function will call :mod:`state.highstate
    <salt.modules.state.highstate>` or :mod:`state.sls
    <salt.modules.state.sls>` based on the arguments passed to this function.
    It exists as a more intuitive way of applying states.

    .. rubric:: APPLYING ALL STATES CONFIGURED IN TOP.SLS (A.K.A. :ref:`HIGHSTATE <running-highstate>`)

    To apply all configured states, simply run ``state.apply``:

    .. code-block:: bash

        salt '*' state.apply

    The following additional arguments are also accepted when applying all
    states configured in top.sls:

    test
        Run states in test-only (dry-run) mode

    mock
        The mock option allows for the state run to execute without actually
        calling any states. This then returns a mocked return which will show
        the requisite ordering as well as fully validate the state run.

        .. versionadded:: 2015.8.4

    pillar
        Custom Pillar values, passed as a dictionary of key-value pairs

        .. code-block:: bash

            salt '*' state.apply stuff pillar='{"foo": "bar"}'

        .. note::
            Values passed this way will override Pillar values set via
            ``pillar_roots`` or an external Pillar source.

    exclude
        Exclude specific states from execution. Accepts a list of sls names, a
        comma-separated string of sls names, or a list of dictionaries
        containing ``sls`` or ``id`` keys. Glob-patterns may be used to match
        multiple states.

        .. code-block:: bash

            salt '*' state.apply exclude=bar,baz
            salt '*' state.apply exclude=foo*
            salt '*' state.apply exclude="[{'id': 'id_to_exclude'}, {'sls': 'sls_to_exclude'}]"

    queue : False
        Instead of failing immediately when another state run is in progress,
        queue the new state run to begin running once the other has finished.

        This option starts a new thread for each queued state run, so use this
        option sparingly.

    localconfig
        Optionally, instead of using the minion config, load minion opts from
        the file specified by this argument, and then merge them with the
        options from the minion config. This functionality allows for specific
        states to be run with their own custom minion configuration, including
        different pillars, file_roots, etc.

        .. code-block:: bash

            salt '*' state.apply localconfig=/path/to/minion.yml


    .. rubric:: APPLYING INDIVIDUAL SLS FILES (A.K.A. :py:func:`STATE.SLS <salt.modules.state.sls>`)

    To apply individual SLS files, pass them as a comma-separated list:

    .. code-block:: bash

        # Run the states configured in salt://stuff.sls (or salt://stuff/init.sls)
        salt '*' state.apply stuff
        # Run the states configured in salt://stuff.sls (or salt://stuff/init.sls)
        # and salt://pkgs.sls (or salt://pkgs/init.sls).
        salt '*' state.apply stuff,pkgs

    The following additional arguments are also accepted when applying
    individual SLS files:

    test
        Run states in test-only (dry-run) mode

    mock
        The mock option allows for the state run to execute without actually
        calling any states. This then returns a mocked return which will show
        the requisite ordering as well as fully validate the state run.

        .. versionadded:: 2015.8.4

    pillar
        Custom Pillar values, passed as a dictionary of key-value pairs

        .. code-block:: bash

            salt '*' state.apply stuff pillar='{"foo": "bar"}'

        .. note::
            Values passed this way will override Pillar values set via
            ``pillar_roots`` or an external Pillar source.

    queue : False
        Instead of failing immediately when another state run is in progress,
        queue the new state run to begin running once the other has finished.

        This option starts a new thread for each queued state run, so use this
        option sparingly.

    concurrent : False
        Execute state runs concurrently instead of serially

        .. warning::

            This flag is potentially dangerous. It is designed for use when
            multiple state runs can safely be run at the same time. Do *not*
            use this flag for performance optimization.

    saltenv
        Specify a salt fileserver environment to be used when applying states

        .. versionchanged:: 0.17.0
            Argument name changed from ``env`` to ``saltenv``

        .. versionchanged:: 2014.7.0
            If no saltenv is specified, the minion config will be checked for an
            ``environment`` parameter and if found, it will be used. If none is
            found, ``base`` will be used. In prior releases, the minion config
            was not checked and ``base`` would always be assumed when the
            saltenv was not explicitly set.

    pillarenv
        Specify a Pillar environment to be used when applying states. This
        can also be set in the minion config file using the
        :conf_minion:`pillarenv` option. When neither the
        :conf_minion:`pillarenv` minion config option nor this CLI argument is
        used, all Pillar environments will be merged together.

    localconfig
        Optionally, instead of using the minion config, load minion opts from
        the file specified by this argument, and then merge them with the
        options from the minion config. This functionality allows for specific
        states to be run with their own custom minion configuration, including
        different pillars, file_roots, etc.

        .. code-block:: bash

            salt '*' state.apply stuff localconfig=/path/to/minion.yml

    sync_mods
        If specified, the desired custom module types will be synced prior to
        running the SLS files:

        .. code-block:: bash

            salt '*' state.apply stuff sync_mods=states,modules
            salt '*' state.apply stuff sync_mods=all

        .. note::
            This option is ignored when no SLS files are specified, as a
            :ref:`highstate <running-highstate>` automatically syncs all custom
            module types.

        .. versionadded:: 2017.7.8,2018.3.3,2019.2.0

After wasting my nightly time I found luckily that the system has installed a new version which is officially not yet relased: 2019.2.1

And thx to the autoupdate function now also my other instances where influenced and I have to manually downgrade dist to

deb https://repo.saltstack.com/apt/debian/9/amd64/archive/2019.2.0 stretch main

Interesting also that my common vagrant setup has setup the line with stretch and not buster, but this seems because you reuse stretch packages for buster ?

AFTER 10 MINUTES of REPAIRING EACH INSTANCE caused by addition broken package dependencies I can run my states again as usual:

# salt-call state.apply
[WARNING ] /usr/lib/python2.7/dist-packages/salt/modules/mysql.py:546: Warning: (1681L, "'PASSWORD' is deprecated and will be removed in a future release.")
  return cur.execute(qry, args)

local:

Summary for local
--------------
Succeeded: 140
Failed:      0
--------------
Total states run:     140
Total run time:     9.274 s

ah… now documentation shows the buggy version as released…

TeleTubbies

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 8
Comments: 35 (13 by maintainers)

Commits related to this issue

install python3-pip and python-pip to work around https://github.com/saltstack/salt/issues/54755 — committed to gacybercenter/kinetic by bitskri3g 5 years ago

Most upvoted comments

I’m using SaltStack for some years now and I must admit that almost EVERY release comes with important and obvious regressions showing a bunch of errors.

Yes code quality control isn’t a simple thing to deal with, yes SaltStack has a lot of modules witten by many people but what could be worse than wating for months (sometimes years) that a good PR is merged, packaged and deployed on one hand and then having bad troubles every time we update SaltStack on the other?

I know that this comment is a bit rude but please understand that this is very frustrating.

https://github.com/saltstack/salt/issues/34096

+10

tomlaredo on Sep 26, 2019

Hello,

So this is patched in #54826 and merged, do you have any idea when the fix will be available in packages ? Do you plan to release a new release soon ?

The bug is blocking on our side (we need it the latest version for debian 10 ^^), installing pip is not really a good solution and we would like to know how long we should expect to wait 😃

Thanks!

the-glu on Oct 3, 2019

+1 to this happening when using states with onlyif and unless requisites.

Furthermore, even AFTER installing python-pip, something feels wrong… a state as simple as below takes ~10sec (!!!) to execute whereas it takes ~150ms to execute without the requisites.

try-this-without-requisites:
  module.run:
    - name: test.ping
    - onlyif:
      - /bin/true

tj90241 on Sep 30, 2019

We found a similar issue by using ‘latest’. A lot of packages were not being installed at all so we had to downgrade to 2019.2 per @CaptainSanders https://github.com/saltstack/salt/issues/54755#issuecomment-544857544

Here’s the error for us:

Traceback (most recent call last): File “/usr/lib/python2.7/dist-packages/salt/loader.py”, line 1607, in load_module mod = imp.load_module(mod_namespace, fn, fpath, desc) File “/usr/lib/python2.7/dist-packages/salt/states/pip_state.py”, line 110, in <module> del pip NameError: name ‘pip’ is not defined 2020-02-14 12:51:29,334 [salt.loader :1624][ERROR ][7182] Failed to import states pkg, this is due most likely to a syntax error: Traceback (most recent call last): File “/usr/lib/python2.7/dist-packages/salt/loader.py”, line 1607, in load_module mod = imp.load_module(mod_namespace, fn, fpath, desc) File “/usr/lib/python2.7/dist-packages/salt/states/pkg.py”, line 77, in <module> import logging File “/usr/lib/python2.7/logging/init.py”, line 26, in <module> import sys, os, time, cStringIO, traceback, warnings, weakref, collections File “/usr/lib/python2.7/os.py”, line 119, in <module> sys.modules[‘os.path’] = path AttributeError: ‘module’ object has no attribute ‘modules’ 2020-02-14 12:51:29,343 [salt.loader :1624][ERROR ][7182] Failed to import states pkgbuild, this is due most likely to a syntax error: Traceback (most recent call last): File “/usr/lib/python2.7/dist-packages/salt/loader.py”, line 1607, in load_module mod = imp.load_module(mod_namespace, fn, fpath, desc) File “/usr/lib/python2.7/dist-packages/salt/states/pkgbuild.py”, line 48, in <module> import logging File “/usr/lib/python2.7/logging/init.py”, line 26, in <module> import sys, os, time, cStringIO, traceback, warnings, weakref, collections File “/usr/lib/python2.7/os.py”, line 119, in <module> sys.modules[‘os.path’] = path AttributeError: ‘module’ object has no attribute ‘modules’

jdrowne on Feb 14, 2020

Also seeing the issue when targeting state using “Unless”, as well as with “OnlyIf” logic. However the root cause seems to be related to changes to the pip_state in 2019.2.1 See issue: #53570

stratusjerry on Sep 26, 2019

bad bot!

TheJJ on Jan 7, 2020

If you’ve installed salt from a repo I recommend downgrading to 2019.2.0. That worked for me.

CaptainSanders on Oct 22, 2019

so you guys decided to remove existing package from the repository instead of releasing a version including the fix? 😃

https://repo.saltstack.com/py3/debian/10/amd64/

2019-10-11 at 10 38

dynek on Oct 11, 2019

Vendorizing might (or might not) be a long term strategy to manage dependencies. Right now, @rallytime please help us with releasing this fix as soon as possible, this makes latest saltstack release (again!) completely broken for some cases.

marek-obuchowicz on Oct 8, 2019

Actually, the first issue would actually be solved by vendorizing said module. Actually it would’ve removed the need for the breaking-commit even. Bundling the deps that you’re deving against would also mean that the second case would never happen. Without bundling module versions, your testers essentially need to test how things work with all different versions and without said core module in order to test comprehensively. This can be unreasonable for testers despite potentially being rockstars.

Another benefit is that it’ll fix that need to try-except-import at the beginning of all modules in order to test whether salt’s able to import a core module or not. Actually thinking about it, that’s probably one aspect of the performance issue that salt seems to have, and salt is just riddled with these types of handlers.

Writing a quick testcase that is similar to the try-except that each core module wraps their import dependent-module header with looks like the following. f1() would normally import something like seco.range for example.

import sys, time
def f1():
    try:
        # Imagine an import pip being here
        raise Exception
    except:
        pass
    return

def f2():
    try:
    return

f = f1 if sys.argv[1] == '1' else f2
t = time.time()
for x in range(4096*1024):
    f()
print(time.time() - t)

This code when calling f2() which simply does nothing has pretty good performance because well…it’s hitting the hot-path. Even if it creates an exception handler, as long as it hits the hot path performance will be minimal.

[2962] user@phatty ~/audit/acrobat-fuzz/lol/deploy$ python ~/f.py
0.522000074387
[2963] user@phatty ~/audit/acrobat-fuzz/lol/deploy$ python3 ~/f.py
0.357666015625

Now when calling f1() which is when an exception is raised and caught which will happen at the beginning of any-and-all lazyloaded modules.

[2964] user@phatty ~/audit/acrobat-fuzz/lol/deploy$ python ~/f.py
4.42999982834
[2965] user@phatty ~/audit/acrobat-fuzz/lol/deploy$ python3 ~/f.py
1.3732171058654785

Yea, that’s 4.4 seconds vs 0.5 seconds for python2, and 1.3 vs 0.3 seconds for python3… This is an exaggerated number of cases of course, but literally every single core module in salt checks for a likely non-existent module, especially during matching.

So if there’s a dependent-module that’s not found, that try-except case is hit all-the-time which has super poor performance because of the cost of dispatching an exception (course this is in a hundred other places in salt). If you were using reload() (despite it being janky) this’d be fine because things like HAS_PIP would still be cached and there’d be no need to check whether the import raised an exception.

But since you use imp.load_module, each exception in the module has to be dispatched if the dependent-module does not exist. Not just that but specifically to everytime a module is imported, that imp.load_module has to re-compile the entire module which can potentially take even longer.

On another note, I sympathize for you guys because everytime you hit a release you can tell that you’re in crunchtime because there’s lapses in the responses to bug reports. If this has been like this since 2016 as noted by tomlaredo, you should consider putting things on pause while you profile and refactor as long as your “paying” customers allow for it.

arizvisa on Oct 7, 2019