salt: [BUG] pip.installed doesn't work with upgrade=True due to inability of getting available version numbers

Description Using pip.installed with upgrade: True results in the following traceback, which is due to how the salt.modules.pip.list_all_versions tries to get the available versions from pypi.

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2171, in call
    ret = self.states[cdata["full"]](
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2105, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 852, in installed
    out = _check_if_installed(
  File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 329, in _check_if_installed
    desired_version = available_versions[-1]
TypeError: 'NoneType' object is not subscriptable

Setup

my_state.sls:

python3-pip:
  pkg.installed


pip-packages:
  pip.installed:
    - upgrade: True
    - names:
      - matplotlib
    - require:
        - pkg: python3-pip

salt-call state.apply my_state on the minion

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2171, in call
    ret = self.states[cdata["full"]](
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2105, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 852, in installed
    out = _check_if_installed(
  File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 329, in _check_if_installed
    desired_version = available_versions[-1]
TypeError: 'NoneType' object is not subscriptable

local:
----------
          ID: python3-pip
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 15:19:29.965834
    Duration: 60.832 ms
     Changes:   
----------
          ID: pip-packages
    Function: pip.installed
        Name: matplotlib
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3/dist-packages/salt/state.py", line 2171, in call
                  ret = self.states[cdata["full"]](
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 2105, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 852, in installed
                  out = _check_if_installed(
                File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 329, in _check_if_installed
                  desired_version = available_versions[-1]
              TypeError: 'NoneType' object is not subscriptable
     Started: 15:19:30.175097
    Duration: 3189.604 ms
     Changes:   

Summary for local
------------
Succeeded: 1
Failed:    1
------------
Total states run:     2
Total run time:   3.250 s

Versions Report salt --versions-report

On master:

Salt Version:
          Salt: 3003.3
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: 3.1.14
        Jinja2: 2.11.3
       libgit2: 1.1.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: 1.4.0
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-8-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

On minion:

Salt Version:
          Salt: 3002.6
 
Dependency Versions:
          cffi: 1.14.5
      cherrypy: 8.9.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: 1.4.4
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-8-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

Pip version on minion:

pip 20.3.4 from /usr/lib/python3/dist-packages/pip (python 3.9)

Additional context My diagnosis is the following: salt.states.pip_state.installed tries to get the last available version via salt.modules.pip.list_all_versions, which tries to get all available version from pypi by calling pip install matplotlib==versions. This returns:

ERROR: Could not find a version that satisfies the requirement matplotlib==versions
ERROR: No matching distribution found for matplotlib==versions

However, salt.modules.pip.list_all_versions tries to regex match this to r"\s*Could not find a version.* \(from versions: (.*)\)"; which doesn’t work so well. The root cause is (of course) that pip doesn’t (didn’t?) have an api to request versions for (a) package(s). See also: https://stackoverflow.com/questions/4888027/python-and-pip-list-all-versions-of-a-package-thats-available?rq=1

I attempted a workaround, where I made the following module: salt://_modules/pip.py:

import json
from urllib.request import urlopen, Request
import pkg_resources

from salt.modules.pip import *


def _query_url(pkg_name, base_url=None):
    if base_url and not '://' in base_url:
        base_url = 'https://{}'.format(base_url)
    url = "{}/pypi/{}/json".format(base_url, pkg_name)
    data = json.load(urlopen(Request(url)))
    versions = data["releases"].keys()
    return versions

def list_all_versions(
    pkg,
    bin_env=None,
    include_alpha=False,
    include_beta=False,
    include_rc=False,
    user=None,
    cwd=None,
    index_url=None,
    extra_index_url=None,
):

    if index_url and not salt.utils.url.validate(index_url, VALID_PROTOS):
        raise CommandExecutionError("'{}' is not a valid URL".format(index_url))

    filtered = []
    if not include_alpha:
        filtered.append("a")
    if not include_beta:
        filtered.append("b")
    if not include_rc:
        filtered.append("rc")
    if filtered:
        excludes = re.compile(r"^((?!{}).)*$".format("|".join(filtered)))
    else:
        excludes = re.compile(r"")

    versions = _query_url(pkg, index_url)

    versions = sorted((v for v in versions if excludes.match(v)), key=pkg_resources.parse_version)

    if not versions:
        return None

    return versions

Where my idea is that since salt.states.pip_state.installed “does the right thing” by calling __salt__["pip.list_all_versions"] my new module would neatly take precedence. Doesn’t work, unfortunately:

[ERROR   ] State 'pip.installed' was not found in SLS 'packages.pip_packages'
Reason: 'pip.installed' is not available.

local:
----------
          ID: python3-pip
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 15:36:02.353821
    Duration: 61.283 ms
     Changes:   
----------
          ID: pip-packages
    Function: pip.installed
        Name: matplotlib
      Result: False
     Comment: State 'pip.installed' was not found in SLS 'packages.pip_packages'
              Reason: 'pip.installed' is not available.
     Changes:   

Summary for local
------------
Succeeded: 1
Failed:    1
------------
Total states run:     2
Total run time:  61.283 ms

Any advice is welcome 😃

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 20 (13 by maintainers)

Most upvoted comments

@pckroon Sorry about the misunderstanding, can reproduce now

root@tdeb11:/home/david# pip3 list | grep matplotlib
matplotlib          3.4.3
root@tdeb11:/home/david# cd /srv/salt/
root@tdeb11:/srv/salt# salt-call --local state.apply my_state 2>&1 | tee ~/mytest.log 
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2171, in call
    ret = self.states[cdata["full"]](
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1241, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2274, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2289, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2322, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 852, in installed
    out = _check_if_installed(
  File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 329, in _check_if_installed
    desired_version = available_versions[-1]
TypeError: 'NoneType' object is not subscriptable

local:
----------
          ID: python3-pip
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 10:18:07.929935
    Duration: 42.081 ms
     Changes:   
----------
          ID: pip-packages
    Function: pip.installed
        Name: matplotlib
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3/dist-packages/salt/state.py", line 2171, in call
                  ret = self.states[cdata["full"]](
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 1241, in __call__
                  return self.loader.run(run_func, *args, **kwargs)
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 2274, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 2289, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 2322, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 852, in installed
                  out = _check_if_installed(
                File "/usr/lib/python3/dist-packages/salt/states/pip_state.py", line 329, in _check_if_installed
                  desired_version = available_versions[-1]
              TypeError: 'NoneType' object is not subscriptable
     Started: 10:18:08.125016
    Duration: 2912.222 ms
     Changes:   

Summary for local
------------
Succeeded: 1
Failed:    1
------------
Total states run:     2
Total run time:   2.954 s
root@tdeb11:/srv/salt# 
root@tdeb11:/srv/salt# pip3 --version
pip 20.3.4 from /usr/lib/python3/dist-packages/pip (python 3.9)
root@tdeb11:/srv/salt#

dmurphy18 on Oct 12, 2021

@pckroon Thanks for using the SaltProject version, makes it easier to debug when coming from our code base. As to a Bullseye version available, there shall be with 3004rc1, in QA.

dmurphy18 on Oct 6, 2021