salt: docker_image.present fails first time when python3-docker module is installed

Description of Issue

I’m trying to use docker_image.present but salt fails to detect the module on first state.apply because the required python modules are installed in that run. On second state.apply the module is detected and used. This is even the case with reload_modules: true set.

On the first run I see:

----------
          ID: Docker
    Function: pkg.installed
      Result: True
     Comment: The following packages were installed/updated: python3-docker
              The following packages were already installed: docker-ce
     Started: 16:12:02.098635
    Duration: 1403.352 ms
     Changes:   
              ----------
              python3-docker:
                  ----------
                  new:
                      1.9.0-1
                  old:
              python3-websocket:
                  ----------
                  new:
                      0.37.0-2
                  old:
----------
          ID: Kubernetes Dependency Image k8s.gcr.io/pause
    Function: docker_image.present
        Name: k8s.gcr.io/pause
      Result: False
     Comment: State 'docker_image.present' was not found in SLS 'kube-minion'
              Reason: 'docker_image' __virtual__ returned False: 'docker.version' is not available.
     Changes:

On the minion itself the salt-minion logs have this error:

salt-minion[138912]: [ERROR   ] State 'docker_image.present' was not found in SLS 'kube-minion'
salt-minion[138912]: Reason: 'docker_image' __virtual__ returned False: 'docker.version' is not available.

However, it works on second run:

----------
          ID: Docker
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed and are at the desired version
     Started: 16:29:44.739748
    Duration: 26.875 ms
     Changes:  
----------
          ID: Kubernetes Dependency Image k8s.gcr.io/pause
    Function: docker_image.present
        Name: k8s.gcr.io/pause
      Result: True
     Comment: Image 'k8s.gcr.io/pause:3.1' was pulled
     Started: 16:29:44.859688
    Duration: 1079.729 ms
     Changes:   
              ----------
              Layers:
                  ----------
                  Pulled:
                      - 67ddbfb20a22
              Status:
                  Downloaded newer image for k8s.gcr.io/pause:3.1
              Time_Elapsed:
                  0.4564225673675537
              retcode:
                  0

Setup

I have two state files. docker.sls:

Docker:
  pkg.installed:
    - pkgs:
        - docker-ce: 18.06.1~ce~3-0~debian
        - python3-docker
    - install_recommends: False
    - reload_modules: true

and kube.sls:

include:
  - docker

Kubernetes Dependency Image k8s.gcr.io/pause:
  docker_image.present:
    - name: k8s.gcr.io/pause
    - tag: 3.1
    - reload_modules: True
    - require:
      - pkg: Docker
      - service: Docker
      - pkg: Kubernetes Dependencies

This may be related to #54332 but in that case pip is the installation method and the error message is different.

Steps to Reproduce Issue

  1. Run salt <minion> state.apply
  2. Get Failure
  3. Run salt <minion> state.apply Success!

Versions Report

On Master:

Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.9 
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-9-amd64
         system: Linux
        version: debian 9.9 

On the Minion (after python3-docker has been installed):

Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: 1.9.0
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.9 
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-9-amd64
         system: Linux
        version: debian 9.9 

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 8
  • Comments: 35 (17 by maintainers)

Commits related to this issue

Most upvoted comments

I actually am experiencing a very similar problem, while configuring Docker + running a container straight away. Basically if I install any module (regardless it is related or not with docker; in this example only the nox module was installed via pip, all the rest using the pkg state) via pip state, the highstate is gonna fail with:

    Function: docker_container.running
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python3/dist-packages/urllib3/util/timeout.py", line 124, in _validate_timeout
                  float(value)
              TypeError: float() argument must be a string or a number, not 'Timeout'

              During handling of the above exception, another exception occurred:

              Traceback (most recent call last):
                File "/usr/lib/python3/dist-packages/docker/api/client.py", line 175, in _retrieve_server_version
                  return self.version(api_version=False)["ApiVersion"]
                File "/usr/lib/python3/dist-packages/docker/api/daemon.py", line 181, in version
                  return self._result(self._get(url), json=True)
                File "/usr/lib/python3/dist-packages/docker/utils/decorators.py", line 46, in inner
                  return f(self, *args, **kwargs)
                File "/usr/lib/python3/dist-packages/docker/api/client.py", line 198, in _get
                  return self.get(url, **self._set_request_timeout(kwargs))
                File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
                  return self.request('GET', url, **kwargs)
                File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
                  resp = self.send(prep, **send_kwargs)
                File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
                  r = adapter.send(request, **kwargs)
                File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
                  timeout=timeout
                File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 587, in urlopen
                  timeout_obj = self._get_timeout(timeout)
                File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 300, in _get_timeout
                  return Timeout.from_float(timeout)
                File "/usr/lib/python3/dist-packages/urllib3/util/timeout.py", line 154, in from_float
                  return Timeout(read=timeout, connect=timeout)
                File "/usr/lib/python3/dist-packages/urllib3/util/timeout.py", line 94, in __init__
                  self._connect = self._validate_timeout(connect, 'connect')
                File "/usr/lib/python3/dist-packages/urllib3/util/timeout.py", line 127, in _validate_timeout
                  "int, float or None." % (name, value))
              ValueError: Timeout value connect was Timeout(connect=60, read=60, total=None), but it must be an int, float or None.

              During handling of the above exception, another exception occurred:

              Traceback (most recent call last):
                File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
                  *cdata["args"], **cdata["kwargs"]
                File "/usr/lib/python3/dist-packages/salt/loader.py", line 2106, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python3/dist-packages/salt/states/docker_container.py", line 1687, in running
                  image_id = _resolve_image(ret, image, client_timeout)
                File "/usr/lib/python3/dist-packages/salt/states/docker_container.py", line 187, in _resolve_image
                  image_id = __salt__["docker.resolve_image_id"](image)
                File "/usr/lib/python3/dist-packages/salt/modules/dockermod.py", line 2057, in resolve_image_id
                  inspect_result = inspect_image(name)
                File "/usr/lib/python3/dist-packages/salt/modules/dockermod.py", line 1989, in inspect_image
                  ret = _client_wrapper("inspect_image", name)
                File "/usr/lib/python3/dist-packages/salt/modules/dockermod.py", line 451, in wrapper
                  __context__["docker.client"] = _get_client(timeout=timeout, **kwargs)
                File "/usr/lib/python3/dist-packages/salt/modules/dockermod.py", line 411, in _get_client
                  ret = docker.APIClient(**client_kwargs)
                File "/usr/lib/python3/dist-packages/docker/api/client.py", line 158, in __init__
                  self._version = self._retrieve_server_version()
                File "/usr/lib/python3/dist-packages/docker/api/client.py", line 183, in _retrieve_server_version
                  'Error while fetching server API version: {0}'.format(e)
              docker.errors.DockerException: Error while fetching server API version: Timeout value connect was Timeout(connect=60, read=60, total=None), but it must be an int, float or None.
     Started: 16:53:52.451899
    Duration: 19.067 ms
     Changes:

If I run the very same highstate once more, it’ll work. On the other hand, if I strip any pip state out of the highstate (e.g., temporarily exclude nox from being installed), it’ll work at the first try. Obviously, that can’t be a viable solution, as even the simplest states usually include pip dependencies.

I have recently run into this on raspbian bullseye, with the latest salt (3005.1). This time the pip docker version 6 is the issue, downgrading it to 5.0.3 fixes the issue for me.

This seems to affect docker_network with test=True the same way.