salt: Upgrading Salt via Salt results in dying minions and broken dpkg

Description of Issue/Question

We use pkg.upgrade dist_upgrade=True to upgrade our servers semi-unattended.

They were running 2017.7.0 prior to issuing the upgrade command and were supposed to be upgraded to 2017.7.1.

However, a majority of the machines had issues where the salt-minion would not be running anymore afterwards and shows the following issue when doing anything with apt afterwards on the machine itself:

E: dpkg was interrupted, you must manually run 'dpkg —configure -a' to correct the problem.

Doing so shows the following issue:

dpkg: error processing package salt-minion (--configure):
 package is in a very bad inconsistent state; you should
 reinstall it before attempting configuration

After running the commands above, this

ISTM that this issue may be caused by the fact that the systemd config has changed, since it seems to only affect our Ubuntu 16.04 machines and works fine on 14.04.

Versions Report

Salt Version:
           Salt: 2017.7.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.4.59-1-pve
         system: Linux
        version: Ubuntu 16.04 xenial

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 19 (14 by maintainers)

Most upvoted comments

Ok, i have been able to replicate that, the problem is the fact that the KillMode=process is not in place when the debian scripts do what they do, which is stop the services before laying down the files, then start it again after laying them down.

This should be fully fixed with the next upgrade, but I will make sure that it gets tested and is verified that it will work starting with 2017.7.4

Thanks, Daniel

gtmanfred on Feb 6, 2018

I deployed the following before the upgrade, which allowed for a smooth one:

salt_minion_killmode_drop_in:
  file.managed:
    - name: /etc/systemd/system/salt-minion.service.d/killmode.conf
    - makedirs: True
    - dir_mode: 755
    - contents:
      - '[Service]'
      - 'KillMode=process'

reload_daemons:
  cmd.run:
    - name: 'systemctl daemon-reload'
    - onchanges_in:
      - file: salt_minion_killmode_drop_in

wwentland on Mar 9, 2018

I have confirmed that the upgrade to 2017.7.4 goes smoothly using our staging repository.

root@salt:~# salt \* pkg.upgrade refresh=true
salt.c.gtmanfred-1263.internal:
    ----------
    gcc-6-base:
        ----------
        new:
            6.3.0-18+deb9u1
        old:
            6.3.0-18
    google-cloud-sdk:
        ----------
        new:
            190.0.1-0
        old:
            187.0.0-0
    libgcc1:
        ----------
        new:
            1:6.3.0-18+deb9u1
        old:
            1:6.3.0-18
    libstdc++6:
        ----------
        new:
            6.3.0-18+deb9u1
        old:
            6.3.0-18
    libtasn1-6:
        ----------
        new:
            4.10-1.1+deb9u1
        old:
            4.10-1.1
    salt-common:
        ----------
        new:
            2017.7.4+ds-1
        old:
            2017.7.3+ds-1
    salt-master:
        ----------
        new:
            2017.7.4+ds-1
        old:
            2017.7.3+ds-1
    salt-minion:
        ----------
        new:
            2017.7.4+ds-1
        old:
            2017.7.3+ds-1
    tzdata:
        ----------
        new:
            2018c-0+deb9u1
        old:
            2017c-0+deb9u1

gtmanfred on Feb 23, 2018

The same thing is happening again with the upgrade of 2017.7.2:

root@web01:~# dpkg -l | grep salt
ii  salt-common                      2017.7.1+ds-1                              all          shared libraries that salt requires for all packages
iFR salt-minion                      2017.7.1+ds-1                              all          client package for salt, the distributed remote execution system

root@web01:~# apt -f install
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. 
root@web01:~# sudo dpkg --configure -a

dpkg: error processing package salt-minion (--configure):
 package is in a very bad inconsistent state; you should
 reinstall it before attempting configuration

Errors were encountered while processing:
 salt-minion

syphernl on Oct 11, 2017