salt: salt-minion fails to start "ERROR: Unable to look-up config values for /etc/salt"

Description of Issue/Question

I upgraded the salt-master to 2016.11.3, then upgraded salt-minion on the same server. The master is OK, but the minion fails to start with this error:

root@[salt-master]:~$service salt-minion start
ERROR: Unable to look-up config values for /etc/salt

I looked in the init script and that error appears to come from a failure to run this for various config items:

_get_salt_config_value() {
    _su_cmd \
        "$MINION_USER" \
        "\
            \"$SALTCALL\" \
            -c \"$CONFIG_DIR\" \
            --no-color \
            --local config.get \
            \"$1\" \
        " \
        2>$ERROR_TO_DEVNULL \
        | sed -r -e '2!d; s/^\s*//;'

If I run salt-call -c /etc/salt/minion --local config.get sock_dir directly I get (which is also the output in /var/log/salt/minion when I try to start the service):

root@[salt-master]:~$salt-call -c /etc/salt --local config.get sock_dir
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
KeyError: 'pillar'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.6/site-packages/salt/scripts.py", line 379, in salt_call
    client.run()
  File "/usr/lib/python2.6/site-packages/salt/cli/call.py", line 48, in run
    caller = salt.cli.caller.Caller.factory(self.config)
  File "/usr/lib/python2.6/site-packages/salt/cli/caller.py", line 79, in factory
    return ZeroMQCaller(opts, **kwargs)
  File "/usr/lib/python2.6/site-packages/salt/cli/caller.py", line 274, in __init__
    super(ZeroMQCaller, self).__init__(opts)
  File "/usr/lib/python2.6/site-packages/salt/cli/caller.py", line 102, in __init__
    self.minion = salt.minion.SMinion(opts)
  File "/usr/lib/python2.6/site-packages/salt/minion.py", line 658, in __init__
    self.gen_modules(initial_load=True)
  File "/usr/lib/python2.6/site-packages/salt/minion.py", line 689, in gen_modules
    pillarenv=self.opts.get('pillarenv'),
  File "/usr/lib/python2.6/site-packages/salt/pillar/__init__.py", line 836, in compile_pillar
    matches = self.top_matches(top)
  File "/usr/lib/python2.6/site-packages/salt/pillar/__init__.py", line 564, in top_matches
    self.opts.get('nodegroups', {}),
  File "/usr/lib/python2.6/site-packages/salt/minion.py", line 2732, in confirm_top
    return getattr(self, funcname)(match)
  File "/usr/lib/python2.6/site-packages/salt/minion.py", line 2965, in compound_match
    str(getattr(self, '{0}_match'.format(engine))(*engine_args, **engine_kwargs))
  File "/usr/lib/python2.6/site-packages/salt/minion.py", line 2826, in pillar_match
    self.opts['pillar'], tgt, delimiter=delimiter
KeyError: 'pillar'

If I simply run /usr/bin/salt-minion directly it connects to the master and I can apply states to it.

Setup

minion config:

master: localhost
id: master

I’ve tried yum removing and yum installing the salt-minion software.

Versions Report

Salt Version:
           Salt: 2016.11.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: Not Installed
          gitdb: 0.6.4
      gitpython: 2.0.8
          ioflo: Not Installed
         Jinja2: 2.7.3
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.20.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: centos 6.7 Final
        machine: x86_64
        release: 2.6.32-573.26.1.el6.x86_64
         system: Linux
        version: CentOS 6.7 Final

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 31 (14 by maintainers)

Most upvoted comments

Sorry it took me so long to get back to this.

The problem is you are targeting with pillars in the pillar top file.

Traceback (most recent call last):
  File "/bin/salt-call", line 8, in <module>
    execfile(__file__)
  File "/root/src/salt/scripts/salt-call", line 11, in <module>
    salt_call()
  File "/root/src/salt/salt/scripts.py", line 372, in salt_call
    client.run()
  File "/root/src/salt/salt/cli/call.py", line 48, in run
    caller = salt.cli.caller.Caller.factory(self.config)
  File "/root/src/salt/salt/cli/caller.py", line 79, in factory
    return ZeroMQCaller(opts, **kwargs)
  File "/root/src/salt/salt/cli/caller.py", line 274, in __init__
    super(ZeroMQCaller, self).__init__(opts)
  File "/root/src/salt/salt/cli/caller.py", line 102, in __init__
    self.minion = salt.minion.SMinion(opts)
  File "/root/src/salt/salt/minion.py", line 629, in __init__
    self.gen_modules(initial_load=True)
  File "/root/src/salt/salt/minion.py", line 660, in gen_modules
    pillarenv=self.opts.get('pillarenv'),
  File "/root/src/salt/salt/pillar/__init__.py", line 851, in compile_pillar
    matches = self.top_matches(top)
  File "/root/src/salt/salt/pillar/__init__.py", line 581, in top_matches
    self.opts.get('nodegroups', {}),
  File "/root/src/salt/salt/minion.py", line 2756, in confirm_top
    return getattr(self, funcname)(match)
  File "/root/src/salt/salt/minion.py", line 2989, in compound_match
    str(getattr(self, '{0}_match'.format(engine))(*engine_args, **engine_kwargs))
  File "/root/src/salt/salt/minion.py", line 2850, in pillar_match
    self.opts['pillar'], tgt, delimiter=delimiter
KeyError: 'pillar'

The only way you can do this is by using ext_pillars. If you aren’t using ext_pillars and have ext_pillar_first: True set, then it will fail.

set ext_pillar_first: True in the minion config (since --local is used`) and this will resolve the problem

Thanks, Daniel

gtmanfred on Apr 13, 2017