salt: Trying to reference existing pillars in new pillars (both first-level and subdirectories/init.sls) leads to unexpected behaviour

Description of Issue/Question

The documentation is not clear, and there are numerous issues on this repo about this.

What is the canonical, elegant way to access a pillar from a pillar?

I have an ext_pillar set up, and I have ext_pillar_first set to True in my master config.

I then need to further assign pillars based on what the pillars that are returned from ext_pillars are.

I’ve found that I can do this in a completely hacky and not at all proper way via opts[‘pillar’][‘ext_pillar_name’] (and such as a result from that dict) - in the pillar_roots top file with no issue.

But I need to consistently be able to access that same data in recursed pillars (e.g. accessing ext_pillar from <pillar_root>/something/init.sls, or even <pillar_root>/not_top.sls which is called from <pillar_root>/top.sls). When I try, it’s a gamble if it works (presumably a cache? Though I have pillar cache disabled) or if I get this:

jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'pillar'

In full:

Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/salt/utils/templates.py", line 393, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.4/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.4/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.4/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.4/site-packages/jinja2/environment.py", line 389, in getitem
    return obj[argument]
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'pillar'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/salt/utils/templates.py", line 170, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3.4/site-packages/salt/utils/templates.py", line 403, in render_jinja_tmpl
    buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'pillar'

Please, please, please, for the love of my sanity - why can’t I reliably access existing pillars in new pillars?

pillars.get('some:key') does not work. salt['pillars.get']('some:key') does not work.

The only thing that DOES work (albeit half the time) is the opts[‘’] hack. Grains, as expected, work fine but you can’t enforce those from the master.

Salt Version:
           Salt: 2018.3.3
 
Dependency Versions:
           cffi: 1.11.5
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.26.8
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.4
         Python: 3.4.9 (default, Aug 14 2018, 21:28:57)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

About this issue

Original URL
State: open
Created 5 years ago
Comments: 26 (4 by maintainers)

Most upvoted comments

Funny I stumbled upon this when a friend of mine tried ext_pillar_first: True the first time today.

We found another “workaround” for this if you just use the pillar dictionary: {{ pillar['key']['subkey'] }} I was wondering why nobody else mentioned this here and in the other referenced issues I’ve read and thought that might help at least some of you.

I was using this for years very successfully with highly complex pillar data (from a custom Netbox module and gitstack) because pillar and grains dictionaries are mentioned in a lot of Salt docs. But the “bug” or missing featue didn’t hit me because I always ran ext_pillar after pillar_roots. 🙂

Armadill0 on Nov 4, 2020

@johnnybubonic I’d avoid saltclass unless you want to deal with the open issues. The person who contributed it (@olivier-mauras) does not respond to mentions and has no activity on Github since June 2018 (life happens).

If you want to go this route, it is worth looking at reclass (Salt ships with builtin reclass adapters for both state and pillar trees). However, this pillar-heavy approach will lead to master performance problems at a certain scale (personally, I used it up to 80-100 nodes, folks at Mirantis reported that it works just fine for hundreds of nodes, and the real limit is unknown).

Also, there are other ext_pillar backends that try to solve the pillar inheritance problem in different ways (one example is varstack_pillar).

Generally speaking, you need to consider the following factors:

The scale of your infrastructure
The number of unique classes of server configurations (are they uniform, or there are lots of snowflakes?)
How dynamic is your infrastructure (disposable nodes, autoscaling, etc.)?
Security

For security, you would want to avoid assigning secrets to nodes based on untrusted data (anything that comes from a minion: grains, fileserver cache, etc. - the only trusted thing is grains.id). So these are best kept in a pillar.

For a highly dynamic infrastructure, reclass/saltclass is a bad fit because you need to create one file per node (although reclass has a workaround).

If the scale is small and you have many snowflakes, then reclass is the right fit. My advice would be to avoid being too DRY at first (it is very hard to design the right data model, and it is quite likely that you’ll end up with highly coupled states). Expect to refactor your state tree once or twice.

For high scale (thousands of nodes), the common advice is to minimize pillar usage (use it only to for sensitive data, or retrieve your secrets using the Vault sdb backend). All other values should be stored in json (or yaml) files in the salt:// file server and processed on minions using Jinja or custom modules.

Hope this helps.

max-arnold on Jan 31, 2019