salt: grains: jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'os' when rendering Pillar

Description Having just upgraded to 3002.2, we are seeing random instances where Pillar fails to render due to core grains (specifically ‘os’) not being defined.

Exception in master log is as follows:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 260, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 505, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'os'
2020-12-29 19:08:01,847 [salt.pillar      :888 ][CRITICAL][3653] Rendering SLS 'global.os_defaults' failed, render error:
Jinja variable 'dict object' has no attribute 'os'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 498, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'os'

Setup The Pillar SLS being rendered which fails with the above is quite straightforward:

$ cat pillar/global/os_defaults.sls 
{% if grains['os'] == 'MacOS' %}
homedirbase: '/Users'
roothomedir: '/var/root'
{% elif grains['os'] == 'Ubuntu' %}
homedirbase: '/home'
roothomedir: '/root'
{% endif %}

Steps to Reproduce the behavior Unfortunately I’ve not found how to reliably reproduce this. We have a few hundred minions, and following the upgrade (we were previously at 2017.7.8) we’re observing this at random (i.e. most of the time, rendering happens correctly, but when it doesn’t, forcing a saltutil.refresh_grains + saltutil.refresh_pillar does the trick, but the problem can crop up again after an indeterminate amount of time).

I can attest that the ‘os’ grain is the only grain that’s failing to be defined, and this is only observed for MacOS minions.

Expected behavior grains[‘os’] to reliably evaluate within Pillar SLS definitions

Versions Report

Salt Version:
          Salt: 3002.2
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: 2.5.1
         gitdb: 2.0.3
     gitpython: 2.1.8
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.4.7
        pygit2: Not Installed
        Python: 3.6.9 (default, Jul 17 2020, 12:50:27)
  python-gnupg: 0.4.1
        PyYAML: 3.12
         PyZMQ: 17.1.2
         smmap: 2.0.3
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.2.5
 
System Versions:
          dist: ubuntu 18.04 Bionic Beaver
        locale: UTF-8
       machine: x86_64
       release: 4.15.0-112-generic
        system: Linux
       version: Ubuntu 18.04 Bionic Beaver

Salt Version:
          Salt: 3002.2
 
Dependency Versions:
          cffi: 1.12.2
      cherrypy: unknown
      dateutil: 2.8.0
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 2.1.15
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.0.7
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.19
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.7.4 (default, Nov 16 2020, 16:27:58)
  python-gnupg: 0.4.4
        PyYAML: 5.3.1
         PyZMQ: 18.0.1
         smmap: 3.0.2
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.1
 
System Versions:
          dist: darwin 19.6.0 
        locale: UTF-8
       machine: x86_64
       release: 19.6.0
        system: Darwin
       version: 10.15.7 x86_64