salt: salt-call pillar overrides broken in 2016.11.8 and 2017.7.2

Description of Issue/Question

Pillar overrides using salt-call fail to pass updated pillar information to ext_pillar. salt, not salt-call, still works however. This worked in 2016.11.7, but due to “fixes” in 2016.11.8, the functionality was broken.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).) Using a command similar to the following and tracking the pillar information, you’ll notice that the override status option isn’t set. Similarly, using simpler options of pillar=‘{{“foo”: “bar”}}’ doesn’t change anything. /usr/bin/salt-call state.highstate pillar=‘{“override”:{“status”: “on”}}’

Steps to Reproduce Issue

(Include debug logs if possible and relevant.) Install salt* 2016.11.7 and pass pillar overrides via salt-call, NOT salt. Notice it works. Update salt* to 2016.11.8 and redo test, you’ll see it fails to work anymore.

It’s worth mentioning I’m using an external pillar and if I log the results of pillar in 2016.11.8 vs. 2016.11.7, I see the following in my logs: Oct 10 15:22:12 XXXXXXXXXXXXXXX salt-master: [CRITICAL] PILLAR: {} Oct 10 15:22:12 XXXXXXXXXXXXXXX journal: 2017-10-10 15:22:12,667 [salt.loaded.ext.pillar.roles][CRITICAL][26500] PILLAR: {} Oct 10 15:24:04 XXXXXXXXXXXXXXX salt-master: [CRITICAL] PILLAR: {} Oct 10 15:24:04 XXXXXXXXXXXXXXX journal: 2017-10-10 15:24:04,275 [salt.loaded.ext.pillar.roles][CRITICAL][26631] PILLAR: {} Oct 10 15:24:04 XXXXXXXXXXXXXXX salt-master: [CRITICAL] PILLAR: {‘override’: {‘status’: ‘on’}} Oct 10 15:24:04 XXXXXXXXXXXXXXX journal: 2017-10-10 15:24:04,942 [salt.loaded.ext.pillar.roles][CRITICAL][26632] PILLAR: {‘override’: {‘status’: ‘on’}}

Oddly enough, running /usr/bin/salt-call pillar.items pillar='{"override":{"status": "on"}}' does show the override in there. For now, I’m reverted back to 2016.11.7 due to this bug and the salt-api bug reported in #43581 which didn’t start manifesting itself til upgrading to 2016.11.8.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) Salt Version: Salt: 2016.11.8

Dependency Versions: cffi: 1.6.0 cherrypy: unknown dateutil: 1.5 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.7.2 libgit2: Not Installed libnacl: 1.4.3 M2Crypto: 0.21.1 Mako: 0.8.1 msgpack-pure: Not Installed msgpack-python: 0.4.8 mysql-python: 1.2.5 pycparser: 2.14 pycrypto: 2.6.1 pycryptodome: 3.4.3 pygit2: Not Installed Python: 2.7.5 (default, Nov 6 2016, 00:28:07) python-gnupg: Not Installed PyYAML: 3.11 PyZMQ: 15.3.0 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.2.1 ZMQ: 4.1.4

System Versions: dist: centos 7.3.1611 Core machine: x86_64 release: 4.9.54-1.el7.centos.x86_64 system: Linux version: CentOS Linux 7.3.1611 Core

BTW, if due to all the pillar override changes made in 2016.11.8, this changed the way pillar is passed to ext_pillar, could we get the documentation updated on how to utilize ext_pillar and pillar overrides, assuming that’s even the root issue.