salt: [BUG] Grains values differ on master and minion

Description I get different results when querying the same grain via the Salt master vs querying directly on the minion. I believe this is a fairly recent issue, but I can’t say exactly which Salt version it started with.

Setup No special setup needed. In the minion configuration I have: grains_cache: False

In the master configuration I have: minion_data_cache: True

Changing minion_data_cache to False does not have any effect on this issue.

Steps to Reproduce the behavior

Querying the grain ip4_interfaces directly on the minion gives me the correct result

# salt-call --out json grains.get ip4_interfaces
{
    "local": {
        "vethdf8bed8": [],
        "lo": [
            "127.0.0.1"
        ],
        "enp0s31f6": [
            "192.168.1.21"
        ],
        "vethb734a2f": [],
        "docker0": [
            "172.17.42.1"
        ],
        "veth03da407": [],
...
        "veth42baa13": []
    }
}

Querying the grain ip4_interfaces on the master gives the wrong result

# salt --out json 'testminion' grains.get ip4_interfaces
{
    "testminion": {
        "lo": [
            "127.0.0.1"
        ],
        "enp0s31f6": [
            "192.168.1.21"
        ]
    }
}

The same difference can also be seen when querying ip_interfaces or ip6_interfaces.

Expected behavior I expect to get the same value for this grain on both the master and the minion. I also expect it to list all the interfaces currently defined on the minion.

Versions Report

salt --versions-report on the minion

Salt Version:
          Salt: 3002.2
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: 4.3.1
         gitdb: 2.0.5
     gitpython: 2.1.11
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.31.0
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.7.3 (default, Jul 25 2020, 13:03:44)
  python-gnupg: 0.4.4
        PyYAML: 3.13
         PyZMQ: 17.1.2
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.1
 
System Versions:
          dist: debian 10 buster
        locale: UTF-8
       machine: x86_64
       release: 4.19.0-14-amd64
        system: Linux
       version: Debian GNU/Linux 10 buster

salt --versions-report on the master

Salt Version:
          Salt: 3002.2
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.5
     gitpython: 2.1.11
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: 0.31.0
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.7.3 (default, Jul 25 2020, 13:03:44)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.1.2
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.1
 
System Versions:
          dist: debian 10 buster
        locale: utf-8
       machine: x86_64
       release: 4.19.0-14-amd64
        system: Linux
       version: Debian GNU/Linux 10 buster

Additional context Running saltutil.sync_all on the minion does not help. Only if I restart the Salt minion, then the master starts to show the correct interfaces and IPs. I believe the list of interfaces shown on the master is valid at the time the Salt minion originally started, which is before many other services started (including Docker, to which most of the missing interfaces are related).

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 18 (11 by maintainers)

Most upvoted comments

@xeacott Thanks! I can confirm that with this patch the master does see the change to the ip4_interfaces grain after running salt testminion saltutil.refresh_grains on the master or applying highstate on the minion.

I also did some additional testing and found that this first got broken with v3001.

kkoppel on May 7, 2021

Correct. Looks like the master never sees any changes to grains, unless the minion is restarted.

kkoppel on Apr 30, 2021