salt: spurious warnings from event.wait when event is not fired

I got the same issue, with Salt 2016.11

Doesn’t look stale…

This is still happening with 2016.11.3.

I’ve followed the documentation for event.wait and have the following states:

apache2:
  service.running:
    - reload: True

api-checks:
  event.wait:
    - name: iland/monitor/api
    - data:
        nodename: {{ grains['nodename'] }}
        ip: {{ grains['ipv4'] | first() }}
    - watch:
      - service: apache2

When there are changes applied during the highstate and the apache2 service is reloaded then I see this on the event bus:

'event_|-api-checks_|-iland/monitor/api_|-wait': {
  '__id__': 'api-checks',
  '__run_num__': 104,
  'changes': {
    'data': {
      'ip': '10.144.252.102',
      'nodename': 'api-proxy01.dev'
    },
    'tag': 'iland/monitor/api'
  },
  'comment': 'Event fired',
  'duration': 31.379,
  'name': 'iland/monitor/api',
  'result': True,
  'start_time': '17:45:33.678152'

However, when the apache2 state does not need to be reloaded and therefore the api-checks state is not run, I receive a warning:

'event_|-api-checks_|-iland/monitor/api_|-wait': {
  '__id__': 'api-checks',
  '__run_num__': 102,
  'changes': {},
  'comment': '',
  'duration': 0.731,
  'name': 'iland/monitor/api',
  'result': True,
  'start_time': '17:43:57.976954',
  'warnings': ["'data' is an invalid keyword argument for 'event.wait'.  If you were \
                trying to pass additional data to be used in a template context, \
                please populate 'context' with 'key: value' pairs. Your approach \
                will work until Salt Oxygen is out. Please update your state files."]
}

In preparation for https://github.com/saltstack/community/blob/master/doc/Community-Message.pdf I’m getting all of our hosts running the 2019.2.x series of Salt and I just hit this. Ouch! Now I remember why I couldn’t upgrade…

Thanks @waynew and @corywright for both investigating and confirming that **kwargs really was all that was required.

Doing some research on this:

• wait was created in this commit
• it was further refined in this commit, where it stay basically unchanged. The comment on the return was “Noop . The state system will call the mod_watch function instead”. The documentation also indicated that supplying arbitrary data was A-OK.
• the behavior was changed in Carbon (2016.11.0) to fail where arguments were unsupported. But it doesn’t appear that this was ever intended to be the case here.

Additionally, I can send this:

salt-call event.send 'bad/warning' arbitrary=data totally=allowed sfun=wait

Which results in this event:

bad/warning     {
    "_stamp": "2020-03-26T20:30:41.228118",
    "cmd": "_minion_event",
    "data": {
        "__pub_fun": "event.send",
        "__pub_jid": "20200326203041214471",
        "__pub_pid": 890,
        "__pub_tgt": "salt-call",
        "arbitrary": "data",
        "sfun": "wait",
        "totally": "allowed"
    },
    "id": "kevin",
    "tag": "bad/warning"
}

If I add a **kwargs parameter to states/event.py wait and then use this state:

funtimes:
  test.succeed_with_changes

Some Event:
  event.wait:
    - name: bad/warning
    - data:
        arbitrary: data
        totally: allowed
    - watch:
      - test: funtimes

Running state.apply results in this event:

bad/warning     {
    "_stamp": "2020-03-26T20:30:41.228118",
    "cmd": "_minion_event",
    "data": {
        "__pub_fun": "event.send",
        "__pub_jid": "20200326203041214471",
        "__pub_pid": 890,
        "__pub_tgt": "salt-call",
        "arbitrary": "data",
        "sfun": "wait",
        "totally": "allowed"
    },
    "id": "kevin",
    "tag": "bad/warning"
}

If you remove the extra bits, data contains the exact same information.

Based on this, the correct fix is to add **kwargs to states.event.wait.

For a current workaround, I would recommend doing something like:

mkdir -p /srv/salt/_states && cp $(python3 -c 'import salt.states.event; print(salt.states.event.__file__)') /srv/salt/_states/event.py
sed -i 's/sfun=None)/sfun=None, **kwargs)/` /srv/salt/_states/event.py
salt \* saltutil.sync_all
```
After that it you should no longer get this error. There are some upcoming changes to `event.py` in future versions of Salt, so you should make sure to keep this file up to date until we get the fix in a released version.

HTH!

As @bicofino says the event.wait function is broken in 2019.2.0. It stops running the state on minion and returns:

Passed invalid arguments: 'data' is an invalid keyword argument for 'event.wait'.

This is critical for us, we have a lot of installations using event.wait as event.send can’t handle the watch function.

With the new salt-minion version(salt-2019.2.0-1) it broke.

Passed invalid arguments: 'data' is an invalid keyword argument for 'event.wait'.

The docs are not clear https://docs.saltstack.com/en/latest/ref/states/all/salt.states.event.html