salt: [BUG] win_pkg weirdness in 3006.1

Description win_pkg is behaving strangely when updating an application. I have about 380 Windows minions all running 3006.1.

A new version of TeamViewer came out and I updated my custom definition. I told all the minions to refresh the database. I told all the minions to install the latest version. Only ~130 out of the 380 updated the application…but all of them reported some level of success.

Setup

My cait_teamviewer.sls definition:

{%
set versions = [
  '15.42.9.0',
  '15.42.8.0'
]
%}
cait_teamviewer:
  {% for version in versions %}
  '{{ version }}':
    full_name: 'TeamViewer Host'
    installer: 'https://cdn-redacted/TeamViewer_Host.msi'
    uninstaller: 'https://cdn-redacted/TeamViewer_Host.msi'
    arch: x86
    install_flags: '/qn /norestart CUSTOMCONFIGID="--redacted--" ASSIGNMENTOPTIONS="--alias %ComputerName% --grant-easy-access --reassign" APITOKEN="--redacted--"'
    uninstall_flags: '/qn /norestart'
    msiexec: True
    locale: en_US
    reboot: False
{% endfor %}

Please be as specific as possible and give set-up details.

• onedir packaging

Steps to Reproduce the behavior Here’s one of the minions that failed to update:

salt 'USSNISDOFC03*' test.version

USSNISDOFC03.redacted.local:
    3006.1



salt 'USSNISDOFC03*' pkg.refresh_db
USSNISDOFC03.redacted.local:
    ----------
    failed:
        0
    success:
        22
    total:
        22

salt 'USSNISDOFC03*' pkg.latest_version cait_teamviewer

USSNISDOFC03.redacted.local:
    15.42.9.0   <-- this is correct


salt 'USSNISDOFC03*' pkg.list_available cait_teamviewer

USSNISDOFC03.redacted.local:
    - 15.42.8.0
    - 15.42.9.0  <-- it obviously sees the latest version

salt 'USSNISDOFC03*' pkg.upgrade_available cait_teamviewer

USSNISDOFC03.redacted.local:
    True  <-- Yup, there's an upgrade available

salt 'USSNISDOFC03*' pkg.install cait_teamviewer version=latest
USSNISDOFC03.redacted.local:
    ----------
    cait_teamviewer:
        ----------
        install status:
            success

# You'll notice this isn't the "normal" return for a system that upgrades properly.
# Normally you see the old version and the new version listed


salt 'USSNISDOFC03*' pkg.install cait_teamviewer version=15.42.9.0

USSNISDOFC03.redacted.local:
    ----------
    cait_teamviewer:
        ----------
        install status:
            success

After running that, I connect in to the machine and see an older version of TeamViewer installed.

If I run the Windows Installer manually:

salt 'USSNISDOFC03*' cmd.run 'msiexec /i https://cdn-redacted/TeamViewer_Host.msi /qn /norestart CUSTOMCONFIGID="-redacted-" ASSIGNMENTOPTIONS="--alias %ComputerName% --grant-easy-access --reassign" APITOKEN="-redacted-"

USSNISDOFC03.redacted.local:


-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 1
# of minions returned: 1
# of minions that did not return: 0
# of minions with errors: 0
-------------------------------------------

salt 'USSNISDOFC03*' pkg.upgrade_available cait_teamviewer
USSNISDOFC03.redacted.local:
    False

…and I can sign in to the machine and see it’s running the latest version.

Contrast all of that with a machine that worked correctly:

USBVESDOFC01.redacted.local:
    ----------
    cait_teamviewer:
        ----------
        new:
            15.42.9.0
        old:
            15.42.8.0

Expected behavior I would expect it to either report an error (if there was one) or that the upgrade was successful, but it appears to be reporting a successful upgrade without showing the version numbers when it really didn’t do anything.

Minions are all Windows 10 or Windows 11 64-bit machines. The successful machines are a mix of Windows 10 and 11. The failing machines are a mix of Windows 10 and 11.

I’m stumped as to why it only works on some machines, but my manual msiexec command (which appears to match what pkg.installed runs on successful machines) works perfectly.

Is there a minion cache of installer files I’m unaware of that isn’t updated by pkg.refresh_db?