salt: [BUG] After upgrade to 3005.3, failure to fetch latest pillar data after being updated in git pillar repo branch

Description of Issue

A minion/master is unable to get the new pillar data from an alternative branch in git after it was updated on remote git pillar repository. This was working with 3005.1, but broke after upgrade to 3005.3. It also seem to be the case for those using 3006.3. Please refer to community discussion: https://saltstackcommunity.slack.com/archives/C7K04SEJC/p1697576330136199

Setup

Amazon Linux 2 salt master, configured with git repository for states and pillar data.

on-prem machine
VM (Virtualbox, KVM, etc. please specify)
VM running on a cloud service, please be explicit and add details
container (Kubernetes, Docker, containerd, etc. please specify)
jails if it is FreeBSD
classic packaging
onedir packaging
used bootstrap to install

Steps to Reproduce Issue

Master config

ext_pillar:
- git:
  - __env__ ssh://git@xxxxxxxxxxxxxxxxx/repo.git:
    - root: pillar
git_pillar_pubkey: xxxxxxxxxxxxxxxx
git_pillar_privkey: xxxxxxxxxxxxxxxxxxx
git_pillar_update_interval: 60
pillar_merge_lists: True

Push an update to an existant pillar called test in repo.git on branch testpillar. On a minion, run salt-call pillar.get test pillarenv=testpillar

Expected behavior

The new value of pillar test should be visible in the output of the salt call on the minion.

Actual behavior

The minion get the old data of pillar test. Please note that when executing the same comand after deleting /var/cache/salt/master/git_pillar, the minion get the correct value

Versions Report

Salt Version:
          Salt: 3005.3

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: 1.5.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.10.1
        Python: 3.9.18 (main, Sep 18 2023, 18:18:39)
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: amzn 2
        locale: utf-8
       machine: x86_64
       release: 4.14.318-241.531.amzn2.x86_64
        system: Linux
       version: Amazon Linux 2

About this issue

Original URL
State: open
Created 8 months ago
Comments: 17 (7 by maintainers)

Most upvoted comments

@Ikramromdhani Won’t make it for Salt 3006.7, that is close to release, waiting on clean tests in the pipeline. Possibly for 3007, but on my backlog, dealing with https://github.com/saltstack/salt/issues/65816, which may be related (similar code changes for pillarenv went in at same time for gitfs locks), writing tests for that fix now. So it is on the block but limited hands since the buyout. Get to it when I can.

dmurphy18 on Feb 15, 2024

We are also observing this error after recently upgrading from salt-3004 to salt-3006.4:

We have not moved to salt-3006.6 due to #65691
However diffing salt/utils/gitfs.py gives no reason to believe the bug is absent in 3006.6.

We have observed that when we use __env__ to effectively map pillarenv to a branch name that:

if we create a new branch feature/testing that is not already in the cache, this will be pulled correctly; and
if we then add a new commit on it, or amend HEAD, that the changes will not be pulled;

However, if we create a new branch named testing then it will be pulled, and subsequent changes to it will be pulled. That is, it seems that only branches with a / in them do not update. Unfortunately we use branch names like release/* or feature/*, like many people do so its a severe issue for us at the moment!

(This evidence conflicts with https://github.com/saltstack/salt/issues/65467#issuecomment-1921904438, although the commenter had not actually confirmed it themselves)

Note:

We are using GitPython rather than pygit2
We do not see this issue with gitfs … changes to our state files are pulled … it is only git_pillar

donkopotamus on Feb 15, 2024

@Ikramromdhani Can you upgrade to latest Salt 3006 and retry, Salt 3005 has passed it’s bug fix (last August 2023) and it about to run out of CVE support on the Feb 25th 2024, 24 days from now.

At this stage nothing will be getting done to Salt 3005 given that it is dead in 24 days, hardly time to debug/test/fix/release cycle, esp. since would have to build classic packages and there is a couple of days in just getting that done.

dmurphy18 on Feb 1, 2024