salt: [BUG] After upgrade to 3005.3, failure to fetch latest pillar data after being updated in git pillar repo branch

Description of Issue

A minion/master is unable to get the new pillar data from an alternative branch in git after it was updated on remote git pillar repository. This was working with 3005.1, but broke after upgrade to 3005.3. It also seem to be the case for those using 3006.3. Please refer to community discussion: https://saltstackcommunity.slack.com/archives/C7K04SEJC/p1697576330136199

Setup

Amazon Linux 2 salt master, configured with git repository for states and pillar data.

  • VM (Virtualbox, KVM, etc. please specify)
  • on-prem machine
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce Issue

Master config

ext_pillar:
- git:
  - __env__ ssh://git@xxxxxxxxxxxxxxxxx/repo.git:
    - root: pillar
git_pillar_pubkey: xxxxxxxxxxxxxxxx
git_pillar_privkey: xxxxxxxxxxxxxxxxxxx
git_pillar_update_interval: 60
pillar_merge_lists: True

Push an update to an existant pillar called test in repo.git on branch testpillar. On a minion, run salt-call pillar.get test pillarenv=testpillar

Expected behavior

The new value of pillar test should be visible in the output of the salt call on the minion.

Actual behavior

The minion get the old data of pillar test. Please note that when executing the same comand after deleting /var/cache/salt/master/git_pillar, the minion get the correct value

Versions Report

Salt Version:
          Salt: 3005.3

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: 1.5.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.10.1
        Python: 3.9.18 (main, Sep 18 2023, 18:18:39)
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: amzn 2
        locale: utf-8
       machine: x86_64
       release: 4.14.318-241.531.amzn2.x86_64
        system: Linux
       version: Amazon Linux 2

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 76 (38 by maintainers)

Most upvoted comments

@dmurphy18 As @OrangeDog notes, substituting '/' -> '_' precludes the possibility of having branches named both a/b and a-b.

Git ref names are specifically designed to generally map well, and be safe, on filesystems. The current method of mapping them onto a directory structure does not seem wrong in any sense … its simpy the traversing of that structure that is wrong.

i.e I think:

  • the branch name a/b/c should continue to be mapped to the directory a/b/c (thats exactly how git itself would represent the branch in a checked out repository, by using .git/refs/heads/a/b/c
  • instead of using os.listdir to traverse that structure, it should simply be walked with os.walk

@dmurphy18 @Ikramromdhani Can you review again this sequence of comments on this issue, which I believe identifies the exact cause of the issue:

The latter in particular identifies the problem as the piece of code that is responsible for dropping fetch_request marker sentinels on the filesystem (see my comments embedded in it such as <== PROBLEM HERE!!).

I have been unable to reproduce this with GitPython

@dmurphy18 Also relevant in there is the following:

  • this problem only occurs if you are using an __env__ definition for git_pillar. i.e. this breaks:

    ext_pillar:
      - git:
        ...
    
        # map other environments to the corresponding branch
        - __env__  git@.../some-repo.git
          - root: salt/pillar
    

    but this does not

    - git:
     ...
    
     # 
     - another/branch  git@.../some-repo.git
       - root: salt/pillar
    

Can you confirm that you are using an __env__ definition when attempting to reproduce it?

A WORKING FIX

If I alter the mentioned code to instead walk the tree at repo_work_hash, rather than simply using listdir to drop sentinels at the top level, then a fetch_request is placed at all directory nodes and then those branches such as some/branch are definitely updated.

@Ikramromdhani I, for example, no longer blow away the entire cache directory … instead I simply

sudo touch /var/cache/salt/master/git_pillar/work/<opaque-hash>/some/branch/fetch_request

on the branch I need fixed and 60 seconds later that branch will have been updated. (Clearly I don’t like having to do this manually all the time though!)

Given this appears to be an issue with ‘/’ in the name of the field, I think changes from PR 65017 to salt/utils/gitfs.py, could be the culprit, the added

        if hasattr(self, "name"):
            self._cache_basehash = self.name
        else:
            hash_type = getattr(hashlib, self.opts.get("hash_type", "md5"))
            # We loaded this data from yaml configuration files, so, its safe
            # to use UTF-8
            self._cache_basehash = str(
                base64.b64encode(hash_type(self.id.encode("utf-8")).digest()),
                encoding="ascii",  # base64 only outputs ascii
            ).replace(
                "/", "_"
            )  # replace "/" with "_" to not cause trouble with file system

whereas previously just did the decode and replace of ‘/’ with ‘_’. Here the now presented name is not getting sanitized, and functions such as os.listdir will interpret the ‘/’ as a path separator.

This is just a thought that I wanted to record and not forget, not saying its the problem, just a probable cause

@Ikramromdhani Sorry but this is on the back-burner and I have to get back to it, given the reduced size of the team, some fireballs got thrown which had to be handled. Be assured it is on the list, and the fix is half done, but tests to be written if memory recall is correct. Apologies but only so much time in the day, with a smaller team, and a laptop rebuild didn’t help either.

@Ikramromdhani Working on fix for Git FS and the Git Pillar is right after it, fix for Git FS is done but have to refractor a new test after PR review (https://github.com/saltstack/salt/pull/65937), after that then work on Git Pillar since it will be possibly affected by GitFS changes.

@Ikramromdhani Won’t make it for Salt 3006.7, that is close to release, waiting on clean tests in the pipeline. Possibly for 3007, but on my backlog, dealing with https://github.com/saltstack/salt/issues/65816, which may be related (similar code changes for pillarenv went in at same time for gitfs locks), writing tests for that fix now. So it is on the block but limited hands since the buyout. Get to it when I can.

We are also observing this error after recently upgrading from salt-3004 to salt-3006.4:

  • We have not moved to salt-3006.6 due to #65691
  • However diffing salt/utils/gitfs.py gives no reason to believe the bug is absent in 3006.6.

We have observed that when we use __env__ to effectively map pillarenv to a branch name that:

  • if we create a new branch feature/testing that is not already in the cache, this will be pulled correctly; and
  • if we then add a new commit on it, or amend HEAD, that the changes will not be pulled;

However, if we create a new branch named testing then it will be pulled, and subsequent changes to it will be pulled. That is, it seems that only branches with a / in them do not update. Unfortunately we use branch names like release/* or feature/*, like many people do so its a severe issue for us at the moment!

(This evidence conflicts with https://github.com/saltstack/salt/issues/65467#issuecomment-1921904438, although the commenter had not actually confirmed it themselves)

Note:

  • We are using GitPython rather than pygit2
  • We do not see this issue with gitfs … changes to our state files are pulled … it is only git_pillar

@Ikramromdhani Can you upgrade to latest Salt 3006 and retry, Salt 3005 has passed it’s bug fix (last August 2023) and it about to run out of CVE support on the Feb 25th 2024, 24 days from now.

At this stage nothing will be getting done to Salt 3005 given that it is dead in 24 days, hardly time to debug/test/fix/release cycle, esp. since would have to build classic packages and there is a couple of days in just getting that done.