salt: [BUG] After upgrade to 3005.3, failure to fetch latest pillar data after being updated in git pillar repo branch
Description of Issue
A minion/master is unable to get the new pillar data from an alternative branch in git after it was updated on remote git pillar repository. This was working with 3005.1, but broke after upgrade to 3005.3. It also seem to be the case for those using 3006.3. Please refer to community discussion: https://saltstackcommunity.slack.com/archives/C7K04SEJC/p1697576330136199
Setup
Amazon Linux 2 salt master, configured with git repository for states and pillar data.
- VM (Virtualbox, KVM, etc. please specify)
- on-prem machine
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce Issue
Master config
ext_pillar:
- git:
- __env__ ssh://git@xxxxxxxxxxxxxxxxx/repo.git:
- root: pillar
git_pillar_pubkey: xxxxxxxxxxxxxxxx
git_pillar_privkey: xxxxxxxxxxxxxxxxxxx
git_pillar_update_interval: 60
pillar_merge_lists: True
Push an update to an existant pillar called test in repo.git on branch testpillar. On a minion, run salt-call pillar.get test pillarenv=testpillar
Expected behavior
The new value of pillar test should be visible in the output of the salt call on the minion.
Actual behavior
The minion get the old data of pillar test. Please note that when executing the same comand after deleting /var/cache/salt/master/git_pillar, the minion get the correct value
Versions Report
Salt Version:
Salt: 3005.3
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.0
libgit2: 1.5.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: 1.10.1
Python: 3.9.18 (main, Sep 18 2023, 18:18:39)
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: amzn 2
locale: utf-8
machine: x86_64
release: 4.14.318-241.531.amzn2.x86_64
system: Linux
version: Amazon Linux 2
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 76 (38 by maintainers)
@dmurphy18 As @OrangeDog notes, substituting
'/' -> '_'precludes the possibility of having branches named botha/banda-b.Git ref names are specifically designed to generally map well, and be safe, on filesystems. The current method of mapping them onto a directory structure does not seem wrong in any sense … its simpy the traversing of that structure that is wrong.
i.e I think:
a/b/cshould continue to be mapped to the directorya/b/c(thats exactly howgititself would represent the branch in a checked out repository, by using.git/refs/heads/a/b/cos.listdirto traverse that structure, it should simply be walked withos.walk@dmurphy18 @Ikramromdhani Can you review again this sequence of comments on this issue, which I believe identifies the exact cause of the issue:
The latter in particular identifies the problem as the piece of code that is responsible for dropping
fetch_requestmarker sentinels on the filesystem (see my comments embedded in it such as<== PROBLEM HERE!!).@dmurphy18 Also relevant in there is the following:
this problem only occurs if you are using an
__env__definition forgit_pillar. i.e. this breaks:but this does not
Can you confirm that you are using an
__env__definition when attempting to reproduce it?A WORKING FIX
If I alter the mentioned code to instead walk the tree at
repo_work_hash, rather than simply usinglistdirto drop sentinels at the top level, then afetch_requestis placed at all directory nodes and then those branches such assome/branchare definitely updated.@Ikramromdhani I, for example, no longer blow away the entire
cachedirectory … instead I simplyon the branch I need fixed and 60 seconds later that branch will have been updated. (Clearly I don’t like having to do this manually all the time though!)
Given this appears to be an issue with ‘/’ in the name of the field, I think changes from PR 65017 to salt/utils/gitfs.py, could be the culprit, the added
whereas previously just did the decode and replace of ‘/’ with ‘_’. Here the now presented name is not getting sanitized, and functions such as os.listdir will interpret the ‘/’ as a path separator.
This is just a thought that I wanted to record and not forget, not saying its the problem, just a probable cause
@Ikramromdhani Sorry but this is on the back-burner and I have to get back to it, given the reduced size of the team, some fireballs got thrown which had to be handled. Be assured it is on the list, and the fix is half done, but tests to be written if memory recall is correct. Apologies but only so much time in the day, with a smaller team, and a laptop rebuild didn’t help either.
@Ikramromdhani Working on fix for Git FS and the Git Pillar is right after it, fix for Git FS is done but have to refractor a new test after PR review (https://github.com/saltstack/salt/pull/65937), after that then work on Git Pillar since it will be possibly affected by GitFS changes.
@Ikramromdhani Won’t make it for Salt 3006.7, that is close to release, waiting on clean tests in the pipeline. Possibly for 3007, but on my backlog, dealing with https://github.com/saltstack/salt/issues/65816, which may be related (similar code changes for pillarenv went in at same time for gitfs locks), writing tests for that fix now. So it is on the block but limited hands since the buyout. Get to it when I can.
We are also observing this error after recently upgrading from
salt-3004tosalt-3006.4:salt-3006.6due to #65691salt/utils/gitfs.pygives no reason to believe the bug is absent in3006.6.We have observed that when we use
__env__to effectively mappillarenvto a branch name that:feature/testingthat is not already in the cache, this will be pulled correctly; andHEAD, that the changes will not be pulled;However, if we create a new branch named
testingthen it will be pulled, and subsequent changes to it will be pulled. That is, it seems that only branches with a/in them do not update. Unfortunately we use branch names likerelease/*orfeature/*, like many people do so its a severe issue for us at the moment!(This evidence conflicts with https://github.com/saltstack/salt/issues/65467#issuecomment-1921904438, although the commenter had not actually confirmed it themselves)
Note:
GitPythonrather thanpygit2gitfs… changes to our state files are pulled … it is onlygit_pillar@Ikramromdhani Can you upgrade to latest Salt 3006 and retry, Salt 3005 has passed it’s bug fix (last August 2023) and it about to run out of CVE support on the Feb 25th 2024, 24 days from now.
At this stage nothing will be getting done to Salt 3005 given that it is dead in 24 days, hardly time to debug/test/fix/release cycle, esp. since would have to build classic packages and there is a couple of days in just getting that done.