salt: [BUG] onedir blocks installation of shared libraries for pygit2 gitfs in 3006.0
Description I did not change any configuration between 3005.1 and 3006.0 but suddently salt-master cannot find my public keys.
Setup
fileserver_backend:
- gitfs
gitfs_provider: pygit2
gitfs_remotes:
- ssh://git@github.com/myorg/mysalt.git:
- base: main
- root: salt
- pubkey: /root/.ssh/id_salt.pub
- privkey: /root/.ssh/id_salt
- https://github.com/saltstack/salt-winrepo-ng.git:
- all_saltenvs: master
- mountpoint: salt://win/repo-ng/salt-winrepo-ng
ext_pillar:
- git:
- main ssh://git@github.com/myorg/mypillar.git:
- env: base
- root: pillar
- pubkey: /root/.ssh/id_pillar.pub
- privkey: /root/.ssh/id_pillar
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
Upgrade from 3005.1 to 3006.0. Run salt-pip install pygit2.
Expected behavior Should fetch gitfs normally.
Screenshots
[CRITICAL] SSH pubkey (/root/.ssh/id_salt.pub) for gitfs remote 'ssh://git@github.com/myorg/mysalt.git' could not be found, path may be incorrect. Note that it may be necessary to clear git_pillar locks to proceed once this is resolved and the master has been started back up. A warning will be logged if this is the case, with instructions.
[CRITICAL] SSH pubkey (/root/.ssh/id_pillar.pub) for git_pillar remote 'main ssh://git@github.com/myorg/mypillar.git' could not be found, path may be incorrect. Note that it may be necessary to clear git_pillar locks to proceed once this is resolved and the master has been started back up. A warning will be logged if this is the case, with instructions.
[CRITICAL] Failed to load gitfs
[CRITICAL] Failed to load git_pillar
[CRITICAL] Master failed pre flight checks, exiting
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3006.0
Python Version:
Python: 3.10.11 (main, Apr 14 2023, 05:57:16) [GCC 11.2.0]
Dependency Versions:
cffi: 1.15.1
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: 1.6.3
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: 1.12.0
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 23.2.0
relenv: 0.11.2
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: debian 11 bullseye
locale: utf-8
machine: x86_64
release: 5.10.0-21-cloud-amd64
system: Linux
version: Debian GNU/Linux 11 bullseye
Additional context
Running cat and ls on those files show they exist fine. The files have permission 0400/-r-------- and are owned by root:root.
About this issue
- Original URL
- State: open
- Created a year ago
- Reactions: 2
- Comments: 50 (41 by maintainers)
Thanks to @MartinEmrich i’ve got it working on AlmaLinux 8.8 and salt-master 3006.3. In my case the steps to take where:
salt-pip install pygit2mkdir /opt/saltstack/salt/.sshand place theknown_hostsfile inside this directory.chown -Rv salt:salt /opt/saltstack/salt/.sshsystemctl edit salt-masterand enter:systemctl restart salt-masterLooks like this is an issue with how the Salt Master process is being run. When running on Ubuntu 20.04 and some of the other OSes systemd is launching the Salt master without a HOME environment variable defined. This results in the
error loading known_hosts:message. It also happens if the HOME variable is set to an empty string. It seems if home is any existing directory the error will not happen. I tracked it down to this commit:https://github.com/libgit2/libgit2/commit/e33d7c06887582540bee013665f88d9625eca46c
That explains why we see it in
pygit2==1.12.1and it is because of the libgit2 upgradelibgit2.so.1.5.0 to 1.6.3.A temporary work around could be to modify the Salt Master’s service file to include a user.
Salt has a method that set’s the HOME environment variable,
salt/utils/verify.py::check_user. However, there are two things preventing that from working. First, we don’t set anything if HOME isn’t already in the environment. Second, the method is getting called to late the environment variable to matter. I think we need to set it before importing pygit2 unless we can find another way configuring pygit2 with a valid directory.Just reinstalled everything on master from scratch and did this:
Then installed latest pygit2:
Started salt-master service and got this:
Stopped salt-master, installed pygit2==1.7.0:
Started salt-master again, got remote repositories cloned in /var/cache/salt/master/gitfs, no errors in log.
(This is on Debian 11 “bullseye” amd64.)
Had to do this again today with a more recent update modified
salt-master.service. Original method did not help but this time:User=rootEnvironment=HOME=/rootdid the trick.
Seems I managed to make my 3006.5 (onedir package from repo.saltstack.io on Debian11) to work by:
salt-pip install --only-binary :all: --no-deps "pygit2==1.7.0"Maybe it’s just me, but official docs seems a bit lagging about the onedir package, recommending to install via OS tools (one for all: pip instead of salt-pip)…
After upgrading on the good ol’ CentOS 7 from 3005 to 3006.3, I also ran into this issue.
salt-pip install --only-binary :all: --no-deps "pygit2==1.7.0"gave me a working pygit2.Here and there, it is mentioned that Salt-Master now runs as non-root, but this was not the case here; it still runs as root just as before.
But the HOME environment variable was indeed missing. Adding this to my
/usr/lib/systemd/system/salt-master.servicefixed it for me:ugh this feels like such a hack to have to do for something that is a basic feature of salt-master (ability to use git filesystem).
In general, no. That’s a big burden that many people really want to avoid (to the extent that they won’t upgrade to onedir because of it). Especially for
pygit2because it usually also means you have to rebuildlibgit2from source as well.Your build probably won’t work. See #58898.
@DaAwesomeP @OrangeDog @dwoz observing the same issue, looking for a workaround.
UPDATE: it seems to work for me with
pygit2==1.7.0:Latest version (1.12.0) throws exactly the same error as @DaAwesomeP described in https://github.com/saltstack/salt/issues/64121#issuecomment-1520141013.
_pygit2.GitError: error loading known_hosts:seems to go away and my gitfs definitions started working on Archlinux after addingWorkingDirectory=~in the[Service]section of/usr/lib/systemd/system/salt-master.service.I don’t think Arch is using the onedir solution nor the salt user account yet. It seems to be running as root and has pieces of itself all over the filesystem on the master.
It sounds like additional bundled shared libraries in wheels may have some issue with onedir, but there must be something particular since pygit2 v1.7.0 works fine but pygit2 v1.12.0 does not. I’m sure this issue will come up again at some point with this package or another. Locking into an older version (either v1.6.1 or v1.7.0 via wheel or source) seems more like a workaround.
I have had a lot of trouble with Python packages built from source across different systems, particularly with Salstack. Using the wheels/bundled libraries with pygit2 solves a lot of portability issues across many systems where the libgit2 could be really any version and the Python version included in onedir will increment as time goes on. More and more Python packages with bindings are bundling their dependencies so that they are consistent across platforms and independent from OS source versions. Having to install dependencies and compile the modules on each machine seems counter to onedir making salt more portable/consistent across installations.
Building from source on Debian 11 uses a now quite old libgit2 that does not support newer SSH keys. The workaround for this is to use the pygit2 wheel (which became always the case with onedir prior to 3006). See #61790.
Also, pygit2 v1.6.1 does not officially support Python 3.10: https://www.pygit2.org/install.html#version-numbers (so again the ability to install the latest version via wheel is much more future-proofed).
I have observed cffi version mismatches when installing additional packages alongside Salt for a while now. So far no issues (I have tried both installing the later version and keeping the older version), but it would be great if Salt updated that dependency. There are other additional packages too that rely on different cffi versions.
It’s not looking for
libssh2.so.1.0.1, it’s looking forlibssh2-3ed487a7.so.1.0.1. It has unpacked that from the.whlinto/tmp, but the onedir environment appears to be blocking it from loading during installation.Running
sudo -u salt ssh github.comand accepting the host does not appear to resolve the issue. I also triedsudo -u salt bash -c "ssh-keyscan github.com >> ~/.ssh/known_hosts"andsalt [MASTER_ID] ssh.set_known_host user=salt hostname=github.com.I verified that the
/home/salt/.ssh/known_hostsfile existed and contained some contents after running these commands.The Salt master does not not run as root anymore, see the release notes:
https://docs.saltproject.io/en/latest/topics/releases/3006.0.html#linux-packaging-salt-master-salt-user-and-group