salt: Unable to access GitHub since algorithm changes: failed to start SSH session: Unable to exchange encryption keys

You do not have to make it id_rsa, you can add this to your /root/.ssh/config to use GitPython

Host github.com
  IdentityFile /path/to/key

And it will also work, git python just uses the git binary directly, so it will use the regular ssh on the system, which will use the .ssh/config file.

If you need multiple deployment keys, you can setup deployment aliases in .ssh/config

Host repoA
  HostName github.com
  IdentityFile /path/to/keyA
Host repoB
  HostName github.com
  IdentityFile /path/to/keyB

And then set them up as

gitfs_remotes:
  - ssh://git@repoA/org/repo-a.git
  - ssh://git@repoB/org/repo-b.git

I was unable to get the code from @deuscapturus to work (no .so files were generated) - but I’ve managed to replace all our PyGit2 configuration with GitPython.

gitfs_remotes:
  - ssh://git@github.com/org/repo-a.git
  - ssh://git@github.com/org/repo-b.git
  - ssh://git@github.com/org/repo-c.git:
    - root: salt

git_pillar_provider: GitPython

ext_pillar:
   - git:
     - master ssh://git@github.com/mention-me/Salt.git:
       - root: pillar

… and that seems to work.

I had to explicitly make the key we use to access repo-c (the private one) the id_rsa/id_rsa.pub and remove git_pillar_pubkey and git_pillar_privkey to avoid errors on start.

We compiled from current master of libssh2. This resolved the issue.

This is super hacky! DO AT YOUR OWN RISK

wget https://github.com/libssh2/libssh2/archive/master.zip
unzip master.zip
cd libssh2-master/
mkdir bin
cd bin
cmake ..
cmake -DBUILD_SHARED_LIBS=ON --build .
sudo mv /usr/lib64/libssh2.so.1.0.1 /usr/lib64/libssh2.so.1.0.1.bak
sudo cp src/libssh2.so.1.0.1 /usr/lib64/libssh2.so.1.0.1

If you are using RedHat, you should have a contract with them, certainly you could open a ticket in their bugzilla and ask.

I’m tearing my hair out trying to get pygit2 working with CentOS and ssh authentication against github. It was working fine with the standard yum package python-pygit2 until the recent github changes. I’m now trying to recompile everything with libssh2 1.8.0 but haven’t been able to find something that works yet. The salt master seems unable to detect that pygit2 is available. “No suitable gitfs provider module is installed” is logged, even though fileserver.update seems to work.

salt-master --versions-report:

Salt Version:
           Salt: 2017.7.4

Dependency Versions:
           cffi: 1.6.0
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.3
         Python: 2.7.5 (default, Aug  4 2017, 00:39:18)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.4.1708 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-693.11.6.el7.x86_64
         system: Linux
        version: CentOS Linux 7.4.1708 Core

Here’s how I installed pygit2:

# epel
sudo yum install epel-release -y

# git
sudo yum install git -y

# gcc
sudo yum install gcc -y

# openssl-devel
sudo yum install openssl-devel -y

# libssh2
curl -L -O https://libssh2.org/download/libssh2-1.8.0.tar.gz
tar -xvf libssh2-1.8.0.tar.gz
rm libssh2-1.8.0.tar.gz
cd libssh2-1.8.0
./configure
make
sudo make install
cd ..
rm -rf libssh2-1.8.0

# cmake
sudo yum install cmake -y

# libgit2
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
curl -L -O https://github.com/libgit2/libgit2/archive/v0.26.0.tar.gz
tar -xvf v0.26.0.tar.gz
rm v0.26.0.tar.gz
cd libgit2-0.26.0
cmake .
make
sudo make install
cd ..
rm -rf libgit2-0.26.0
echo "LD_LIBRARY_PATH=/usr/local/lib" | sudo tee --append /etc/environment

# pip
sudo yum install python-pip -y
sudo pip install --upgrade pip

# python-devel
sudo yum install python-devel -y

# libffi
sudo yum install python-cffi -y

# pygit2
sudo pip install pygit2

When I manually run fileserver.update it seems to work.

sudo salt-run fileserver.update
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] LazyLoaded fileserver.update
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt1
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Loading static grains from /etc/salt/grains
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] pygit2 gitfs_provider enabled
[DEBUG   ] LazyLoaded git.envs
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Updating git fileserver cache
[DEBUG   ] Set update lock for gitfs remote 'ssh://git@github.com/***.git'
[DEBUG   ] Fetching gitfs remote 'ssh://git@github.com/***.git'
[DEBUG   ] gitfs remote 'ssh://git@github.com/***' is up-to-date
[DEBUG   ] pygit2 does not support detecting stale refs for authenticated remotes, saltenvs will not reflect branches/tags removed from remote 'ssh://git@github.com/***.git'
[DEBUG   ] Removed update lock for gitfs remote 'ssh://git@github.com/***.git'
[DEBUG   ] Updating roots fileserver cache
[DEBUG   ] LazyLoaded local_cache.prep_jid
[DEBUG   ] LazyLoaded nested.output
True
[INFO    ] Runner completed: 20180227125334567456
[DEBUG   ] Runner return: True

But in the salt master logs after restarting the service:

2018-02-27 13:09:39,332 [salt.utils.verify:525 ][WARNING ][13820] Insecure logging configuration detected! Sensitive data may be logged.
2018-02-27 13:09:39,332 [salt.cli.daemons :82  ][INFO    ][13820] Setting up the Salt Master
2018-02-27 13:09:39,417 [salt.crypt       :315 ][DEBUG   ][13820] Loaded master key: /etc/salt/pki/master/master.pem
2018-02-27 13:09:39,417 [salt.daemons.masterapi:266 ][INFO    ][13820] Preparing the root key for local communication
2018-02-27 13:09:39,417 [salt.daemons.masterapi:227 ][DEBUG   ][13820] Removing stale keyfile: /var/cache/salt/master/.root_key
2018-02-27 13:09:39,418 [salt.daemons.masterapi:26  ][PROFILE ][13820] Beginning pwd.getpwall() call in masterapi access_keys function
2018-02-27 13:09:39,418 [salt.daemons.masterapi:26  ][PROFILE ][13820] End pwd.getpwall() call in masterapi access_keys function
2018-02-27 13:09:39,419 [salt.utils.process:96  ][DEBUG   ][13820] Created pidfile: /var/run/salt-master.pid
2018-02-27 13:09:39,420 [salt.cli.daemons :82  ][INFO    ][13820] Starting up the Salt Master
2018-02-27 13:09:39,425 [salt.utils.gitfs :2251][CRITICAL][13820] No suitable gitfs provider module is installed.
2018-02-27 13:09:39,434 [salt.utils.lazy  :100 ][DEBUG   ][13820] Could not LazyLoad git.envs: 'git.envs' is not available.
2018-02-27 13:09:39,434 [salt.utils.lazy  :100 ][DEBUG   ][13820] Could not LazyLoad roots.init: 'roots.init' is not available.
2018-02-27 13:09:39,435 [salt.utils.gitfs :2251][CRITICAL][13820] No suitable git_pillar provider module is installed.
2018-02-27 13:09:39,436 [salt.master      :517 ][CRITICAL][13820] Failed to load git_pillar
2018-02-27 13:09:39,436 [salt.master      :518 ][CRITICAL][13820] Master failed pre flight checks, exiting

Edit: I’ve finally gotten it to work with a variation of duescapturus’ suggestion. I had the same problem as edhgoose (no .so files were generated) but this worked:

sudo yum install gcc -y
sudo yum install openssl-devel -y
curl -L -O https://libssh2.org/download/libssh2-1.8.0.tar.gz
tar -xvf libssh2-1.8.0.tar.gz
cd libssh2-1.8.0
./configure
make
sudo cp /usr/lib64/libssh2.so.1.0.1 /usr/lib64/libssh2.so.1.0.1.bak
sudo cp src/.libs/libssh2.so.1.0.1 /usr/lib64/libssh2.so.1.0.1
sudo ln -s -f /usr/lib64/libssh2.so.1.0.1 /usr/lib64/libssh2.so.1
sudo yum install python-pygit2 -y

After running the above the salt master was able to get files from github. This is still very hacky though obviously.

Ubuntu 17.10 also has the updated version of libssh2 in its repos. So if you can upgrade your salt-master to that, it’s another path to get your gitfs/git pillars working.

I built a new Salt master today and ran into this issue again due to the latest release of pygit2 v0.26.4. I’m not sure if it’s compatible with Salt 2018.3.2. I ended up getting around this problem by installing pygit2 v0.26.3 which was harder than you would think because 0.26.4 is the only version available via yum.

# Remove current version of pygit2
sudo yum remove python-pygit2 -y

# Install tools required for building libgit2
sudo yum install gcc -y
sudo yum install openssl-devel -y
sudo yum install cmake -y
sudo yum install python-devel -y

# Compile and install libgit2 v0.26.3
curl -L -O https://github.com/libgit2/libgit2/archive/v0.26.3.tar.gz
tar xzf v0.26.3.tar.gz
cd libgit2-0.26.3/
cmake .
make
sudo make install
cd ..
rm -rf libgit2-0.26.3
rm -f v0.26.3.tar.gz

# Install pip so that we can install pygit2
sudo yum install python-pip -y

# Install an older version of pygit2
sudo pip install pygit2==0.26.3

Note: This fix needs to be applied in addition to the one in my earlier comment.

Also that token should be fixed, so you should be able to sign up now for the community slack