salt: GPG Decryption Failed

Description of Issue/Question

My salt environment has been setup for 2 years now and I’ve been experiencing GPG decryption failed every, now and then.

Based on salt master logs:

{ "highstate-timestamp": "2018-12-16 17:03:58,595", "name": "salt.loaded.int.render.gpg", "levelname": "WARNING ", "message": "Could not decrypt cipher -----BEGIN PGP MESSAGE-----
Version: GnuPG v1

hQEMAw9iLKXotYWZAQgAqWanFkBMos5tEeH0P6dyHonlA5pkUv2xFj4oDGoaBCG2
TwbvD3/HXI6rTvvvSK757nvKiTHR+Aq6LLVYJ0+S2HYuhG5A33FWCvZ/pMQBCHhg
1hQ/oTzuHLSZAsk9dhtEskNnXJRt0iGkmAeH5qUk8ZOBHNUHsLzggQ56BosSuglA
e63QAf9PMvd0tgNGnC7ZBmchWWDh8VPHJCotQFrEb6BvDvKXvp5fSVed/7QFPQK3
5mlasbKWwPQBZwe66HZ2xs/XhF60cvz1Jh5lv/KjQtrGnkzXcr6pe9vBZjmItPw7
d8GyWy9BFSAGFY5iINkFpg8bnPAYM9oIKjHw+B/jxNJJAWHvb54Fggl8wNdA3gDj
Yd74jjvM3OC0i7JDUAacjg7hdTxLbL9JxNNKvBhCtRzcTfy9Uinm2N6W1tuEBu4N
TT9L3ZlUAAn0nA==
=+diQ
-----END PGP MESSAGE-----
, received: [GNUPG:] ENC_TO <reducted>
gpg: can't connect to the agent: IPC connect call failed
[GNUPG:] KEY_CONSIDERED <reducted>
gpg: encrypted with 2048-bit RSA key, ID <reducted>, created 2017-02-07
      "companykey"
[GNUPG:] NO_SECKEY <reducted>
[GNUPG:] BEGIN_DECRYPTION
[GNUPG:] DECRYPTION_FAILED
gpg: decryption failed: No secret key
[GNUPG:] END_DECRYPTION
" }

But on the next scheduled highstate, i don’t experience the problem.

Setup

/srv/salt/pillar/schedule.sls:

schedule:
  highstate:
    function: state.highstate
    seconds: 300
    splay: 60

/srv/salt/pillar/db.sls

#!jinja|yaml|gpg
api:
  db_username: |
    -----BEGIN PGP MESSAGE-----
    Version: GnuPG v1

    hQEMAw9iLKXotYWZAQgAqWanFkBMos5tEeH0P6dyHonlA5pkUv2xFj4oDGoaBCG2
    TwbvD3/HXI6rTvvvSK757nvKiTHR+Aq6LLVYJ0+S2HYuhG5A33FWCvZ/pMQBCHhg
    1hQ/oTzuHLSZAsk9dhtEskNnXJRt0iGkmAeH5qUk8ZOBHNUHsLzggQ56BosSuglA
    e63QAf9PMvd0tgNGnC7ZBmchWWDh8VPHJCotQFrEb6BvDvKXvp5fSVed/7QFPQK3
    5mlasbKWwPQBZwe66HZ2xs/XhF60cvz1Jh5lv/KjQtrGnkzXcr6pe9vBZjmItPw7
    d8GyWy9BFSAGFY5iINkFpg8bnPAYM9oIKjHw+B/jxNJJAWHvb54Fggl8wNdA3gDj
    Yd74jjvM3OC0i7JDUAacjg7hdTxLbL9JxNNKvBhCtRzcTfy9Uinm2N6W1tuEBu4N
    TT9L3ZlUAAn0nA==
    =+diQ
    -----END PGP MESSAGE-----

Also, I don’t generate my keys anymore. I just copy it manually from my stash to /etc/salt/gpgkeys, then just do import the public key:

/usr/bin/gpg --import /etc/salt/gpgkeys/public_key.gpg

Right now here’s the contents of my /etc/salt/gpgkeys:

$ ls /etc/salt/gpgkeys/
S.gpg-agent
S.gpg-agent.browser
S.gpg-agent.extra
S.gpg-agent.ssh
private-keys-v1.d
public_key.gpg
pubring.gpg
secring.gpg
trustdb.gpg

Steps to Reproduce Issue

I tried from salt master to list keys and it doesn’t fail:

$ sudo salt 'server-1' pillar.items 
server-1:
    ----------
    api:
        ----------
        db_username:
            db_username

I’m not exactly sure how to reproduce this because this only happened just once in a blue moon (but this has happened multiple times before).

I was reading some issues posted in Google about this same issue: http://joshbolling.com/2017/05/21/quick-tip-fix-gpg-cant-connect-to-the-agent-ipc-connect-call-failed-error/ https://blog.badgerops.net/2017/05/31/gpg-encrypting-secrets-with-saltstack/

According to those two:

• Error was being caused by a GPG agent that was already running

Am i doing something wrong here? Is it because of the directory where the keys are located? Is it because my scheduler is too fast?

Versions Report

Salt Version: Salt: 2018.3.3

Dependency Versions: cffi: 1.11.5 cherrypy: Not Installed dateutil: 2.7.5 docker-py: Not Installed gitdb: 2.0.3 gitpython: 2.1.8 ioflo: Not Installed Jinja2: 2.10 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: 1.0.7 msgpack-pure: Not Installed msgpack-python: 0.5.6 mysql-python: Not Installed pycparser: 2.19 pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: Not Installed Python: 2.7.15rc1 (default, Nov 12 2018, 14:31:15) python-gnupg: 0.4.1 PyYAML: 3.13 PyZMQ: 16.0.2 RAET: Not Installed smmap: 2.0.3 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.2.5

System Versions: dist: Ubuntu 18.04 bionic locale: UTF-8 machine: x86_64 release: 4.15.0-1021-aws system: Linux version: Ubuntu 18.04 bionic