salt: [salt-ssh, 3001.2, 3001.3, 3002.1] ssh password authentication seems to be broken

Description I can no longer use salt-ssh to execute states after upgrading to salt-ssh 3001.3 or 3002.1, previous versions do work (3001.2 and 3002). Trying to execute test.version on one of the hosts in the roster file I get:

$ salt-ssh -i ‘sta-dca-payments-us’ test.version -l debug

salt@6d7841e67085:~$ salt-ssh -i 'sta-dca-payments-us' test.version -l debug
[INFO    ] Loading Saltfile from '/home/salt/Saltfile'
[DEBUG   ] Reading configuration from /home/salt/Saltfile
[DEBUG   ] Reading configuration from /home/salt/config/master
[DEBUG   ] Configuration file path: /home/salt/config/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] LazyLoaded flat.targets
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] compile template: ./config/roster
[DEBUG   ] LazyLoaded gpg.render
[DEBUG   ] Results of YAML rendering: 
OrderedDict([('sta-dca-payments-us', OrderedDict([('host', 'sta-dca-payments-us.company.com'), ('passwd', '-----BEGIN PGP MESSAGE-----EDITED -----END PGP MESSAGE-----\n')])), ('sta-dca-payments-db', OrderedDict([('host', 'sta-dca-payments-db.company.com'), ('passwd', '-----BEGIN PGP MESSAGE-----EDITED-----END PGP MESSAGE-----\n')])), ('sta-dca-payments-db-replica', OrderedDict([('host', 'sta-dca-payments-db-replica.company.com'), ('passwd', '-----BEGIN PGP MESSAGE-----EDITED-----END PGP MESSAGE-----\n')])), ('sta-dca-payments-logs', OrderedDict([('host', 'sta-dca-payments-logs.company.com'), ('passwd', '-----BEGIN PGP MESSAGE-----EDITED-----END PGP MESSAGE-----\n')]))])
[PROFILE ] Time (in seconds) to render './config/roster' using 'yaml' renderer: 0.005864858627319336
[DEBUG   ] Reading GPG keys from: /secret-storage
[PROFILE ] Time (in seconds) to render './config/roster' using 'gpg' renderer: 0.12304568290710449
[DEBUG   ] LazyLoaded roster_matcher.targets
[DEBUG   ] Matched minions: {'sta-dca-payments-us': {'user': 'deployment-user', 'sudo': True, 'tty': True, 'host': 'sta-dca-payments-us.company.com', 'passwd': "EDITING THIS PASSWORD BUT IT APPEARS TO BE CORRECT"}}
Saving key "/home/salt/pki_dir/ssh/salt-ssh.rsa" failed: passphrase is too short (minimum five characters)
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[DEBUG   ] LazyLoaded local_cache.prep_jid
[DEBUG   ] Adding minions for job 20201111134559415761: ['sta-dca-payments-us']
[DEBUG   ] Could not LazyLoad test.version: 'test.version' is not available.
[DEBUG   ] Performing shimmed, blocking command as follows:
test.version
[DEBUG   ] Executing command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user  mkdir -p 
[DEBUG   ] Child Forked! PID: 130  STDOUT_FD: 12  STDERR_FD: 14
[DEBUG   ] Terminal Command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user mkdir -p
[DEBUG   ] Executing command: scp -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user  /tmp/shim_07zynizf sta-dca-payments-us.company.com:.5849daacfc5a.py
[DEBUG   ] Child Forked! PID: 131  STDOUT_FD: 12  STDERR_FD: 14
[DEBUG   ] Terminal Command: scp -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user /tmp/shim_07zynizf sta-dca-payments-us.company.com:.5849daacfc5a.py
[DEBUG   ] Executing command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user  /bin/sh '$HOME/.5849daacfc5a.py'
[DEBUG   ] Child Forked! PID: 133  STDOUT_FD: 12  STDERR_FD: 14
[DEBUG   ] Terminal Command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user /bin/sh  '$HOME/.5849daacfc5a.py'
[DEBUG   ] Executing command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user  rm '$HOME/.5849daacfc5a.py'
[DEBUG   ] Child Forked! PID: 134  STDOUT_FD: 12  STDERR_FD: 14
[DEBUG   ] Terminal Command: ssh sta-dca-payments-us.company.com -t -t -o KbdInteractiveAuthentication=no -o PasswordAuthentication=yes -o GSSAPIAuthentication=no -o ConnectTimeout=65 -o StrictHostKeyChecking=no -o Port=22 -o IdentityFile=/home/salt/pki_dir/ssh/salt-ssh.rsa -o User=deployment-user rm $HOME/.5849daacfc5a.py
[DEBUG   ] RETCODE sta-dca-payments-us.company.com: 127
[DEBUG   ] SHIM retcode(127) and command: Password:
[DEBUG   ] LazyLoaded nested.output
sta-dca-payments-us:
    ----------
    retcode:
        127
    stderr:
        no such identity: /home/salt/pki_dir/ssh/salt-ssh.rsa: No such file or directory
        Connection to sta-dca-payments-us.company.com closed.
    stdout:
        Password: 
        /bin/sh: 0: Can't open $HOME/.5849daacfc5a.py

Setup /home/salt/Saltfile

salt-ssh:
  roster_file: ./config/roster
  config_dir: ./config
  ssh_max_procs: 30
  ssh_wipe: True
  ssh_log_file: /home/salt/salt-ssh.log
  log_file: /home/salt/salt.log

/home/salt/config/master:

cachedir: /home/salt/cachedir
yaml_utf8: True
pki_dir: /home/salt/pki_dir
pillar_opts: True
roster_defaults:
  user: deployment-user
  sudo: True
  tty: True
file_roots:
  base:
    - /home/salt/saltstack/salt
pillar_roots:
  sta:
    - /home/salt/saltstack/pillar/sta
gpg_keydir: /secret-storage

/home/salt/config/roster:

#!yaml|gpg
# -*- coding: utf-8 -*-
# vim: ft=yaml
---
sta-dca-payments-us:
  host: sta-dca-payments-us.company.com
  passwd: |
    -----BEGIN PGP MESSAGE-----
    
    -----END PGP MESSAGE-----

(This file includes more hosts with this same structure)

Steps to Reproduce the behavior See the initial description, it happens with any state/module.

Expected behavior I expect salt-ssh to execute state/modules like it used to.

Versions Report

Salt Version:
           Salt: 3001.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.2
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 1.0.0
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.9 (default, Oct  8 2020, 12:12:24)
   python-gnupg: Not Installed
         PyYAML: 5.3.1
          PyZMQ: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: Not Installed
 
System Versions:
           dist: ubuntu 18.04 bionic
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 5.6.15-050615-lowlatency
         system: Linux
        version: Ubuntu 18.04 bionic

Additional context salt-ssh is executed inside a docker container, our image is based on ubuntu bionic with salt-ssh installed with pip basically is something like the following:

FROM ubuntu:bionic

ENV DEBIAN_FRONTEND=noninteractive
ENV BUILD_DEPENDENCIES python3-pip \
	dpkg-dev \
        g++ \
        gcc \
        libc6-dev \
        make \
        python3-dev \
        python3-setuptools \
        python3-wheel

ENV GPG_PACKAGES gnupg gpg-agent

ENV USEFUL_TOOLS vim less

RUN apt-get update && \
apt-get install --no-install-recommends -y \
        $GPG_PACKAGES \
	$USEFUL_TOOLS \
        locales \
        openssh-client \
	sshpass \
	$BUILD_DEPENDENCIES && \
LANG="C.UTF-8" pip3 install salt-ssh==3001.3 && \
apt-get remove --purge -y $BUILD_DEPENDENCIES && \
apt-get clean && rm -rf /var/lib/apt/lists/*

ENV LANG en_US.UTF-8
ENV LANGUAGE ""
ENV LC_CTYPE "en_US.UTF-8"
ENV LC_NUMERIC en_US.UTF-8
ENV LC_TIME en_US.UTF-8
ENV LC_COLLATE "en_US.UTF-8"
ENV LC_MONETARY en_US.UTF-8
ENV LC_MESSAGES "en_US.UTF-8"
ENV LC_PAPER en_US.UTF-8
ENV LC_NAME en_US.UTF-8
ENV LC_ADDRESS en_US.UTF-8
ENV LC_TELEPHONE en_US.UTF-8
ENV LC_MEASUREMENT en_US.UTF-8
ENV LC_IDENTIFICATION en_US.UTF-8
ENV LC_ALL ""
RUN useradd -m salt -s /bin/bash
COPY Saltfile /home/salt/Saltfile
RUN chown salt:salt /home/salt/Saltfile
USER salt:salt
RUN mkdir -p /home/salt/config \
             /home/salt/cachedir \
             /home/salt/pki_dir \
             /home/salt/.bin \
             /home/salt/saltstack \
             /home/salt/saltstack/salt \
             /home/salt/saltstack/pillar
COPY launch-salt.sh /home/salt/.bin/launch-salt.sh
COPY master /home/salt/config/master
COPY roots_config /roots_config
WORKDIR /home/salt

ENTRYPOINT ["/home/salt/.bin/launch-salt.sh"]

Our launch-salt.sh setups gpg based on some based bind mounts and setups the environment for salt-ssh to work, but this has nothing to do with the issue, since installing a previous version fixes it. I am willing to provide more details if needed.