podman: Unable to authenticate with podman-remote over ssh to drive remote podman.sock

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Trying to run Podman on a remote machine and use the podman-remote client to drive it.

Following instructions here: https://github.com/containers/podman/blob/master/docs/tutorials/mac_win_client.md

Podman on the remote machine seems to be working fine, but it cannot be driven by the local podman because the local Podman fails to authenticate properly over SSH.

Steps to reproduce the issue:

1. Setup podman on remote machine per: https://github.com/containers/podman/blob/master/docs/tutorials/mac_win_client.md
2. Ensure correct SSH key is in ssh agent
3. Add remote connection to podman: podman system connection add test ssh://192.168.122.1/run/user/1000/podman/podman.sock
4. Attempt to drive remote podman from local machine: podman-remote ps

Describe the results you received:

Authentication error:

Error: Failed to create sshClient: Connection to bastion host (ssh://ben@192.168.2.186:22/run/user/1000/podman/podman.sock) failed.: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Describe the results you expected:

I expected podman-remote ps to behave normally, outputting something like this:

[ben@benssystem76 ~]$ podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

Additional information you deem important (e.g. issue happens only occasionally):

I also tried adding the connection with an explicit identity file instead of relying on the SSH agent:

podman system connection add test2 --identity /home/ben/.ssh/id_rsa ssh://192.168.122.1/run/user/1000/podman/podman.sock

When running podman-remote ps with that connection, I am prompted for the passphrase for the SSH key (as I would expect) but I get the same error message indicating that authentication failed, I think because podman didn’t do the SSH handshake properly or something:

Error: Failed to create sshClient: Connection to bastion host (ssh://ben@192.168.2.186:22/run/user/1000/podman/podman.sock) failed.: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

I also checked to make sure that /run/user/1000/podman/podman.sock existed on the remote machine, and it did. Remote user id is 1000 as expected.

Output of podman version:

[ben@benssystem76 config-files]$ podman version
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.14.9
Built:        Wed Sep 30 13:31:11 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 81d18b6c3ffc266abdef7ca94c1450e669a6a388'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: journald
  hostname: benssystem76
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.18-200.fc32.x86_64
  linkmode: dynamic
  memFree: 1297784832
  memTotal: 33637113856
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 16869486592
  swapTotal: 16869486592
  uptime: 30h 0m 31.03s (Approximately 1.25 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/ben/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/ben/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 601
  runRoot: /run/user/1000/containers
  volumePath: /home/ben/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601494271
  BuiltTime: Wed Sep 30 13:31:11 2020
  GitCommit: ""
  GoVersion: go1.14.9
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.1-7.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Remote podman is the same latest version (2.1.1), running on an F33 Server that is in a KVM VM on a Dell R620 host. Local podman version 2.1.1 is on an F32 Workstation with Gnome 3, and a nice photo of my family as the wallpaper background and screensaver.