podman: Unable to connect to remote vm with podman: ssh: rejected: connect failed (open failed)

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I’m unable to connect to a linux vm from podman on a mac.

Steps to reproduce the issue:

Get a VM on OCI. I got the free tier with the following “shape” and Oracle-Linux-8.4-aarch64-2021.10.25-0:

Type	Value
Shape	VM.Standard.A1.Flex
OCPU count	4
Network bandwidth (Gbps)	4
Memory (GB)	24
Local disk	Block storage only

I specified the public ssh keys. In addition to using my existing rsa keys, I also created a ed25519 key to rule out issues with key algorithm, and added its public key to the machine. I am able to log into a machine thus provisioned:

❯ ssh -i ~/.ssh/oci/id_ed25519 opc@192.9.226.230
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Sun Nov 28 18:46:56 2021 from 52.119.114.22
[opc@instance-20211128-1024 ~]$

Install podman on the vm:

sudo yum module enable -y container-tools
sudo yum -y install podman
systemctl --user enable podman.socket
sudo loginctl enable-linger opc

Try to add this connection to local podman

❯ podman system connection add oci --identity ~/.ssh/oci/id_ed25519 ssh://opc@192.9.226.230:22/run/user/1000/podman/podman.sock
❯ podman system connection list
Name        Identity                                 URI
oci*        /Users/pratikmallya/.ssh/oci/id_ed25519  ssh://opc@192.9.226.230:22/run/user/1000/podman/podman.sock

Try to get info:

❯ podman --log-level=debug info --debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called info.PersistentPreRunE(podman --log-level=debug info --debug)
DEBU[0000] SSH Ident Key "/Users/pratikmallya/.ssh/oci/id_ed25519" SHA256:15QBOSVd+96gZ3p2cUCPFM4aRE45EksAGSgeEp8UV2s ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.NrigAZAzMU/Listeners", ssh-agent signer(s) enabled
DEBU[0000] SSH Agent Key SHA256:8IK+xJRl6YCCCr4D4js9tgxHQTJCifwdKDUuPecRSSA ssh-rsa
DEBU[0000] SSH Agent Key SHA256:bJPexGaDqkxNtFQmSJysQKIwjhHC+vJ15d8kdLOnZW0 ssh-rsa
DEBU[0000] DoRequest Method: GET URI: http://d/v3.4.2/libpod/_ping
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v3.4.2/libpod/_ping": ssh: rejected: connect failed (open failed)

Describe the results you received: podman client failed to connect to the remote linux with podman installed

Describe the results you expected: Expected podman to connect successfully

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

server:
[opc@instance-20211128-1024 ~]$ podman version
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.7
Built:        Wed Nov 10 16:08:24 2021
OS/Arch:      linux/arm64

client:
❯ brew info podman
podman: stable 3.4.2 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/usr/local/Cellar/podman/3.4.2 (170 files, 40MB) *
  Poured from bottle on 2021-11-15 at 09:13:31
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✔, go-md2man ✘
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
fish completions have been installed to:
  /usr/local/share/fish/vendor_completions.d
==> Analytics
install: 11,680 (30 days), 40,436 (90 days), 69,182 (365 days)
install-on-request: 11,682 (30 days), 40,439 (90 days), 69,130 (365 days)
build-error: 0 (30 days)

Output of podman info --debug:

❯ podman --log-level=debug info --debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called info.PersistentPreRunE(podman --log-level=debug info --debug)
DEBU[0000] SSH Ident Key "/Users/pratikmallya/.ssh/oci/id_ed25519" SHA256:15QBOSVd+96gZ3p2cUCPFM4aRE45EksAGSgeEp8UV2s ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.NrigAZAzMU/Listeners", ssh-agent signer(s) enabled
DEBU[0000] SSH Agent Key SHA256:8IK+xJRl6YCCCr4D4js9tgxHQTJCifwdKDUuPecRSSA ssh-rsa
DEBU[0000] SSH Agent Key SHA256:bJPexGaDqkxNtFQmSJysQKIwjhHC+vJ15d8kdLOnZW0 ssh-rsa
DEBU[0000] DoRequest Method: GET URI: http://d/v3.4.2/libpod/_ping
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v3.4.2/libpod/_ping": ssh: rejected: connect failed (open failed)

Package info (e.g. output of rpm -q podman or apt list podman):

[opc@instance-20211128-1024 ~]$ rpm -q podman
podman-3.3.1-9.0.1.module+el8.5.0+20416+d687fed7.aarch64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 1
Comments: 36 (13 by maintainers)

Most upvoted comments

Having same issues as described here. Tried all the solutions that solved for some but none worked. Podman on the linux server is 4.5.1 on Fedora 38 amd64. Client is MacOS installed from brew, also 4.5.1 but ARM64. SSH works, podman is running on Linux system.

> podman --log-level=debug ps
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level=debug ps) 
DEBU[0000] Found SSH_AUTH_SOCK "/Users/iru/.gnupg/S.gpg-agent.ssh", ssh-agent signer enabled 
DEBU[0000] SSH Agent Key SHA256:[REDACTED] ssh-rsa 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.5.1/libpod/_ping 
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v4.5.1/libpod/_ping": strconv.Atoi: parsing "": invalid syntax
DEBU[0000] Shutting down engines

SSHD logs on host shows the login works. Setting selinux enforcing to 0 doesn’t change the results. Where is this http request attempt is coming from?

irusensei on Jul 1, 2023

I solved the issue in my case! The problem was that tcp forwarding was not permitted in the sshd config. It is enabled by default but on this server it was somehow disabled. So I changed the AllowTcpForwarding to yes in the /etc/ssh/sshd_config There is also a AllowStreamLocalForwarding option that I’m not sure if it matters but I set to yes just in case.

LKajan on Jun 10, 2022

I have the same issue. My podman host is on RHEL 8 (podman 3.4.2, newest that is provided in the official repo). Client is on Windows 11 (podman 3.4.4).

On host machine: I have setted up a rootless configuration. Socket is started and listening. podman --remote info works. podman --remote run hello-world works.

On client machine: podman system info gives the following error:

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v3.4.4/libpod/_ping": ssh: rejected: connect failed (open failed)

Just like @pratikmallya also noticed sshd accepts the connection on a host machine.

LKajan on May 16, 2022

@pratikmallya I’m going to re-open the issue as you asked. But the problem is not upstream podman The problems is your distribution has not released a version of podman new enough for the vm. You actually state as such. One thing you could do is build podman yourself on the mac by cloning upstream 3.3.1. We very well could be in the same position as this if you were using and older version of Ubuntu or even Fedora.

baude on Mar 3, 2022

is it possible you forgot to start the podman service? i see the enable but nothing to start it ?

baude on Jan 3, 2022