podman: podman generated remote connections may fail from possible mismatch between "localhost" and 127.0.0.1 listener

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman cannot create connect to the machine

Steps to reproduce the issue:

  1. brew install podman

  2. podman machine init

~ podman machine init
Extracting compressed file
Image resized.
Machine init complete
To start your machine run:

	podman machine start
  1. podman machine init
~ podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Users/tomanik:/Users/tomanik

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

	podman machine set --rootful

API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.

Machine "podman-machine-default" started successfully

  1. podman machine info
~ podman machine info
Host:
  Arch: arm64
  CurrentMachine: podman-machine-default
  DefaultMachine: podman-machine-default
  EventsDir: /var/folders/l3/0fn2jt1d62b7s3kmv4q7m0yh0000gv/T/podman-run--1/podman
  MachineConfigDir: /Users/tomanik/.config/containers/podman/machine/qemu
  MachineImageDir: /Users/tomanik/.local/share/containers/podman/machine/qemu
  MachineState: Running
  NumberOfMachines: 1
  OS: darwin
  VMType: qemu
Version:
  APIVersion: 4.3.0
  Built: 1666115757
  BuiltTime: Wed Oct 19 02:55:57 2022
  GitCommit: ""
  GoVersion: go1.18.7
  Os: darwin
  OsArch: darwin/arm64
  Version: 4.3.0

Describe the results you received:

~ podman --log-level DEBUG run quay.io/podman/hello
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level DEBUG run quay.io/podman/hello)
DEBU[0000] SSH Ident Key "/Users/tomanik/.ssh/podman-machine-default" SHA256:u......uSJjM ssh-ed25519
Error: failed to connect: dial tcp [::1]:51112: connect: connection refused

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

There is almost identical issue with previous version: https://github.com/containers/podman/issues/12728

But is still does not work.

I did try recreate the machine and unset SSH_AUTH_SOCK - it did not work.

Ironically, I was able to run docker-compose up -d, only with the issue that it did not accepted environment variables.

Output of podman version:

➜  ~ podman version
Error: failed to connect: dial tcp [::1]:51112: connect: connection refused

Output of podman info:

➜  ~ podman info
Error: failed to connect: dial tcp [::1]:51112: connect: connection refused

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

➜  ~ brew info podman
==> podman: stable 4.3.0 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/4.3.0 (185 files, 47.6MB) *
  Poured from bottle on 2022-11-10 at 15:44:45
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0 and GPL-3.0-or-later
==> Dependencies
Build: go-md2man ✘, go@1.18 ✘
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions

To restart podman after an upgrade:
  brew services restart podman
Or, if you don't want/need a background service you can just run:
  /opt/homebrew/opt/podman/bin/podman system service --time=0
==> Analytics
install: 28,592 (30 days), 75,743 (90 days), 226,171 (365 days)
install-on-request: 27,263 (30 days), 72,937 (90 days), 222,845 (365 days)
build-error: 23 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

MacBook M1 macOS 12.6.1

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 37 (12 by maintainers)

Commits related to this issue

Most upvoted comments

Had the same problem, I had to create a new machine and then

podman system connection default my-machine

Solved the problem.

On Windows I reproduce the same error. From time to time I have to do the following :

podman machine stop podman machine rm podman machine init

otherwise podman won’t start again.

I have to repeat this steps once every 2-3 days …

@uaru Can you manually change the ssh host url in ~/.config/containers/containers.conf from localhost to 127.0.0.1 to see if this changes anything.

For me, changing localhost to 127.0.0.1 did not help unfortunately. I do not have any gvproxy process running. Mac OS Ventura 13.0.1 running on Apple M1 Pro chip.

% podman version
Error: failed to connect: dial tcp 127.0.0.1:50284: connect: connection refused
% brew info podman
==> podman: stable 4.3.1 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/4.3.1 (185 files, 47.6MB) *
  Poured from bottle on 2022-11-16 at 15:13:30
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0 and GPL-3.0-or-later
==> Dependencies
Build: go-md2man ✘, go@1.18 ✘
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions

To restart podman after an upgrade:
  brew services restart podman
Or, if you don't want/need a background service you can just run:
  /opt/homebrew/opt/podman/bin/podman system service --time=0
==> Analytics
install: 29,067 (30 days), 79,649 (90 days), 241,349 (365 days)
install-on-request: 26,537 (30 days), 75,187 (90 days), 235,714 (365 days)
build-error: 15 (30 days)

Update installing the mac helper resolved the issue sudo /opt/homebrew/Cellar/podman/4.3.1/bin/podman-mac-helper install

This does not resolve OPs issue, but may resolve others.

Folks, remember to follow the machine start prompts, if applicable:

This is Macbook M1 Max Darwin my-mac 22.2.0 Darwin Kernel Version 22.2.0: Fri Nov 11 02:03:51 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T6000 arm64

The system helper service is not installed; the default Docker API socket
address can't be used by podman. If you would like to install it run the
following commands:

	sudo /opt/homebrew/Cellar/podman/4.3.1/bin/podman-mac-helper install
	podman machine stop; podman machine start

You can still connect Docker API clients by setting DOCKER_HOST using the
following command in your terminal session:

	export DOCKER_HOST='unix:///Users/user1/.local/share/containers/podman/machine/podman-machine-default/podman.sock'

Once I added the system helper service I have had zero issues with pulling/running containers.

On Windows I reproduce the same error. From time to time I have to do the following :

podman machine stop podman machine rm podman machine init

otherwise podman won’t start again.

I have to repeat this steps once every 2-3 days …

And there is no ./.config/containers/containers.conf in my user folder

I have this exact same issue as well.

And there is no ./.config/containers/containers.conf in my user folder

First of all everyone keeps commenting with different kind of issues. The symptom may be similar but there are definitely different kind of problems reported here. Pleas look at the error messages carefully.

Based on my https://github.com/containers/podman/issues/16470#issuecomment-1313558006 this seems to be one problem which is easily fixed. The error message for this is dial tcp [::1]:<random port>: connect: connection refused Based on the comments this works and is what the original issue is about, I will try to fix it later.


And to everyone please stop hijacking existing issues with your problems, unless the error is the same or you are actually sure this is the same issue please just create a new separate issue. I triage almost all issues here. It is much simpler for me to close them as duplicates when they are actually are the same issue and link them the the correct ones. This allows users and developers to properly debug issues and not getting side tracked all the time by other unrelated comments. Thank you for your understanding.

@jeremycaine just a small tip, you can also do -c after podman if you switch frequently and don’t want to change the default (e.g. podman -c crc run -it ubi8)

  1. The port may be used. You need to kill the process that occupies the port

  2. Ports may be reserved;

View reserved ports:

netsh interface ipv4 show excludedportrange protocol=tcp

Add excluded reserved port:

net stop winnat

netsh int ipv4 add excludedportrange  protocol=tcp startport=5402 numberofports=1

net start winnat