crc: [BUG][Ubuntu][Bare metal] OpenShift cluster becomes Unreachable

General information

  • OS: Linux
  • Hypervisor: QEMU / libvirt
  • Did you run crc setup before starting it: Yes
  • Running CRC on: Baremetal-Server

CRC version

CRC version: 2.20.0+f3a947

CRC status

DEBU CRC version: 2.20.0+f3a947
DEBU OpenShift version: 4.13.0
DEBU Podman version: 4.4.4
DEBU Running 'crc status'
CRC VM:          Running
OpenShift:       Unreachable (v4.13.0)
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     26.29GB
Cache Directory: /home/ubuntu/.crc/cache

CRC config

- consent-telemetry                     : no
- cpus                                  : 12
- memory                                : 27648
- network-mode                          : user

Host Operating System

# Put the output of `cat /etc/os-release`
PRETTY_NAME="Ubuntu 22.04.2 LTS"

Steps to reproduce

  1. Follow the instructions listed in https://github.com/crc-org/crc/wiki/Debugging-guide
  2. crc ip returns 127.0.0.1
  3. ssh -i ~/.crc/machines/crc/id_ecdsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null core@127.0.0.1 returns core@127.0.0.1: Permission denied (publickey).

Expected

SSH into the VM should be possible. However, could it be due to the fact that crc ip is returning the IP address of the loopback interface, and should return a 192.168.x.x IP instead?

ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 74:56:3c:56:f1:1e brd ff:ff:ff:ff:ff:ff
    inet 162.19.88.183/24 metric 100 brd 162.19.88.255 scope global dynamic enp3s0f0
       valid_lft 62233sec preferred_lft 62233sec
    inet6 2001:41d0:306:2bb7::/56 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::7656:3cff:fe56:f11e/64 scope link
       valid_lft forever preferred_lft forever
4: enp3s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 74:56:3c:56:f1:1f brd ff:ff:ff:ff:ff:ff
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:d5:4d:5a brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

Worth noting that what prompted me to try to SSH into the VM is the fact that the OpenShift cluster is appearing as Unreachable (I’ve read through this previous issue by the way: https://github.com/crc-org/crc/issues/1888).

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

The cluster starts without visible issues, without needing to delete the existing install.

About this issue

  • Original URL
  • State: open
  • Created 10 months ago
  • Comments: 33 (20 by maintainers)

Most upvoted comments

For the record, we’re spinning up crc on another VM/instance (CentOS this time). Will report findings tomorrow (whether we’re noticing the same crashes/unavailability of the cluster).

@dabcoder As per daemon logs there is continue warning around endpoint not connected which is coming from https://github.com/google/gvisor/blob/master/pkg/tcpip/errors.go#L448 , will inspect what is happening and ask for more info.

Sep 04 13:20:53 ns3220936 crc[356229]: level=info msg="new connection from vm(3):2383898004 to vm(4294967295):1024"
Sep 04 13:21:56 ns3220936 crc[356229]: W0904 13:21:56.762835  356229 gonet.go:457] ep.GetRemoteAddress() failed: endpoint not connected
Sep 04 13:21:56 ns3220936 crc[356229]: W0904 13:21:56.853472  356229 gonet.go:457] ep.GetRemoteAddress() failed: endpoint not connected
Sep 04 13:21:57 ns3220936 crc[356229]: W0904 13:21:57.309752  356229 gonet.go:457] ep.GetRemoteAddress() failed: endpoint not connected
Sep 04 13:21:57 ns3220936 crc[356229]: W0904 13:21:57.346149  356229 gonet.go:457] ep.GetRemoteAddress() failed: endpoint not connected
Sep 04 13:21:57 ns3220936 crc[356229]: W0904 13:21:57.804982  356229 gonet.go:457] ep.GetRemoteAddress() failed: endpoint not connected
Sep 04 13:21:57 ns3220936 crc[356229]: W0904 13:21:57.831189  356229 gonet.go:457] ep.GetRemoteAddress() failed: endpoint not connected
Sep 04 13:21:57 ns3220936 crc[356229]: W0904 13:21:57.850099  356229 gonet.go:457] ep.GetRemoteAddress() failed: endpoint not connected
Sep 04 13:21:57 ns3220936 crc[356229]: W0904 13:21:57.921616  356229 gonet.go:457] ep.GetRemoteAddress() failed: endpoint not connected
Sep 04 13:21:58 ns3220936 crc[356229]: W0904 13:21:58.009339  356229 gonet.go:457] ep.GetRemoteAddress() failed: endpoint not connected