minikube: [minikube 1.26.x/1.27.x/1.28.x] Fails upon startup with podman using rootless

What Happened?

Using

{
    "container-runtime": "cri-o",
    "driver": "podman",
    "memory": "8192",
    "rootless": true
}

😄 minikube v1.26.0 on Fedora 36 ▪ MINIKUBE_ROOTLESS=true ✨ Using the podman driver based on user configuration 📌 Using rootless Podman driver 👍 Starting control plane node minikube in cluster minikube 🚜 Pulling base image … 💾 Downloading Kubernetes v1.24.1 preload … > preloaded-images-k8s-v18-v1…: 473.80 MiB / 473.80 MiB 100.00% 42.14 Mi > gcr.io/k8s-minikube/kicbase: 386.00 MiB / 386.00 MiB 100.00% 24.25 MiB p E0623 09:01:31.296248 2614845 cache.go:203] Error downloading kic artifacts: not yet implemented, see issue #8426 🔥 Creating podman container (CPUs=2, Memory=8192MB) … 🤦 StartHost failed, but will try again: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125 stdout:

stderr: Error: volume with name minikube already exists: volume already exists

🔄 Restarting existing podman container for “minikube” … 😿 Failed to start podman container. Running “minikube delete” may fix it: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube

❌ Exiting due to GUEST_PROVISION: Failed to start host: podman inspect ip minikube: podman container inspect -f minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube

╭───────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ 😿 If the above advice does not help, please let us know: │ │ 👉 https://github.com/kubernetes/minikube/issues/new/choose │ │ │ │ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │ │ │ ╰───────────────────────────────────────────────────────────────────────────────────────────╯

Attach the log file

• ==> Audit <==

• |---------|------|----------|------|---------|---------------------|----------| | Command | Args | Profile | User | Version | Start Time | End Time | |---------|------|----------|------|---------|---------------------|----------| | start | | minikube | jpedersen | v1.26.0 | 23 Jun 22 09:01 EDT | | |---------|------|----------|------|---------|---------------------|----------|

• ==> Last Start <==

• Log file created at: 2022/06/23 09:01:14 Running on machine: localhost Binary: Built with gc go1.18.3 for linux/amd64 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0623 09:01:14.288768 2614845 out.go:296] Setting OutFile to fd 1 … I0623 09:01:14.288842 2614845 out.go:348] isatty.IsTerminal(1) = true I0623 09:01:14.288844 2614845 out.go:309] Setting ErrFile to fd 2… I0623 09:01:14.288847 2614845 out.go:348] isatty.IsTerminal(2) = true I0623 09:01:14.289234 2614845 root.go:329] Updating PATH: /home/jpedersen/.minikube/bin I0623 09:01:14.289675 2614845 out.go:303] Setting JSON to false I0623 09:01:14.303388 2614845 start.go:115] hostinfo: {“hostname”:“localhost.localdomain”,“uptime”:1280733,“bootTime”:1654708541,“procs”:490,“os”:“linux”,“platform”:“fedora”,“platformFamily”:“fedora”,“platformVersion”:“36”,“kernelVersion”:“5.17.12-300.fc36.x86_64”,“kernelArch”:“x86_64”,“virtualizationSystem”:“kvm”,“virtualizationRole”:“host”,“hostId”:“210e73a9-68eb-4c2f-86ec-60f335bd6933”} I0623 09:01:14.303437 2614845 start.go:125] virtualization: kvm host I0623 09:01:14.309894 2614845 out.go:177] 😄 minikube v1.26.0 on Fedora 36 I0623 09:01:14.313124 2614845 out.go:177] ▪ MINIKUBE_ROOTLESS=true W0623 09:01:14.313185 2614845 preload.go:295] Failed to list preload files: open /home/jpedersen/.minikube/cache/preloaded-tarball: no such file or directory I0623 09:01:14.313201 2614845 notify.go:193] Checking for updates… I0623 09:01:14.316410 2614845 driver.go:360] Setting default libvirt URI to qemu:///system I0623 09:01:14.472425 2614845 podman.go:123] podman version: 4.1.1 I0623 09:01:14.478711 2614845 out.go:177] ✨ Using the podman driver based on user configuration I0623 09:01:14.481610 2614845 start.go:284] selected driver: podman I0623 09:01:14.481617 2614845 start.go:805] validating driver “podman” against <nil> I0623 09:01:14.481646 2614845 start.go:816] status for podman: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:} I0623 09:01:14.481793 2614845 cli_runner.go:164] Run: podman system info --format json I0623 09:01:14.693353 2614845 info.go:287] podman info: {Host:{BuildahVersion:1.26.1 CgroupVersion:v2 Conmon:{Package:conmon-2.1.0-2.fc36.x86_64 Path:/usr/bin/conmon Version:conmon version 2.1.0, commit: } Distribution:{Distribution:fedora Version:36} MemFree:1655164928 MemTotal:66862510080 OCIRuntime:{Name:crun Package:crun-1.4.5-1.fc36.x86_64 Path:/usr/bin/crun Version:crun version 1.4.5 commit: c381048530aa750495cf502ddb7181f2ded5b400 spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:17086799872 SwapTotal:17179860992 Arch:amd64 Cpus:16 Eventlogger:journald Hostname:localhost.localdomain Kernel:5.17.12-300.fc36.x86_64 Os:linux Security:{Rootless:true} Uptime:355h 45m 32.99s (Approximately 14.79 days)} Registries:{Search:[registry.fedoraproject.org registry.access.redhat.com docker.io quay.io localhost:5000]} Store:{ConfigFile:/home/jpedersen/.config/containers/storage.conf ContainerStore:{Number:1} GraphDriverName:overlay GraphOptions:{} GraphRoot:/home/jpedersen/.local/share/containers/storage GraphStatus:{BackingFilesystem:extfs NativeOverlayDiff:true SupportsDType:true UsingMetacopy:false} ImageStore:{Number:6} RunRoot:/run/user/1000/containers VolumePath:/home/jpedersen/.local/share/containers/storage/volumes}} I0623 09:01:14.693439 2614845 start_flags.go:296] no existing cluster config was found, will generate one from the flags I0623 09:01:14.706172 2614845 start_flags.go:835] Wait components to verify : map[apiserver:true system_pods:true] I0623 09:01:14.712472 2614845 out.go:177] 📌 Using rootless Podman driver I0623 09:01:14.715528 2614845 cni.go:95] Creating CNI manager for “” I0623 09:01:14.715533 2614845 cni.go:162] “podman” driver + cri-o runtime found, recommending kindnet I0623 09:01:14.715543 2614845 start_flags.go:305] Found “CNI” CNI - setting NetworkPlugin=cni I0623 09:01:14.715550 2614845 start_flags.go:310] config: {Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.32@sha256:9190bd2393eae887316c97a74370b7d5dad8f0b2ef91ac2662bc36f7ef8e0b95 Memory:8192 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.24.1 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:cri-o CRISocket: NetworkPlugin:cni FeatureGates:KubeletInUserNamespace=true ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jpedersen:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath:} I0623 09:01:14.718852 2614845 out.go:177] 👍 Starting control plane node minikube in cluster minikube I0623 09:01:14.721983 2614845 cache.go:120] Beginning downloading kic base image for podman with cri-o I0623 09:01:14.724943 2614845 out.go:177] 🚜 Pulling base image … I0623 09:01:14.730749 2614845 preload.go:132] Checking if preload exists for k8s version v1.24.1 and runtime cri-o I0623 09:01:14.730901 2614845 cache.go:147] Downloading gcr.io/k8s-minikube/kicbase:v0.0.32@sha256:9190bd2393eae887316c97a74370b7d5dad8f0b2ef91ac2662bc36f7ef8e0b95 to local cache I0623 09:01:14.731495 2614845 image.go:59] Checking for gcr.io/k8s-minikube/kicbase:v0.0.32@sha256:9190bd2393eae887316c97a74370b7d5dad8f0b2ef91ac2662bc36f7ef8e0b95 in local cache directory I0623 09:01:14.731721 2614845 image.go:119] Writing gcr.io/k8s-minikube/kicbase:v0.0.32@sha256:9190bd2393eae887316c97a74370b7d5dad8f0b2ef91ac2662bc36f7ef8e0b95 to local cache I0623 09:01:14.792187 2614845 preload.go:119] Found remote preload: https://storage.googleapis.com/minikube-preloaded-volume-tarballs/v18/v1.24.1/preloaded-images-k8s-v18-v1.24.1-cri-o-overlay-amd64.tar.lz4 I0623 09:01:14.792203 2614845 cache.go:57] Caching tarball of preloaded images I0623 09:01:14.792393 2614845 preload.go:132] Checking if preload exists for k8s version v1.24.1 and runtime cri-o I0623 09:01:14.795869 2614845 out.go:177] 💾 Downloading Kubernetes v1.24.1 preload … I0623 09:01:14.798918 2614845 preload.go:238] getting checksum for preloaded-images-k8s-v18-v1.24.1-cri-o-overlay-amd64.tar.lz4 … I0623 09:01:14.948593 2614845 download.go:101] Downloading: https://storage.googleapis.com/minikube-preloaded-volume-tarballs/v18/v1.24.1/preloaded-images-k8s-v18-v1.24.1-cri-o-overlay-amd64.tar.lz4?checksum=md5:4c8ad2429eafc79a0e5a20bdf41ae0bc -> /home/jpedersen/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.24.1-cri-o-overlay-amd64.tar.lz4 I0623 09:01:27.403083 2614845 preload.go:249] saving checksum for preloaded-images-k8s-v18-v1.24.1-cri-o-overlay-amd64.tar.lz4 … I0623 09:01:27.403159 2614845 preload.go:256] verifying checksumm of /home/jpedersen/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.24.1-cri-o-overlay-amd64.tar.lz4 … I0623 09:01:28.396230 2614845 cache.go:60] Finished verifying existence of preloaded tar for v1.24.1 on cri-o I0623 09:01:28.396451 2614845 profile.go:148] Saving config to /home/jpedersen/.minikube/profiles/minikube/config.json … I0623 09:01:28.396464 2614845 lock.go:35] WriteFile acquiring /home/jpedersen/.minikube/profiles/minikube/config.json: {Name:mkb9c351ba1576e37af5fa49932a884f1bd885a6 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>} I0623 09:01:31.296227 2614845 cache.go:150] successfully saved gcr.io/k8s-minikube/kicbase:v0.0.32@sha256:9190bd2393eae887316c97a74370b7d5dad8f0b2ef91ac2662bc36f7ef8e0b95 as a tarball E0623 09:01:31.296248 2614845 cache.go:203] Error downloading kic artifacts: not yet implemented, see issue #8426 I0623 09:01:31.296267 2614845 cache.go:208] Successfully downloaded all kic artifacts I0623 09:01:31.296318 2614845 start.go:352] acquiring machines lock for minikube: {Name:mka018440ecc214ae079b0f3318b8bab19ffd57a Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>} I0623 09:01:31.296461 2614845 start.go:356] acquired machines lock for “minikube” in 123.343µs I0623 09:01:31.296493 2614845 start.go:91] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.32@sha256:9190bd2393eae887316c97a74370b7d5dad8f0b2ef91ac2662bc36f7ef8e0b95 Memory:8192 CPUs:2 DiskSize:20000 VMDriver: Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.24.1 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:cri-o CRISocket: NetworkPlugin:cni FeatureGates:KubeletInUserNamespace=true ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.24.1 ContainerRuntime:cri-o ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jpedersen:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath:} &{Name: IP: Port:8443 KubernetesVersion:v1.24.1 ContainerRuntime:cri-o ControlPlane:true Worker:true} I0623 09:01:31.296608 2614845 start.go:131] createHost starting for “” (driver=“podman”) I0623 09:01:31.303053 2614845 out.go:204] 🔥 Creating podman container (CPUs=2, Memory=8192MB) … I0623 09:01:31.303523 2614845 start.go:165] libmachine.API.Create for “minikube” (driver=“podman”) I0623 09:01:31.303548 2614845 client.go:168] LocalClient.Create starting I0623 09:01:31.303715 2614845 main.go:134] libmachine: Creating CA: /home/jpedersen/.minikube/certs/ca.pem I0623 09:01:31.582193 2614845 main.go:134] libmachine: Creating client certificate: /home/jpedersen/.minikube/certs/cert.pem I0623 09:01:31.747218 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:31.910681 2614845 cli_runner.go:164] Run: podman network inspect minikube --format “{{range .}}{{if eq .Driver “bridge”}}{{(index .Subnets 0).Subnet}},{{(index .Subnets 0).Gateway}}{{end}}{{end}}” I0623 09:01:32.056211 2614845 network_create.go:76] Found existing network {name:minikube subnet:0xc00118c750 gateway:[0 0 0 0 0 0 0 0 0 0 255 255 192 168 49 1] mtu:0} I0623 09:01:32.056260 2614845 kic.go:106] calculated static IP “192.168.49.2” for the “minikube” container I0623 09:01:32.056411 2614845 cli_runner.go:164] Run: podman ps -a --format {{.Names}} I0623 09:01:32.197216 2614845 cli_runner.go:164] Run: podman container inspect minikube --format {{.Config.Labels}} I0623 09:01:32.356615 2614845 kic.go:154] Found already existing abandoned minikube container, will try to delete. I0623 09:01:32.356787 2614845 cli_runner.go:164] Run: podman container inspect minikube --format={{.State.Status}} I0623 09:01:32.518253 2614845 cli_runner.go:164] Run: podman exec --privileged -t minikube /bin/bash -c “sudo init 0” I0623 09:01:33.681451 2614845 cli_runner.go:164] Run: podman container inspect minikube --format={{.State.Status}} I0623 09:01:33.825755 2614845 oci.go:660] temporary error: container minikube status is Running but expect it to be exited I0623 09:01:33.825774 2614845 oci.go:666] Successfully shutdown container minikube I0623 09:01:33.826017 2614845 cli_runner.go:164] Run: podman rm -f -v minikube I0623 09:01:44.436407 2614845 cli_runner.go:217] Completed: podman rm -f -v minikube: (10.610369038s) I0623 09:01:44.436628 2614845 cli_runner.go:164] Run: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true W0623 09:01:44.511888 2614845 cli_runner.go:211] podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true returned with exit code 125 I0623 09:01:44.511948 2614845 client.go:171] LocalClient.Create took 13.208391098s I0623 09:01:46.513016 2614845 ssh_runner.go:195] Run: sh -c “df -h /var | awk ‘NR==2{print $5}’” I0623 09:01:46.513139 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:46.651830 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:46.691885 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 I0623 09:01:46.692042 2614845 retry.go:31] will retry after 276.165072ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:46.968848 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:47.112367 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:47.154247 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 I0623 09:01:47.154399 2614845 retry.go:31] will retry after 540.190908ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:47.694965 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:47.855900 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:47.919498 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 W0623 09:01:47.919719 2614845 start.go:281] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube

W0623 09:01:47.919734 2614845 start.go:248] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:47.919858 2614845 ssh_runner.go:195] Run: sh -c “df -BG /var | awk ‘NR==2{print $4}’” I0623 09:01:47.919956 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:48.060680 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:48.121141 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 I0623 09:01:48.121271 2614845 retry.go:31] will retry after 234.428547ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:48.356174 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:48.509568 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:48.566594 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 I0623 09:01:48.566737 2614845 retry.go:31] will retry after 346.739061ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:48.914419 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:49.058822 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:49.121049 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 I0623 09:01:49.121116 2614845 retry.go:31] will retry after 667.587979ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:49.789137 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:49.932957 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:49.992948 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 W0623 09:01:49.993080 2614845 start.go:296] error running df -BG /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube

W0623 09:01:49.993092 2614845 start.go:253] error getting GiB of /var that is available: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:49.993099 2614845 start.go:134] duration metric: createHost completed in 18.696484428s I0623 09:01:49.993117 2614845 start.go:81] releasing machines lock for “minikube”, held for 18.696637626s W0623 09:01:49.993142 2614845 start.go:599] error starting host: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125 stdout:

stderr: Error: volume with name minikube already exists: volume already exists I0623 09:01:49.993769 2614845 cli_runner.go:164] Run: podman container inspect minikube --format={{.State.Status}} W0623 09:01:50.035055 2614845 cli_runner.go:211] podman container inspect minikube --format={{.State.Status}} returned with exit code 125 I0623 09:01:50.035138 2614845 delete.go:82] Unable to get host status for minikube, assuming it has already been deleted: state: unknown state “minikube”: podman container inspect minikube --format={{.State.Status}}: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube W0623 09:01:50.035330 2614845 out.go:239] 🤦 StartHost failed, but will try again: creating host: create: creating: setting up container node: creating volume for minikube container: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125 stdout:

stderr: Error: volume with name minikube already exists: volume already exists

I0623 09:01:50.035402 2614845 start.go:614] Will try again in 5 seconds … I0623 09:01:55.035794 2614845 start.go:352] acquiring machines lock for minikube: {Name:mka018440ecc214ae079b0f3318b8bab19ffd57a Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>} I0623 09:01:55.036083 2614845 start.go:356] acquired machines lock for “minikube” in 261.133µs I0623 09:01:55.036106 2614845 start.go:94] Skipping create…Using existing machine configuration I0623 09:01:55.036112 2614845 fix.go:55] fixHost starting: I0623 09:01:55.036549 2614845 cli_runner.go:164] Run: podman container inspect minikube --format={{.State.Status}} W0623 09:01:55.077543 2614845 cli_runner.go:211] podman container inspect minikube --format={{.State.Status}} returned with exit code 125 I0623 09:01:55.077578 2614845 fix.go:103] recreateIfNeeded on minikube: state= err=unknown state “minikube”: podman container inspect minikube --format={{.State.Status}}: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:55.077584 2614845 fix.go:108] machineExists: true. err=unknown state “minikube”: podman container inspect minikube --format={{.State.Status}}: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube W0623 09:01:55.077590 2614845 fix.go:129] unexpected machine state, will restart: unknown state “minikube”: podman container inspect minikube --format={{.State.Status}}: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:55.083445 2614845 out.go:177] 🔄 Restarting existing podman container for “minikube” … I0623 09:01:55.092255 2614845 cli_runner.go:164] Run: podman start minikube W0623 09:01:55.125132 2614845 cli_runner.go:211] podman start minikube returned with exit code 125 I0623 09:01:55.125216 2614845 cli_runner.go:164] Run: podman inspect minikube I0623 09:01:55.292051 2614845 errors.go:84] Postmortem inspect (“podman inspect minikube”): – stdout – [ { “Name”: “minikube”, “Driver”: “local”, “Mountpoint”: “/home/jpedersen/.local/share/containers/storage/volumes/minikube/_data”, “CreatedAt”: “2022-06-23T01:48:07.818393788-04:00”, “Labels”: { “created_by.minikube.sigs.k8s.io”: “true”, “name.minikube.sigs.k8s.io”: “minikube” }, “Scope”: “local”, “Options”: {}, “MountCount”: 0 } ]

– /stdout – I0623 09:01:55.292202 2614845 cli_runner.go:164] Run: podman logs --timestamps minikube W0623 09:01:55.357282 2614845 cli_runner.go:211] podman logs --timestamps minikube returned with exit code 125 W0623 09:01:55.357317 2614845 errors.go:89] Failed to get postmortem logs. podman logs --timestamps minikube :podman logs --timestamps minikube: exit status 125 stdout:

stderr: Error: no container with name or ID “minikube” found: no such container I0623 09:01:55.357482 2614845 cli_runner.go:164] Run: podman system info --format json I0623 09:01:55.534338 2614845 info.go:287] podman info: {Host:{BuildahVersion:1.26.1 CgroupVersion:v2 Conmon:{Package:conmon-2.1.0-2.fc36.x86_64 Path:/usr/bin/conmon Version:conmon version 2.1.0, commit: } Distribution:{Distribution:fedora Version:36} MemFree:1159290880 MemTotal:66862510080 OCIRuntime:{Name:crun Package:crun-1.4.5-1.fc36.x86_64 Path:/usr/bin/crun Version:crun version 1.4.5 commit: c381048530aa750495cf502ddb7181f2ded5b400 spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:17086799872 SwapTotal:17179860992 Arch:amd64 Cpus:16 Eventlogger:journald Hostname:localhost.localdomain Kernel:5.17.12-300.fc36.x86_64 Os:linux Security:{Rootless:true} Uptime:355h 46m 13.86s (Approximately 14.79 days)} Registries:{Search:[registry.fedoraproject.org registry.access.redhat.com docker.io quay.io localhost:5000]} Store:{ConfigFile:/home/jpedersen/.config/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/home/jpedersen/.local/share/containers/storage GraphStatus:{BackingFilesystem:extfs NativeOverlayDiff:true SupportsDType:true UsingMetacopy:false} ImageStore:{Number:6} RunRoot:/run/user/1000/containers VolumePath:/home/jpedersen/.local/share/containers/storage/volumes}} I0623 09:01:55.534397 2614845 errors.go:106] postmortem podman info: {Host:{BuildahVersion:1.26.1 CgroupVersion:v2 Conmon:{Package:conmon-2.1.0-2.fc36.x86_64 Path:/usr/bin/conmon Version:conmon version 2.1.0, commit: } Distribution:{Distribution:fedora Version:36} MemFree:1159290880 MemTotal:66862510080 OCIRuntime:{Name:crun Package:crun-1.4.5-1.fc36.x86_64 Path:/usr/bin/crun Version:crun version 1.4.5 commit: c381048530aa750495cf502ddb7181f2ded5b400 spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:17086799872 SwapTotal:17179860992 Arch:amd64 Cpus:16 Eventlogger:journald Hostname:localhost.localdomain Kernel:5.17.12-300.fc36.x86_64 Os:linux Security:{Rootless:true} Uptime:355h 46m 13.86s (Approximately 14.79 days)} Registries:{Search:[registry.fedoraproject.org registry.access.redhat.com docker.io quay.io localhost:5000]} Store:{ConfigFile:/home/jpedersen/.config/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/home/jpedersen/.local/share/containers/storage GraphStatus:{BackingFilesystem:extfs NativeOverlayDiff:true SupportsDType:true UsingMetacopy:false} ImageStore:{Number:6} RunRoot:/run/user/1000/containers VolumePath:/home/jpedersen/.local/share/containers/storage/volumes}} I0623 09:01:55.534580 2614845 network_create.go:272] running [podman network inspect minikube] to gather additional debugging logs… I0623 09:01:55.534609 2614845 cli_runner.go:164] Run: podman network inspect minikube I0623 09:01:55.668149 2614845 network_create.go:277] output of [podman network inspect minikube]: – stdout – [ { “name”: “minikube”, “id”: “8b5429f977923686307bb50cfaa110e135e2e878bd5e3dc0d17a293467dcc6f8”, “driver”: “bridge”, “network_interface”: “podman2”, “created”: “2022-06-14T12:51:56.4525086-04:00”, “subnets”: [ { “subnet”: “192.168.49.0/24”, “gateway”: “192.168.49.1” } ], “ipv6_enabled”: false, “internal”: false, “dns_enabled”: true, “labels”: { “created_by.minikube.sigs.k8s.io”: “true” }, “ipam_options”: { “driver”: “host-local” } } ]

– /stdout – I0623 09:01:55.668326 2614845 cli_runner.go:164] Run: podman system info --format json I0623 09:01:55.843391 2614845 info.go:287] podman info: {Host:{BuildahVersion:1.26.1 CgroupVersion:v2 Conmon:{Package:conmon-2.1.0-2.fc36.x86_64 Path:/usr/bin/conmon Version:conmon version 2.1.0, commit: } Distribution:{Distribution:fedora Version:36} MemFree:1159413760 MemTotal:66862510080 OCIRuntime:{Name:crun Package:crun-1.4.5-1.fc36.x86_64 Path:/usr/bin/crun Version:crun version 1.4.5 commit: c381048530aa750495cf502ddb7181f2ded5b400 spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL} SwapFree:17086799872 SwapTotal:17179860992 Arch:amd64 Cpus:16 Eventlogger:journald Hostname:localhost.localdomain Kernel:5.17.12-300.fc36.x86_64 Os:linux Security:{Rootless:true} Uptime:355h 46m 14.14s (Approximately 14.79 days)} Registries:{Search:[registry.fedoraproject.org registry.access.redhat.com docker.io quay.io localhost:5000]} Store:{ConfigFile:/home/jpedersen/.config/containers/storage.conf ContainerStore:{Number:0} GraphDriverName:overlay GraphOptions:{} GraphRoot:/home/jpedersen/.local/share/containers/storage GraphStatus:{BackingFilesystem:extfs NativeOverlayDiff:true SupportsDType:true UsingMetacopy:false} ImageStore:{Number:6} RunRoot:/run/user/1000/containers VolumePath:/home/jpedersen/.local/share/containers/storage/volumes}} I0623 09:01:55.844041 2614845 cli_runner.go:164] Run: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube W0623 09:01:55.907583 2614845 cli_runner.go:211] podman container inspect -f {{.NetworkSettings.IPAddress}} minikube returned with exit code 125 I0623 09:01:55.907781 2614845 ssh_runner.go:195] Run: sh -c “df -h /var | awk ‘NR==2{print $5}’” I0623 09:01:55.907916 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:56.045351 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:56.090963 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 I0623 09:01:56.091111 2614845 retry.go:31] will retry after 164.129813ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:56.255743 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:56.420380 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:56.478226 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 I0623 09:01:56.478367 2614845 retry.go:31] will retry after 223.863569ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:56.702925 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:56.846621 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:56.879456 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 I0623 09:01:56.879617 2614845 retry.go:31] will retry after 450.512921ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:57.330257 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:57.465756 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:57.500734 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 W0623 09:01:57.500886 2614845 start.go:281] error running df -h /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube

W0623 09:01:57.500902 2614845 start.go:248] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:57.501002 2614845 ssh_runner.go:195] Run: sh -c “df -BG /var | awk ‘NR==2{print $4}’” I0623 09:01:57.501070 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:57.636717 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:57.689810 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 I0623 09:01:57.689948 2614845 retry.go:31] will retry after 328.409991ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:58.018619 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:58.157473 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:58.192903 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 I0623 09:01:58.193069 2614845 retry.go:31] will retry after 267.848952ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:58.461631 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:58.599604 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:58.635539 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 I0623 09:01:58.635754 2614845 retry.go:31] will retry after 495.369669ms: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:59.131481 2614845 cli_runner.go:164] Run: podman version --format {{.Version}} I0623 09:01:59.268483 2614845 cli_runner.go:164] Run: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube W0623 09:01:59.312101 2614845 cli_runner.go:211] podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube returned with exit code 125 W0623 09:01:59.312232 2614845 start.go:296] error running df -BG /var: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube

W0623 09:01:59.312243 2614845 start.go:253] error getting GiB of /var that is available: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: get port 22 for “minikube”: podman container inspect -f “‘{{(index (index .NetworkSettings.Ports “22/tcp”) 0).HostPort}}’” minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube I0623 09:01:59.312250 2614845 fix.go:57] fixHost completed within 4.276138739s I0623 09:01:59.312256 2614845 start.go:81] releasing machines lock for “minikube”, held for 4.276164848s W0623 09:01:59.312420 2614845 out.go:239] 😿 Failed to start podman container. Running “minikube delete” may fix it: podman inspect ip minikube: podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube

I0623 09:01:59.318523 2614845 out.go:177] W0623 09:01:59.321492 2614845 out.go:239] ❌ Exiting due to GUEST_PROVISION: Failed to start host: podman inspect ip minikube: podman container inspect -f minikube: exit status 125 stdout:

stderr: Error: inspecting object: no such container minikube

W0623 09:01:59.321520 2614845 out.go:239] W0623 09:01:59.322809 2614845 out.go:239] ╭───────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ 😿 If the above advice does not help, please let us know: │ │ 👉 https://github.com/kubernetes/minikube/issues/new/choose │ │ │ │ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │ │ │ ╰───────────────────────────────────────────────────────────────────────────────────────────╯ I0623 09:01:59.327563 2614845 out.go:177]

Operating System

Redhat/Fedora

Driver

Podman