crc: [BUG] OKD 4.12 does not start because of missing kubeadmin-password file

General information

OS: Linux
Hypervisor: KVM
Did you run crc setup before starting it (Yes)?
Running CRC on: Laptop

CRC version

crc version
CRC version: 2.18.0+4ea3a1
OpenShift version: 4.12.13
Podman version: 4.4.1

CRC status

crc status --log-level debug
DEBU CRC version: 2.18.0+4ea3a1                   
DEBU OpenShift version: 4.12.13                   
DEBU Podman version: 4.4.1                        
DEBU Running 'crc status'                         
CRC VM:          Running
OpenShift:       Stopped (v4.12.0-0.okd-2023-02-18-033438)
RAM Usage:       5.292GB of 25.22GB
Disk Usage:      16.23GB of 32.68GB (Inside the CRC VM)
Cache Usage:     38.11GB
Cache Directory: /home/user/.crc/cache

CRC config

crc status --log-level debug
DEBU CRC version: 2.18.0+4ea3a1                   
DEBU OpenShift version: 4.12.13                   
DEBU Podman version: 4.4.1                        
DEBU Running 'crc status'                         
CRC VM:          Running
OpenShift:       Stopped (v4.12.0-0.okd-2023-02-18-033438)
RAM Usage:       5.292GB of 25.22GB
Disk Usage:      16.23GB of 32.68GB (Inside the CRC VM)
Cache Usage:     38.11GB
Cache Directory: /home/user/.crc/cache

Host Operating System

cat /etc/os-release
NAME="Fedora Linux"
VERSION="38 (Workstation Edition)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora Linux 38 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f38/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Steps to reproduce

download latest crc binary
crc config set preset okd
crc setup ( which downloads okd 4.12)
crc start

Expected

succesfull setup

Actual

fails with :

Failed to update kubeadmin user password: Cannot generate the kubeadmin user password: open /home/user/.crc/machines/crc/kubeadmin-password: no such file or directory

Indeed the directory and the file do not exist, which seems to be the cause of the issue. If I set these up with an empty file, the start continues and then complains about the machine qemu-1-crc existing, which seems to be the crc machine. When I remove that, the same problem occurs.

The issue does not seem to happen with OCP 4.12, only OKD.

Logs


INFO Checking if CRC bundle is extracted in '$HOME/.crc' 
INFO Checking if /home/user/.crc/cache/crc_okd_libvirt_4.12.0-0.okd-2023-02-18-033438_amd64.crcbundle exists 
INFO Getting bundle for the CRC executable        
INFO Downloading bundle: /home/user/.crc/cache/crc_okd_libvirt_4.12.0-0.okd-2023-02-18-033438_amd64.crcbundle... 
Getting image source signatures
Copying blob 0d76ba17a55c done  
Copying config 9d8cdd8dcc done  
Writing manifest to image destination
Storing signatures
INFO Extracting the image bundle layer...         
crc_okd_libvirt_4.12.0-0.okd-2023-02-18-033438_amd64.crcbundle:  3.86 GiB / 3.86 GiB [-----------------------------------------------------------------] 100.00%
INFO Verifying the bundle signature...            
INFO Uncompressing /home/user/.crc/cache/crc_okd_libvirt_4.12.0-0.okd-2023-02-18-033438_amd64.crcbundle 
crc.qcow2:  14.26 GiB / 14.26 GiB [--------------------------------------------------------------------------------------------------------------------] 100.00%
oc:  124.65 MiB / 124.65 MiB [-------------------------------------------------------------------------------------------------------------------------] 100.00%
Your system is correctly setup for using CRC. Use 'crc start' to start the instance
[user@fedora Downloads]$ crc start

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

Snippet of debug log:

DEBU error: Temporary error: pull secret not updated to disk - sleeping 2s DEBU retry loop: attempt 41
DEBU Running SSH command: <hidden>
DEBU SSH command succeeded
DEBU error: Temporary error: pull secret not updated to disk - sleeping 2s DEBU retry loop: attempt 42
DEBU Running SSH command: <hidden>
DEBU SSH command succeeded

then

Failed to update kubeadmin user password: Cannot generate the kubeadmin user password: open /home/user/.crc/machines/crc/kubeadmin-password: no such file or directory

Failed to update kubeadmin user password: Cannot generate the kubeadmin user password: open

About this issue

Original URL
State: closed
Created a year ago
Comments: 25 (20 by maintainers)

Commits related to this issue

machine: start: Generate new kubeadmin file for okd as well It's only done for the openshift preset at the moment, but okd also expects a kubeadmin file to be created. This should fix part of https:/... — committed to cfergeau/crc by cfergeau a year ago
machine: start: Generate new kubeadmin file for okd as well It's only done for the openshift preset at the moment, but okd also expects a kubeadmin file to be created. This should fix part of https:/... — committed to cfergeau/crc by cfergeau a year ago
CrcBundleInfo's IsOpenShift() now considers OKD to also be OpenShift There are numerous places in the CRC code base that calls IsOpenShift() to run logic that is not intended for Podman setups or oth... — committed to bergner/crc by deleted user a year ago
CrcBundleInfo's IsOpenShift() now considers OKD to also be OpenShift There are numerous places in the CRC code base that calls IsOpenShift() to run logic that is not intended for Podman setups or oth... — committed to bergner/crc by deleted user a year ago

Most upvoted comments

https://github.com/crc-org/snc/pull/733 is how it should be handled as part of bundle generation.

praveenkumar on Jun 5, 2023

Looks like this is issue of our bundle generation script https://github.com/crc-org/snc/blob/master/createdisk.sh#L116-L122 we are setting right internal IP in case of OCP but not in case of OKD, we will put a PR and may be when next bundle is generated shouldn’t have any issue.

praveenkumar on Jun 5, 2023

I tested with the latest crc and it works like a charm. Thank you! OKD version : 4.13.0-0.okd-2023-06-04-080300 https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/2.24.1 Closing this, thanks again!

VGerris on Aug 21, 2023

Ok, this is another ugly workaround that made my instance start.

ssh -i /home/youruser/.crc/machines/crc/id_ecdsa -p 22 core@192.168.130.11 to login to the vm
sudo su to become root
vi /etc/systemd/system/kubelet.service ( or use your favorite editor that is installed 😃 -> change KUBE_NODE_IP to 192.168.126.11
systemctl restart kubelet -> restarts kubelet server

The above got my cluster working but only the developer account seemed to exist. The authentication is configured with htacess. To have oc run as admin and fix the kubeadmin user you can do the following ( being logged in to the VM):

export KUBEADMIN=/opt/kubeadmin -> oc uses this for auth
oc create user kubeadmin -> creates the user
https://docs.okd.io/4.13/authentication/identity_providers/configuring-htpasswd-identity-provider.html , in short
oc get secret htpass-secret -ojsonpath={.data.htpasswd} -n openshift-config | base64 --decode > users.htpasswd -> gets secret with users
update password for kubeadmin ( I haven´t tried it the one printed worked, I copied the developer one so it is ‘developer’
oc create secret generic htpass-secret --from-file=htpasswd=users.htpasswd --dry-run=client -o yaml -n openshift-config | oc replace -f -
oc adm policy add-cluster-role-to-user cluster-admin kubeadmin -> make kubeadmin cluster admin
should work to login now with kubeadmin with password ‘developer’

At restart I get : Failed to update pull secret on the disk: Temporary error: pull secret not updated to disk (x207) but I can use the cluster now.

I cannot find how the mentioned var is set so not sure how this should be fixed. This has some good pointers: https://github.com/crc-org/crc/issues/1888 Perhaps someone with more knowledge can look into that? At this moment I cannot say it this is a consequence of the PR, but it seems unlikely to me.

The root of the issue is that the kubelet service uses the wrong IP to look at and start etcd , which is set by KUBE_NODE_IP in the systemd file.

I will leave this for a bit now and hope someone can clarify this further and/or fix it structurally, thank you.

VGerris on Jun 3, 2023

I tried both https://github.com/cfergeau/crc “okd” branch and https://github.com/crc-org/crc HEAD of the master branch.

Thanks for testing/improving my branch! 😃

cfergeau on May 15, 2023