kubefire: Failed to establish a connection with cluster after creation

It was working just fine but after my computer lost power while I had a cluster running I am unable to start subsequent clusters.

➜  ~ kubefire cluster create --bootstrapper=kubeadm demo
INFO[2020-08-03T14:29:06+02:00] initializing cluster configuration            cluster=demo
INFO[2020-08-03T14:29:06+02:00] saving cluster configurations                 cluster=demo
INFO[2020-08-03T14:29:06+02:00] creating cluster (demo)                      
INFO[2020-08-03T14:29:06+02:00] creating master nodes of cluster (demo)      
INFO[2020-08-03T14:29:06+02:00] creating node (demo-master-1)                
[sudo] password for rugwiro: 
INFO[0015] Created VM with ID "fcff98a0b99793bb" and name "demo-master-1" 
INFO[0028] Networking is handled by "cni"               
INFO[0028] Started Firecracker VM "fcff98a0b99793bb" in a container with ID "ignite-fcff98a0b99793bb" 
INFO[2020-08-03T14:29:45+02:00] waiting nodes of cluster (demo) are running  
INFO[2020-08-03T14:29:45+02:00] initializing cluster (demo)                  
INFO[2020-08-03T14:29:45+02:00] waiting all nodes initialization finished    
INFO[2020-08-03T14:29:47+02:00] bootstrapping the first master node (demo-master-1) 
Usage:
............

FATA[2020-08-03T14:29:47+02:00] failed to run kubefire                        error="failed to deploy cluster (demo): dial tcp :22: connect: connection refused"

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 27 (15 by maintainers)

Most upvoted comments

I believe something messed up in the below directories managed by ignite when you encountered power off.

/var/lib/firecracker
/usr/local/bin/ignite{,d}

It will be better to find the steps to reproduce, then report the issue to ignite project.

innobead on Aug 6, 2020

!

So, it works after reinstall by kubefire uninstall and kubefire install?

Yes

rugwirobaker on Aug 6, 2020

I think it’s working, I’m it’s currently running kubeadm

➜  ~ kubefire cluster create --log-level=debug --bootstrapper=kubeadm demo
INFO[2020-08-06T15:09:24+02:00] initializing cluster configuration            cluster=demo
DEBU[2020-08-06T15:09:24+02:00] &{Name:demo Bootstrapper:kubeadm Pubkey: Prikey: Image:innobead/kubefire-opensuse-leap:15.2 KernelImage:innobead/kubefire-kernel-4.19.125-amd64:latest KernelArgs:console=ttyS0 reboot=k panic=1 pci=off ip=dhcp security=apparmor apparmor=1 Admin:{Count:0 Memory:512MB Cpus:1 DiskSize:2GB Cluster:0xc0000b6780} Master:{Count:1 Memory:2GB Cpus:2 DiskSize:10GB Cluster:0xc0000b6780} Worker:{Count:0 Memory:2GB Cpus:2 DiskSize:10GB Cluster:0xc0000b6780} ExtraOptions:} 
DEBU[2020-08-06T15:09:24+02:00] getting cluster configurations                cluster=demo
INFO[2020-08-06T15:09:24+02:00] saving cluster configurations                 cluster=demo
INFO[2020-08-06T15:09:24+02:00] creating cluster                              cluster=demo
DEBU[2020-08-06T15:09:24+02:00] getting cluster configurations                cluster=demo
INFO[2020-08-06T15:09:24+02:00] creating master nodes of cluster              cluster=demo
INFO[2020-08-06T15:09:24+02:00] creating node                                 node=demo-master-1
INFO[2020-08-06T15:09:24+02:00] time="2020-08-06T15:09:24+02:00" level=info msg="Starting image import..." 
INFO[2020-08-06T15:10:27+02:00] time="2020-08-06T15:10:27+02:00" level=info msg="Imported OCI image \"innobead/kubefire-opensuse-leap:15.2\" (292.6 MB) to base image with UID \"4514d798339a7ce8\"" 
INFO[2020-08-06T15:10:42+02:00] time="2020-08-06T15:10:42+02:00" level=info msg="Imported OCI image \"innobead/kubefire-kernel-4.19.125-amd64:latest\" (154.4 MB) to kernel image with UID \"fa11be4acc96109b\"" 
INFO[2020-08-06T15:10:56+02:00] time="2020-08-06T15:10:56+02:00" level=info msg="Created VM with ID \"1f6b10b707fadad4\" and name \"demo-master-1\"" 
INFO[2020-08-06T15:11:02+02:00] time="2020-08-06T15:11:02+02:00" level=info msg="Networking is handled by \"cni\""
time="2020-08-06T15:11:02+02:00" level=info msg="Started Firecracker VM \"1f6b10b707fadad4\" in a container with ID \"ignite-1f6b10b707fadad4\"" 
DEBU[2020-08-06T15:11:02+02:00] getting cluster                               cluster=demo
DEBU[2020-08-06T15:11:02+02:00] getting cluster configurations                cluster=demo
DEBU[2020-08-06T15:11:02+02:00] listing nodes of cluster                      cluster=demo
DEBU[2020-08-06T15:11:02+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=~demo- -t {{.ObjectMeta.Name}}] 
DEBU[2020-08-06T15:11:03+02:00] getting node                                  node=demo-master-1
DEBU[2020-08-06T15:11:03+02:00] VM ID			IMAGE					KERNEL						SIZE	CPUS	     
DEBU[2020-08-06T15:11:03+02:00] MEMORY	CREATED	STATUS	IPS		PORTS	NAME
1f6b10b707fadad4	innobead/kubefire-opensuse-leap:15.2	innobead/kubefire-kernel-4.19.125-amd64:latest	10.0 GB	2	2.0 GB	1s ago	Up 1s	10.61.0.47		demo-master-1 
DEBU[2020-08-06T15:11:03+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.ObjectMeta.Labels.cluster}}] 
DEBU[2020-08-06T15:11:03+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Spec.CPUs}}] 
DEBU[2020-08-06T15:11:03+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Spec.Memory}}] 
DEBU[2020-08-06T15:11:03+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Spec.DiskSize}}] 
DEBU[2020-08-06T15:11:03+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Status.Kernel.ID}}] 
DEBU[2020-08-06T15:11:03+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Status.Running}}] 
DEBU[2020-08-06T15:11:03+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Status.IPAddresses}}] 
DEBU[2020-08-06T15:11:03+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Status.Image.ID}}] 
INFO[2020-08-06T15:11:03+02:00] waiting nodes of cluster are running          cluster=demo
DEBU[2020-08-06T15:11:03+02:00] listing nodes of cluster                      cluster=demo
DEBU[2020-08-06T15:11:03+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=~demo- -t {{.ObjectMeta.Name}}] 
DEBU[2020-08-06T15:11:04+02:00] getting node                                  node=demo-master-1
DEBU[2020-08-06T15:11:04+02:00] VM ID			IMAGE					KERNEL						SIZE	CPUS	MEMORY	CREATED 
DEBU[2020-08-06T15:11:04+02:00] 	STATUS	IPS		PORTS	NAME
1f6b10b707fadad4	innobead/kubefire-opensuse-leap:15.2	innobead/kubefire-kernel-4.19.125-amd64:latest	10.0 GB	2	2.0 GB	2s ago	Up 2s	10.61.0.47		demo-master-1 
DEBU[2020-08-06T15:11:04+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.ObjectMeta.Labels.cluster}}] 
DEBU[2020-08-06T15:11:04+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Spec.CPUs}}] 
DEBU[2020-08-06T15:11:04+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Spec.Memory}}] 
DEBU[2020-08-06T15:11:04+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Spec.DiskSize}}] 
DEBU[2020-08-06T15:11:04+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Status.Running}}] 
DEBU[2020-08-06T15:11:04+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Status.IPAddresses}}] 
DEBU[2020-08-06T15:11:04+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Status.Image.ID}}] 
DEBU[2020-08-06T15:11:04+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.Status.Kernel.ID}}] 
INFO[2020-08-06T15:11:04+02:00] initializing cluster                          cluster=demo
INFO[2020-08-06T15:11:04+02:00] initializing node                             node=demo-master-1
INFO[2020-08-06T15:11:04+02:00] waiting all nodes initialization finished    
INFO[2020-08-06T15:11:35+02:00] running swapoff -a                            node=demo-master-1
INFO[2020-08-06T15:11:36+02:00] running curl -sSLO https://raw.githubusercontent.com/innobead/kubefire/master/scripts/install-prerequisites-kubeadm.sh  node=demo-master-1
INFO[2020-08-06T15:11:39+02:00] running chmod +x install-prerequisites-kubeadm.sh  node=demo-master-1
......

Reinstalling prerequisites exitied with an error:

➜  ~ kubefire install
INFO[2020-08-06T14:26:21+02:00] downloading https://raw.githubusercontent.com/innobead/kubefire/master/scripts/install-prerequisites.sh to save /home/rugwiro/.kubefire/bin/master/install-prerequisites.sh  force=false version=master
INFO[2020-08-06T14:26:21+02:00] running script (install-prerequisites.sh)     version=master
INFO[2020-08-06T14:26:21+02:00] running /home/rugwiro/.kubefire/bin/master/install-prerequisites.sh  version=master
INFO[2020-08-06T14:26:23+02:00] + TMP_DIR=/tmp/kubefire                      
INFO[2020-08-06T14:26:23+02:00] ++ go env GOARCH                             
INFO[2020-08-06T14:26:23+02:00] ++ echo amd64                                
INFO[2020-08-06T14:26:23+02:00] + GOARCH=amd64                               
INFO[2020-08-06T14:26:23+02:00] + CONTAINERD_VERSION=v1.3.4                  
INFO[2020-08-06T14:26:23+02:00] + IGNITE_VERION=v0.7.1                       
INFO[2020-08-06T14:26:23+02:00] + CNI_VERSION=v0.8.6                         
INFO[2020-08-06T14:26:23+02:00] + RUNC_VERSION=v1.0.0-rc91                   
INFO[2020-08-06T14:26:23+02:00] + mkdir -p /tmp/kubefire                     
INFO[2020-08-06T14:26:23+02:00] + pushd /tmp/kubefire
/tmp/kubefire /home/rugwiro 
INFO[2020-08-06T14:26:23+02:00] + trap cleanup EXIT ERR INT TERM             
INFO[2020-08-06T14:26:23+02:00] + check_virtualization                       
INFO[2020-08-06T14:26:23+02:00] + lscpu                                      
INFO[2020-08-06T14:26:23+02:00] + grep Virtualization                        
INFO[2020-08-06T14:26:23+02:00] Virtualization:      VT-x                    
INFO[2020-08-06T14:26:23+02:00] + lsmod                                      
INFO[2020-08-06T14:26:23+02:00] + grep kvm                                   
INFO[2020-08-06T14:26:23+02:00] kvm_intel             212992  8
kvm                   598016  1 kvm_intel
irqbypass              16384  11 kvm 
INFO[2020-08-06T14:26:23+02:00] + install_runc
+ _check_version /usr/local/bin/runc -version v1.0.0-rc91 
INFO[2020-08-06T14:26:23+02:00] + set +o pipefail                            
INFO[2020-08-06T14:26:23+02:00] + local exec_name=/usr/local/bin/runc        
INFO[2020-08-06T14:26:23+02:00] + local exec_version_cmd=-version            
INFO[2020-08-06T14:26:23+02:00] + local version=v1.0.0-rc91                  
INFO[2020-08-06T14:26:23+02:00] + command -v /usr/local/bin/runc             
INFO[2020-08-06T14:26:23+02:00] + return 1                                   
INFO[2020-08-06T14:26:23+02:00] + curl -sSL https://github.com/opencontainers/runc/releases/download/v1.0.0-rc91/runc.amd64 -o runc 
INFO[2020-08-06T14:27:57+02:00] + chmod +x runc                              
INFO[2020-08-06T14:27:57+02:00] + sudo mv runc /usr/local/bin/               
INFO[2020-08-06T14:27:57+02:00] + install_containerd                         
INFO[2020-08-06T14:27:57+02:00] + _check_version /usr/local/bin/containerd --version v1.3.4
+ set +o pipefail
+ local exec_name=/usr/local/bin/containerd 
INFO[2020-08-06T14:27:57+02:00] + local exec_version_cmd=--version
+ local version=v1.3.4
+ command -v /usr/local/bin/containerd 
INFO[2020-08-06T14:27:57+02:00] + return 1                                   
INFO[2020-08-06T14:27:57+02:00] + local version=1.3.4
+ local dir=containerd-1.3.4 
INFO[2020-08-06T14:27:57+02:00] + curl -sSLO https://github.com/containerd/containerd/releases/download/v1.3.4/containerd-1.3.4.linux-amd64.tar.gz 
INFO[2020-08-06T14:34:05+02:00] + mkdir -p containerd-1.3.4                  
INFO[2020-08-06T14:34:05+02:00] + tar -zxvf containerd-1.3.4.linux-amd64.tar.gz -C containerd-1.3.4 
INFO[2020-08-06T14:34:05+02:00] bin/                                         
INFO[2020-08-06T14:34:05+02:00] bin/containerd-shim-runc-v2                  
INFO[2020-08-06T14:34:05+02:00] bin/containerd-stress                        
INFO[2020-08-06T14:34:06+02:00] bin/containerd                               
INFO[2020-08-06T14:34:06+02:00] bin/containerd-shim                          
INFO[2020-08-06T14:34:06+02:00] bin/ctr                                      
INFO[2020-08-06T14:34:06+02:00] bin/containerd-shim-runc-v1                  
INFO[2020-08-06T14:34:06+02:00] + chmod +x containerd-1.3.4/bin/containerd containerd-1.3.4/bin/containerd-shim containerd-1.3.4/bin/containerd-shim-runc-v1 containerd-1.3.4/bin/containerd-shim-runc-v2 containerd-1.3.4/bin/containerd-stress containerd-1.3.4/bin/ctr 
INFO[2020-08-06T14:34:06+02:00] + sudo mv containerd-1.3.4/bin/containerd containerd-1.3.4/bin/containerd-shim containerd-1.3.4/bin/containerd-shim-runc-v1 containerd-1.3.4/bin/containerd-shim-runc-v2 containerd-1.3.4/bin/containerd-stress containerd-1.3.4/bin/ctr /usr/local/bin/ 
INFO[2020-08-06T14:34:06+02:00] + curl -sSLO https://raw.githubusercontent.com/containerd/containerd/v1.3.4/containerd.service 
INFO[2020-08-06T14:34:07+02:00] + sudo groupadd containerd                   
INFO[2020-08-06T14:34:07+02:00] groupadd: group 'containerd' already exists  
INFO[2020-08-06T14:34:07+02:00] + cleanup
+ rm -rf /tmp/kubefire             
Usage:
  kubefire install [flags]

Flags:
      --force   force to download
  -h, --help    help for install

Global Flags:
      --log-level string   log level, options: [panic, fatal, error, warning, info, debug, trace] (default "info")
      --output string      output format, options: [default, json, yaml] (default "default")

FATA[2020-08-06T14:34:07+02:00] failed to run kubefire                        error="failed to run script (install-prerequisites.sh): exit status 9"

Oh, please use ignite install --force, because the install script got changed, so need to use --force to download the latest install script. this was probably due a slow connection(containerd took some time) now I’m on faster.

rugwirobaker on Aug 6, 2020

➜  ~ sudo ignite run innobead/kubefire-opensuse-leap:15.2 --ssh -d --kernel-image="innobead/kubefire-kernel-4.19.125-amd64:latest" --cpus=2 --memory=2GB --size=10GB
[sudo] password for rugwiro: 
INFO[0014] Created VM with ID "a51fff443f037ebf" and name "fragrant-firefly" 
INFO[0028] Networking is handled by "cni"               
INFO[0028] Started Firecracker VM "a51fff443f037ebf" in a container with ID "ignite-a51fff443f037ebf" 
INFO[0028] Waiting for the ssh daemon within the VM to start... 
INFO[0039] Waiting for the ssh daemon within the VM to start... 
➜  ~ sudo ignite ps --all
VM ID	IMAGE	KERNEL	SIZE	CPUS	MEMORY	CREATED	STATUS	IPS	PORTS	NAME

rugwirobaker on Aug 6, 2020

Old ignite version?

➜  ~ sudo ignite ps --all
VM ID	IMAGE	KERNEL	SIZE	CPUS	MEMORY	CREATED	STATUS	IPS	PORTS	NAME

➜  ~ sudo ignite inspect demo-master-1
Error: accepts 2 arg(s), received 1
Usage:
  ignite inspect <kind> <object> [flags]

Flags:
  -h, --help              help for inspect
  -o, --output string     Output the object in the specified format (default "json")
  -t, --template string   Format the output using the given Go template

Global Flags:
      --log-level loglevel      Specify the loglevel for the program (default info)
      --network-plugin plugin   Network plugin to use. Available options are: [cni docker-bridge] (default cni)
  -q, --quiet                   The quiet mode allows for machine-parsable output by printing only IDs
      --runtime runtime         Container runtime to use. Available options are: [docker containerd] (default containerd)

rugwirobaker on Aug 6, 2020

System info

➜  ~ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

➜  ~ kubefire version
Version: master
Build: a2e67de-20200806105119

➜  ~ kubefire info                                                        
CONTAINERD	IGNITE	CNI   	RUNC       
v1.3.4    	v0.7.1	v0.8.6	1.0.0-rc91

The logs

➜  ~ kubefire cluster create --log-level=debug --bootstrapper=kubeadm demo
INFO[2020-08-06T11:04:29+02:00] initializing cluster configuration            cluster=demo
DEBU[2020-08-06T11:04:29+02:00] &{Name:demo Bootstrapper:kubeadm Pubkey: Prikey: Image:innobead/kubefire-opensuse-leap:15.2 KernelImage:innobead/kubefire-kernel-4.19.125-amd64:latest KernelArgs:console=ttyS0 reboot=k panic=1 pci=off ip=dhcp security=apparmor apparmor=1 Admin:{Count:0 Memory:512MB Cpus:1 DiskSize:2GB Cluster:0xc000158780} Master:{Count:1 Memory:2GB Cpus:2 DiskSize:10GB Cluster:0xc000158780} Worker:{Count:0 Memory:2GB Cpus:2 DiskSize:10GB Cluster:0xc000158780} ExtraOptions:} 
DEBU[2020-08-06T11:04:29+02:00] getting cluster configurations                cluster=demo
INFO[2020-08-06T11:04:29+02:00] saving cluster configurations                 cluster=demo
INFO[2020-08-06T11:04:29+02:00] creating cluster                              cluster=demo
DEBU[2020-08-06T11:04:29+02:00] getting cluster configurations                cluster=demo
INFO[2020-08-06T11:04:29+02:00] creating master nodes of cluster              cluster=demo
INFO[2020-08-06T11:04:29+02:00] creating node                                 node=demo-master-1
INFO[2020-08-06T11:04:44+02:00] time="2020-08-06T11:04:44+02:00" level=info msg="Created VM with ID \"839dd9c7d67f38b7\" and name \"demo-master-1\"" 
INFO[2020-08-06T11:04:49+02:00] time="2020-08-06T11:04:49+02:00" level=info msg="Networking is handled by \"cni\"" 
INFO[2020-08-06T11:04:49+02:00] time="2020-08-06T11:04:49+02:00" level=info msg="Started Firecracker VM \"839dd9c7d67f38b7\" in a container with ID \"ignite-839dd9c7d67f38b7\"" 
DEBU[2020-08-06T11:04:49+02:00] getting cluster                               cluster=demo
DEBU[2020-08-06T11:04:49+02:00] getting cluster configurations                cluster=demo
DEBU[2020-08-06T11:04:49+02:00] listing nodes of cluster                      cluster=demo
DEBU[2020-08-06T11:04:49+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=~demo- -t {{.ObjectMeta.Name}}] 
INFO[2020-08-06T11:04:49+02:00] waiting nodes of cluster are running          cluster=demo
DEBU[2020-08-06T11:04:49+02:00] listing nodes of cluster                      cluster=demo
DEBU[2020-08-06T11:04:49+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=~demo- -t {{.ObjectMeta.Name}}] 
INFO[2020-08-06T11:04:49+02:00] initializing cluster                          cluster=demo
INFO[2020-08-06T11:04:49+02:00] waiting all nodes initialization finished    
DEBU[2020-08-06T11:04:49+02:00] getting node                                  node=demo-master-1
DEBU[2020-08-06T11:04:49+02:00] VM ID	IMAGE	KERNEL	SIZE	CPUS	MEMORY          
DEBU[2020-08-06T11:04:49+02:00] 	CREATED	STATUS	IPS	PORTS	NAME               
DEBU[2020-08-06T11:04:49+02:00] [sudo ignite ps --all -f {{.ObjectMeta.Name}}=demo-master-1 -t {{.ObjectMeta.Labels.cluster}}] 
Usage:
  kubefire cluster create [name] [flags]

Flags:
      --admin-count int        count of admin node
      --admin-cpu int          CPUs of admin node (default 1)
      --admin-memory string    memory of admin node (default "512MB")
      --admin-size string      disk size of admin node (default "2GB")
      --bootstrapper string    bootstrapper type, options: [kubeadm, skuba, k3s] (default "kubeadm")
      --extra-opts string      extra options (ex: key=value,...) for bootstrapper
      --force                  force to recreate
  -h, --help                   help for create
      --image string           rootfs container image (default "innobead/kubefire-opensuse-leap:15.2")
      --kernel-args string     kernel arguments (default "console=ttyS0 reboot=k panic=1 pci=off ip=dhcp security=apparmor apparmor=1")
      --kernel-image string    kernel container image (default "innobead/kubefire-kernel-4.19.125-amd64:latest")
      --master-count int       count of master node (default 1)
      --master-cpu int         CPUs of master node (default 2)
      --master-memory string   memory of master node (default "2GB")
      --master-size string     disk size of master node (default "10GB")
      --pubkey string          Public key
      --worker-count int       count of worker node
      --worker-cpu int         CPUs of worker node (default 2)
      --worker-memory string   memory of worker node (default "2GB")
      --worker-size string     disk size of worker node (default "10GB")

Global Flags:
      --log-level string   log level, options: [panic, fatal, error, warning, info, debug, trace] (default "info")
      --output string      output format, options: [default, json, yaml] (default "default")

FATA[2020-08-06T11:04:50+02:00] failed to run kubefire                        error="failed to deploy cluster (demo): demo-master-1 node available"

rugwirobaker on Aug 6, 2020

I have built from the latest version

➜  ~ kubefire version
Version: master
Build: 163a5e4-20200804121625

log output:

➜  ~ kubefire cluster create --bootstrapper=kubeadm hulk
INFO[2020-08-04T12:22:24+02:00] initializing cluster configuration            cluster=hulk
INFO[2020-08-04T12:22:24+02:00] saving cluster configurations                 cluster=hulk
INFO[2020-08-04T12:22:24+02:00] creating cluster (hulk)                      
INFO[2020-08-04T12:22:24+02:00] creating master nodes of cluster (hulk)      
INFO[2020-08-04T12:22:24+02:00] creating node (hulk-master-1)                
INFO[2020-08-04T12:22:45+02:00] time="2020-08-04T12:22:45+02:00" level=info msg="Imported OCI image \"innobead/kubefire-kernel-4.19.125-amd64:latest\" (154.4 MB) to kernel image with UID \"4d254a6cba0a3b7c\"" 
INFO[2020-08-04T12:22:56+02:00] time="2020-08-04T12:22:56+02:00" level=info msg="Created VM with ID \"a4c0faa8207ad771\" and name \"hulk-master-1\"" 
INFO[2020-08-04T12:23:07+02:00] time="2020-08-04T12:23:07+02:00" level=info msg="Networking is handled by \"cni\""
time="2020-08-04T12:23:07+02:00" level=info msg="Started Firecracker VM \"a4c0faa8207ad771\" in a container with ID \"ignite-a4c0faa8207ad771\"" 
INFO[2020-08-04T12:23:07+02:00] waiting nodes of cluster (hulk) are running  
INFO[2020-08-04T12:23:07+02:00] initializing cluster (hulk)                  
INFO[2020-08-04T12:23:07+02:00] waiting all nodes initialization finished    
INFO[2020-08-04T12:23:07+02:00] VM ID	IMAGE	KERNEL	SIZE	CPUS	MEMORY	CREATED	 
INFO[2020-08-04T12:23:07+02:00] STATUS	IPS	PORTS	NAME                        
Usage:
  kubefire cluster create [name] [flags]

Flags:
      --admin-count int        count of admin node
      --admin-cpu int          CPUs of admin node (default 1)
      --admin-memory string    memory of admin node (default "512MB")
      --admin-size string      disk size of admin node (default "2GB")
      --bootstrapper string    bootstrapper type, options: [kubeadm, skuba, k3s] (default "kubeadm")
      --extra-opts string      extra options (ex: key=value,...) for bootstrapper
      --force                  force to recreate
  -h, --help                   help for create
      --image string           rootfs container image (default "innobead/kubefire-opensuse-leap:15.2")
      --kernel-args string     kernel arguments (default "console=ttyS0 reboot=k panic=1 pci=off ip=dhcp security=apparmor apparmor=1")
      --kernel-image string    kernel container image (default "innobead/kubefire-kernel-4.19.125-amd64:latest")
      --master-count int       count of master node (default 1)
      --master-cpu int         CPUs of master node (default 2)
      --master-memory string   memory of master node (default "2GB")
      --master-size string     disk size of master node (default "10GB")
      --pubkey string          Public key
      --worker-count int       count of worker node
      --worker-cpu int         CPUs of worker node (default 2)
      --worker-memory string   memory of worker node (default "2GB")
      --worker-size string     disk size of worker node (default "10GB")

Global Flags:
      --log-level string   log level, options: [panic, fatal, error, warning, info, debug, trace] (default "info")
      --output string      output format, options: [default, json, yaml] (default "default")

FATA[2020-08-04T12:23:07+02:00] failed to run kubefire                        error="failed to deploy cluster (hulk): hulk-master-1 node available"

Thanks! But, (please help pull the latest code again af3ebf1)

Please run kubefire cluster create with --log-level=debug.
Also, run sudo ignite ps and sudo ignite inspect <vm name like hulk-master-1 in your case> to provide the output log as I mentioned before.

innobead on Aug 4, 2020

Tried a trace and it seems I can’t ssh into the VM for some reason.

➜  ~ kubefire cluster create --log-level trace --bootstrapper=kubeadm demo
INFO[2020-08-03T14:47:43+02:00] initializing cluster configuration            cluster=demo
DEBU[2020-08-03T14:47:43+02:00] &{Name:demo Bootstrapper:kubeadm Pubkey: Prikey: Image:innobead/kubefire-opensuse-leap:15.2 KernelImage:innobead/kubefire-kernel-5.4.43-amd64:latest KernelArgs:console=ttyS0 reboot=k panic=1 pci=off ip=dhcp security=apparmor apparmor=1 Admin:{Count:0 Memory:512MB Cpus:1 DiskSize:2GB Cluster:<nil>} Master:{Count:1 Memory:2GB Cpus:2 DiskSize:10GB Cluster:<nil>} Worker:{Count:0 Memory:2GB Cpus:2 DiskSize:10GB Cluster:<nil>} ExtraOptions:} 
DEBU[2020-08-03T14:47:43+02:00] getting cluster configurations                cluster=demo

...........

TRAC[2020-08-03T14:47:59+02:00] dial tcp :22: connect: connection refused
github.com/innobead/kubefire/pkg/util/ssh.(*Client).Init
	/home/rugwiro/workspace/kubefire/pkg/util/ssh/commander.go:63
github.com/innobead/kubefire/pkg/util/ssh.NewClient
	/home/rugwiro/workspace/kubefire/pkg/util/ssh/commander.go:53
github.com/innobead/kubefire/pkg/bootstrap.(*KubeadmBootstrapper).bootstrap
	/home/rugwiro/workspace/kubefire/pkg/bootstrap/kubeadm.go:155
github.com/innobead/kubefire/pkg/bootstrap.(*KubeadmBootstrapper).Deploy
	/home/rugwiro/workspace/kubefire/pkg/bootstrap/kubeadm.go:52
github.com/innobead/kubefire/cmd/kubefire/cmd/cluster.glob..func3
	/home/rugwiro/workspace/kubefire/cmd/kubefire/cmd/cluster/create.go:76
github.com/spf13/cobra.(*Command).execute
	/home/rugwiro/workspace/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842
github.com/spf13/cobra.(*Command).ExecuteC
	/home/rugwiro/workspace/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950
github.com/spf13/cobra.(*Command).Execute
	/home/rugwiro/workspace/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main
	/home/rugwiro/workspace/kubefire/cmd/kubefire/kubefire.go:48
runtime.main
	/usr/local/go/src/runtime/proc.go:203
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1373

I did notice a Nil in that output that seems like Bootstrap configuration input ({Count:0 Memory:512MB Cpus:1 DiskSize:2GB Cluster:<nil>}).

The nil should be fine, it’s more about the field not filling. This is a bug but it’s not related to the issue you encountered.

innobead on Aug 3, 2020