kind: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"

Dear KinD community,

I’d like to readdress (from here, btw @BenTheElder thanks a lot for the feedback) this error:

ERROR: failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"

What happened:

  • creating the 1st cluster runs fine
  • creating a 2nd cluster fails

What you expected to happen:

Have it all running fine: the second cluster be created as well.


How to reproduce it (as minimally and precisely as possible):

Starting from a clean state (ok, not purely clean; deleted any kind cluster), I create a first multi-node cluster using this config:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster

nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker

and the cluster is successfully created.

Now, trying to create a second multi-node cluster with an ingress (as per this nice kind doc) based on this config:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster

nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
- role: worker
- role: worker

the cluster creation fails:

❯ kind create cluster --name istioinaction --config istioinaction_cluster.yaml --retain; kind export logs --name=istioinaction
Creating cluster "istioinaction" ...
 ✓ Ensuring node image (kindest/node:v1.25.2) 🖼
 ✗ Preparing nodes 📦 📦 📦  
ERROR: failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"
Exporting logs for cluster "istioinaction" to:
/tmp/297015354
ERROR: [command "docker exec --privileged istioinaction-worker2 sh -c 'tar --hard-dereference -C /var/log/ -chf - . || (r=$?; [ $r -eq 1 ] || exit $r)'" failed with error: exit status 1, [[command "docker exec --privileged istioinaction-worker2 cat /kind/version" failed with error: exit status 1, command "docker exec --privileged istioinaction-worker2 journalctl --no-pager" failed with error: exit status 1, command "docker exec --privileged istioinaction-worker2 journalctl --no-pager -u containerd.service" failed with error: exit status 1, command "docker exec --privileged istioinaction-worker2 journalctl --no-pager -u kubelet.service" failed with error: exit status 1, command "docker exec --privileged istioinaction-worker2 crictl images" failed with error: exit status 1], command "docker exec --privileged istioinaction-control-plane crictl images" failed with error: exit status 1, command "docker exec --privileged istioinaction-worker crictl images" failed with error: exit status 1]]
❯

The logs export (to that /tmp/297015354 folder) collected these files and folders: docker-info.txt istioinaction-control-plane istioinaction-worker istioinaction-worker2 kind-version.txt.

A brief look for erros in it shows different details:

…/297015354 ❯ grep -Ri error *
istioinaction-control-plane/journal.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.832102115Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.19.0-76051900-generic\\n\"): skip plugin" type=io.containerd.snapshotter.v1
istioinaction-control-plane/journal.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.832286765Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
istioinaction-control-plane/journal.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.832314360Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
istioinaction-control-plane/journal.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.833018086Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
istioinaction-control-plane/journal.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.833153084Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
istioinaction-control-plane/journal.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.833932795Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
istioinaction-control-plane/journal.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.833954489Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
istioinaction-control-plane/journal.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.834239050Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false BaseRuntimeSpec:/etc/containerd/cri-base.json NetworkPluginConfDir: NetworkPluginMaxConfNum:0} test-handler:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false BaseRuntimeSpec:/etc/containerd/cri-base.json NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:true IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[k8s.gcr.io:{Endpoints:[https://registry.k8s.io https://k8s.gcr.io]}] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:registry.k8s.io/pause:3.7 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
istioinaction-control-plane/journal.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.834481935Z" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to create cni conf monitor for default: failed to create fsnotify watcher: too many open files"
istioinaction-control-plane/containerd.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.832102115Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.19.0-76051900-generic\\n\"): skip plugin" type=io.containerd.snapshotter.v1
istioinaction-control-plane/containerd.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.832286765Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
istioinaction-control-plane/containerd.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.832314360Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
istioinaction-control-plane/containerd.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.833018086Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
istioinaction-control-plane/containerd.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.833153084Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
istioinaction-control-plane/containerd.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.833932795Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
istioinaction-control-plane/containerd.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.833954489Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
istioinaction-control-plane/containerd.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.834239050Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false BaseRuntimeSpec:/etc/containerd/cri-base.json NetworkPluginConfDir: NetworkPluginMaxConfNum:0} test-handler:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false BaseRuntimeSpec:/etc/containerd/cri-base.json NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:true IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[k8s.gcr.io:{Endpoints:[https://registry.k8s.io https://k8s.gcr.io]}] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:registry.k8s.io/pause:3.7 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
istioinaction-control-plane/containerd.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.834481935Z" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to create cni conf monitor for default: failed to create fsnotify watcher: too many open files"
istioinaction-control-plane/inspect.json:            "Error": "",
istioinaction-control-plane/images.log:E1019 06:54:40.612221     140 remote_image.go:125] "ListImages with filter from image service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService" filter="&ImageFilter{Image:&ImageSpec{Image:,Annotations:map[string]string{},},}"
istioinaction-control-plane/images.log:time="2022-10-19T06:54:40Z" level=fatal msg="listing images: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"
istioinaction-worker/journal.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.478730633Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.19.0-76051900-generic\\n\"): skip plugin" type=io.containerd.snapshotter.v1
istioinaction-worker/journal.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.479064829Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
istioinaction-worker/journal.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.479120664Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
istioinaction-worker/journal.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.479515619Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
istioinaction-worker/journal.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.479786921Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
istioinaction-worker/journal.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.481060666Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
istioinaction-worker/journal.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.481106352Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
istioinaction-worker/journal.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.481498498Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false BaseRuntimeSpec:/etc/containerd/cri-base.json NetworkPluginConfDir: NetworkPluginMaxConfNum:0} test-handler:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false BaseRuntimeSpec:/etc/containerd/cri-base.json NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:true IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[k8s.gcr.io:{Endpoints:[https://registry.k8s.io https://k8s.gcr.io]}] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:registry.k8s.io/pause:3.7 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
istioinaction-worker/journal.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.481860970Z" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to create cni conf monitor for default: failed to create fsnotify watcher: too many open files"
istioinaction-worker/containerd.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.478730633Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.19.0-76051900-generic\\n\"): skip plugin" type=io.containerd.snapshotter.v1
istioinaction-worker/containerd.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.479064829Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
istioinaction-worker/containerd.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.479120664Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
istioinaction-worker/containerd.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.479515619Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
istioinaction-worker/containerd.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.479786921Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
istioinaction-worker/containerd.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.481060666Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
istioinaction-worker/containerd.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.481106352Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
istioinaction-worker/containerd.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.481498498Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false BaseRuntimeSpec:/etc/containerd/cri-base.json NetworkPluginConfDir: NetworkPluginMaxConfNum:0} test-handler:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false BaseRuntimeSpec:/etc/containerd/cri-base.json NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:true IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[k8s.gcr.io:{Endpoints:[https://registry.k8s.io https://k8s.gcr.io]}] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:registry.k8s.io/pause:3.7 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
istioinaction-worker/containerd.log:Oct 19 06:54:39 istioinaction-worker containerd[105]: time="2022-10-19T06:54:39.481860970Z" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to create cni conf monitor for default: failed to create fsnotify watcher: too many open files"
istioinaction-worker/inspect.json:            "Error": "",
istioinaction-worker/images.log:E1019 06:54:40.606981     126 remote_image.go:125] "ListImages with filter from image service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService" filter="&ImageFilter{Image:&ImageSpec{Image:,Annotations:map[string]string{},},}"
istioinaction-worker/images.log:time="2022-10-19T06:54:40Z" level=fatal msg="listing images: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"
istioinaction-worker2/journal.log:Error response from daemon: Container 8838abd1202c757ad92140b2d59fe13a61372c8a3c3b6a9f5329ba51b4c78d5a is not running
istioinaction-worker2/containerd.log:Error response from daemon: Container 8838abd1202c757ad92140b2d59fe13a61372c8a3c3b6a9f5329ba51b4c78d5a is not running
istioinaction-worker2/kubelet.log:Error response from daemon: Container 8838abd1202c757ad92140b2d59fe13a61372c8a3c3b6a9f5329ba51b4c78d5a is not running
istioinaction-worker2/inspect.json:            "Error": "",
istioinaction-worker2/images.log:Error response from daemon: Container 8838abd1202c757ad92140b2d59fe13a61372c8a3c3b6a9f5329ba51b4c78d5a is not running
istioinaction-worker2/kubernetes-version.txt:Error response from daemon: Container 8838abd1202c757ad92140b2d59fe13a61372c8a3c3b6a9f5329ba51b4c78d5a is not running
…/297015354 ❯

I’ll try to address such error and get back here with updates:

istioinaction-control-plane/journal.log:Oct 19 06:54:39 istioinaction-control-plane containerd[106]: time="2022-10-19T06:54:39.834481935Z" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to create cni conf monitor for default: failed to create fsnotify watcher: too many open files"

Anything else we need to know?:

Interesting is that it happens in one of my two systems (a workstation and laptop), both running the same up-to-date Linux distro, with the same versions of the tools (captured in the Environment section below).


Environment:

  • kind version: (use kind version): v0.16.0
  • Kubernetes version: (use kubectl version): client v1.25.3, server v1.25.2
  • Docker version: (use docker info): 20.10.12
  • OS (e.g. from /etc/os-release): Pop!_OS 22.04 LTS (nothing custom, all stock and up to date, running kernel 5.19.0-76051900-generic)

Thanks



Update

Tried again after increasing the max number of open files, at least to eliminate one type of issues.

  • Increased the limit (as per this article)
    • I had to restart the OS, since just restarting the terminal or even relogin didn’t reflect the updated value
    • After restart and relogin, ulimit -n now returns 4096 (instead of 1024).
  • Removed again both kind clusters and recreate them
    • As before, first creation succedded, the second creation failed, see below the details
…/istioinaction_cluster ❯ kind create cluster --name istioinaction --config istioinaction_cluster.yaml --retain; kind export logs --name=istioinaction
Creating cluster "istioinaction" ...
 ✓ Ensuring node image (kindest/node:v1.25.2) 🖼
 ✓ Preparing nodes 📦 📦 📦  
 ✓ Writing configuration 📜 
 ✗ Starting control-plane 🕹️ 
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged istioinaction-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
Command Output: I1019 08:33:46.378882     138 initconfiguration.go:254] loading configuration from "/kind/kubeadm.conf"
W1019 08:33:46.380591     138 initconfiguration.go:331] [config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta3, Kind=JoinConfiguration
[init] Using Kubernetes version: v1.25.2
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I1019 08:33:46.385114     138 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I1019 08:33:46.764714     138 certs.go:522] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [istioinaction-control-plane kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 172.22.0.6 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
I1019 08:33:47.032475     138 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I1019 08:33:47.115572     138 certs.go:522] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I1019 08:33:47.229402     138 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I1019 08:33:47.315526     138 certs.go:522] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [istioinaction-control-plane localhost] and IPs [172.22.0.6 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [istioinaction-control-plane localhost] and IPs [172.22.0.6 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I1019 08:33:48.183185     138 certs.go:78] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I1019 08:33:48.270474     138 kubeconfig.go:103] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I1019 08:33:48.365006     138 kubeconfig.go:103] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I1019 08:33:48.609181     138 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I1019 08:33:48.680612     138 kubeconfig.go:103] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I1019 08:33:48.797441     138 kubelet.go:66] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I1019 08:33:48.957010     138 manifests.go:99] [control-plane] getting StaticPodSpecs
I1019 08:33:48.957892     138 certs.go:522] validating certificate period for CA certificate
I1019 08:33:48.958200     138 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I1019 08:33:48.958243     138 manifests.go:125] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I1019 08:33:48.958267     138 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I1019 08:33:48.958291     138 manifests.go:125] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I1019 08:33:48.958316     138 manifests.go:125] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I1019 08:33:48.963858     138 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
I1019 08:33:48.963888     138 manifests.go:99] [control-plane] getting StaticPodSpecs
I1019 08:33:48.964318     138 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I1019 08:33:48.964339     138 manifests.go:125] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I1019 08:33:48.964361     138 manifests.go:125] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I1019 08:33:48.964365     138 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I1019 08:33:48.964368     138 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I1019 08:33:48.964372     138 manifests.go:125] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I1019 08:33:48.964376     138 manifests.go:125] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I1019 08:33:48.965003     138 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
I1019 08:33:48.965013     138 manifests.go:99] [control-plane] getting StaticPodSpecs
I1019 08:33:48.965185     138 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I1019 08:33:48.965549     138 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I1019 08:33:48.966115     138 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I1019 08:33:48.966125     138 waitcontrolplane.go:83] [wait-control-plane] Waiting for the API server to be healthy
I1019 08:33:48.966576     138 loader.go:374] Config loaded from file:  /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I1019 08:33:48.967526     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:33:49.470031     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:49.971225     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:50.470105     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:50.970958     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:51.470700     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:51.971168     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:52.468094     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:33:52.969972     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:53.470912     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:53.970151     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:54.469538     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:54.971224     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:55.468324     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:33:55.971215     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:56.469818     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:56.970654     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:57.470793     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:33:57.970709     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:58.469536     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:33:58.971352     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:59.471438     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:33:59.970232     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:00.468352     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:00.971337     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:01.471119     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:01.971130     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:02.470937     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:02.970944     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:03.468697     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:03.970635     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:04.471243     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:04.970022     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:05.470170     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:05.971090     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:06.469021     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:06.971437     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:07.470188     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:07.969655     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:08.470585     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:08.971150     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:09.470948     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:09.968512     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:10.471091     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:10.970950     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:11.470772     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:11.970641     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:12.471121     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:12.970796     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:13.469657     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:13.970755     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:14.469647     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:14.971277     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:15.468415     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:15.971184     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:16.470100     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:16.971240     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:17.470118     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:17.971169     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:18.470146     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:18.971121     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:19.469232     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:19.970861     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:20.470663     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:20.971253     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:21.470264     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:21.971420     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:22.471166     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:22.970702     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:23.471066     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:23.969758     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:24.470674     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:24.969647     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:25.470281     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:25.971221     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:26.471081     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:26.969568     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:27.471201     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:27.970608     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:28.470529     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] Initial timeout of 40s passed.
I1019 08:34:28.970462     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I1019 08:34:29.470694     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:29.970146     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:30.471298     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:30.970661     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:31.468498     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:31.971122     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:32.471069     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:32.971823     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:33.469308     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:33.971161     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I1019 08:34:34.469059     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:34.970983     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:35.470859     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:35.969190     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:36.471134     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:36.971332     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:37.471213     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:37.969178     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:38.470475     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:38.971128     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:39.471124     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:39.969997     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:40.470927     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:40.969726     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:41.469485     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:41.969559     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:42.471258     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:42.970661     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:43.471101     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:43.971038     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I1019 08:34:44.469265     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:44.971304     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:45.469225     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:45.971083     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:46.471025     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:46.970943     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:47.470816     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:47.970701     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:48.469560     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:48.969713     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:49.471189     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:49.970883     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:50.470915     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:50.969179     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:51.470295     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:51.971203     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:52.470992     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:52.969357     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:53.470621     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:34:53.971002     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:54.471050     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:54.970680     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:55.469107     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:55.970938     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:56.469391     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:56.971271     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:57.471240     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:57.968990     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:58.470977     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:58.969104     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:59.470192     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:34:59.970201     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:00.471134     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:00.970031     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:01.471235     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:01.971361     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:02.471344     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:02.971093     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:03.471077     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:03.969673     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I1019 08:35:04.470295     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:04.971075     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:05.470176     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:05.970076     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:06.469154     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:06.971025     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:07.470573     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:07.971373     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:08.471293     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:08.971217     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:09.470973     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:09.970868     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:10.470761     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:10.969784     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:11.468420     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:11.970965     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:12.471031     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:12.969764     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:13.470599     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:13.970545     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:14.471050     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:14.969166     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:15.470137     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:15.969122     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:16.469474     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:16.970247     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:17.470161     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:17.971104     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:18.471212     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:18.971197     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:19.470211     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:19.968877     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:20.470799     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:20.968647     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:21.470791     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:21.969206     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:22.469942     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:22.970075     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:23.471114     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:23.969145     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:24.471179     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:24.969603     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:25.471045     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:25.970841     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:26.468113     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:26.970162     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:27.469747     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:27.971270     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:28.468833     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:28.970914     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:29.470726     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:29.969398     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:30.471243     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:30.971009     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:31.470992     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:31.970269     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:32.470831     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:32.969406     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:33.471313     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:33.971228     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:34.471121     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:34.970751     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:35.470733     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:35.969370     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:36.471368     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:36.970203     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:37.471144     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:37.971011     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:38.470719     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1019 08:35:38.969197     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:39.469199     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:39.970471     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:40.471170     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:40.971100     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:41.470752     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:41.970822     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:42.470876     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:42.970803     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:43.470628     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1019 08:35:43.971096     138 round_trippers.go:553] GET https://istioinaction-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
	- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
	Once you have found the failing container, you can inspect its logs with:
	- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
couldn't initialize a Kubernetes cluster
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase
	cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:108
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
	cmd/kubeadm/app/cmd/init.go:154
github.com/spf13/cobra.(*Command).execute
	vendor/github.com/spf13/cobra/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
	vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run
	cmd/kubeadm/app/kubeadm.go:50
main.main
	cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:250
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1594
error execution phase wait-control-plane
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:421
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
	cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.newCmdInit.func1
	cmd/kubeadm/app/cmd/init.go:154
github.com/spf13/cobra.(*Command).execute
	vendor/github.com/spf13/cobra/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
	vendor/github.com/spf13/cobra/command.go:902
k8s.io/kubernetes/cmd/kubeadm/app.Run
	cmd/kubeadm/app/kubeadm.go:50
main.main
	cmd/kubeadm/kubeadm.go:25
runtime.main
	/usr/local/go/src/runtime/proc.go:250
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1594
Exporting logs for cluster "istioinaction" to:
/tmp/1990444523
…/istioinaction_cluster took 2m1s❯

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (7 by maintainers)

Most upvoted comments

Also running into the same issue on Mac M1:

my command:

kind create cluster --config local-k8s-tests/bootstrap-k8s/kind.yaml --retain

the error:

at 14:22:23 ❯ kind create cluster --config local-k8s-tests/bootstrap-k8s/kind.yaml --retain #; kind export logs --name jimconn; kind delete cluster --name jimconn
Creating cluster "jimconn" ...
 ✓ Ensuring node image (kindest/node:v1.23.13) 🖼
 ✗ Preparing nodes 📦
ERROR: failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"

my config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: jimconn
featureGates: {}
networking:
  ipFamily: ipv4
nodes:
  - role: control-plane
    image: kindest/node:v1.23.13@sha256:e7968cda1b4ff790d5b0b5b0c29bda0404cdb825fd939fe50fd5accc43e3a730

Interesting items of note: I updated my config to run a 1.23 cluster vs a 1.21 cluster. A 1.21 cluster comes up fine. 1.23 and 1.25 cluster images fail with the same error as noted above.

image was found here

kind version: v0.14.0 go1.18.2 darwin/arm64

kind-issue-2972-jimconn-log-export.tar.gz

Ah geez. I just realized that I was using the amd64 image instead of the arm64 image, which is easy to do. I missed the drop-down menu in the docker link above. Selecting the correct version and updating the sha in my config solved my issue.

Rather than deleting this post, I’ll leave it in case anyone else runs into the same issue I did. The proper link I should have used is here.

+3
notjames on Dec 20, 2022

Reporting the same issue on Mac M1 w/ DOCKER_DEFAULT_PLATFORM=linux/amd64

#2718

TLDR that’s not supported, the platform needs to match to run Kubernetes. We need to determine the best option to handle this still.

FTR, mine wasn’t the same as @dxps because I wasn’t trying to use DOCKER_DEFAULT_PLATFORM. Mine was just user error.

+1
notjames on Dec 21, 2022
Read more comments on GitHub
← kind: coredns stuck at ContainerCreating
kind: CrashLoopBackOff Error in kube-proxy with kernel versions 5.12.2.arch1-1 and 5.10.35-1-lts →