kubernetes: kubelet (1.27.1) fails to mount a hostPath volume for a static pod

What happened?

This is a bit hard to reproduce, as it happens in a very specific setup, the problems only happens with kubelet 1.27.1, downgrading to 1.26.4 fixes the problem.

In a single-node cluster controlplane pods run as static pods, Rook/Ceph is deployed, and some PVs/PVCs backed by Ceph.

After a unclean shutdown related to some PVs backed by Ceph not being unmounted (this is not a Kubernetes), the kubelet which comes up after a reboot doesn’t start controlplane static pods, which essentially blocks every other pod from starting (as API server is done).

After a reboot, kubelet is running, but it keeps reporting an error about static pods like:

172.20.0.2: {"ts":1683110056228.5916,"caller":"kubelet/kubelet.go:1875","msg":"Unable to attach or mount volumes for pod; skipping pod","pod":{"name":"kube-apiserver-talos-default-controlplane-1","namespace":"kube-system"},"err":"unmounted volumes=[audit secrets config], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition"}

All of the volumes are hostPath volumes in a static pod:

        volumes:
            - hostPath:
                path: /system/secrets/kubernetes/kube-apiserver
              name: secrets
            - hostPath:
                path: /system/config/kubernetes/kube-apiserver
              name: config
            - hostPath:
                path: /var/log/audit/kube
              name: audit

The kubelet is stuck in this state “forever”, I tested at least for 8 hours.

If I downgrade the kubelet (without machine reboot) to 1.26.4, it immediately starts the static pods bringing up the node online.

If I try same scenario with reboot with kubelet 1.26.4, the problem doesn’t occur either.

What did you expect to happen?

Kubelet should run static pods after a reboot.

How can we reproduce it (as minimally and precisely as possible)?

It’s a bit complicated set of workloads, but it’s 100% reproducible. I could provide more logs if needed.

Anything else we need to know?

/sig node

Kubernetes version

1.27.1

Cloud provider

OS version

Talos Linux v1.5.0-alpha

Install tools

Container runtime (CRI) and version (if applicable)

containerd 1.6.20

Related plugins (CNI, CSI, …) and versions (if applicable)