talos: Reboot and reset both hang when k8s API endpoint cannot be reached during bootstrap

Bug Report

Description

I’m working on a greenfield Talos deployment. I built out the config files, but typo’d the IP address of the cluster endpoint. I then applied the config and attempted to bootstrap one of the nodes. I noticed the incorrect cluster endpoint, and attempted to reset the node:

$ talosctl reset --talosconfig talosconfig -e 10.2.1.1 -n 10.2.1.1 --reboot --graceful=false --debug
◳ ◲ watching nodes: [10.2.1.1]
    * 10.2.1.1: actor ID: aa1f7b97-d422-4842-8b17-44e53b3046d3
Signal received, aborting, press Ctrl+C once again to abort immediately...
console logs for nodes ["10.2.1.1"]:
10.2.1.1: user: warning: [2024-10-27T20:27:11.106848015Z]: [talos] reset request received. actorID: aa1f7b97-d422-4842-8b17-44e53b3046d3
10.2.1.1: user: warning: [2024-10-27T20:27:11.106963015Z]: [talos] reset failed: locked

This fails with a reset failed: locked message. I then attempted to reboot the node, in hopes that this would give me the opportunity to reset without a reinstall:

$ talosctl reboot --talosconfig talosconfig -e 10.2.1.1 -n 10.2.1.1 --debug
◲ watching nodes: [10.2.1.1]
◱ watching nodes: [10.2.1.1]
    * 10.2.1.1: task: stopAllPods action: START
console logs for nodes ["10.2.1.1"]:
10.2.1.1: user: warning: [2024-10-27T20:29:42.485414055Z]: [talos] reboot via API received. actor id: 4070e73e-a662-442b-9336-110793384ef3
10.2.1.1: user: warning: [2024-10-27T20:29:42.485627055Z]: [talos] task stopAllPods (1/1): failed: context canceled
10.2.1.1: user: warning: [2024-10-27T20:29:42.485693055Z]: [talos] phase cleanup (1/11): failed
10.2.1.1: user: warning: [2024-10-27T20:29:42.485716055Z]: [talos] reboot sequence: failed
10.2.1.1: user: warning: [2024-10-27T20:29:42.485741055Z]: [talos] reboot failed: error running phase 1 in reboot sequence: task 1/1: failed, context canceled
10.2.1.1: user: warning: [2024-10-27T20:29:42.485779055Z]: [talos] reboot sequence: 11 phase(s)
10.2.1.1: user: warning: [2024-10-27T20:29:42.485792055Z]: [talos] phase cleanup (1/11): 1 tasks(s)
10.2.1.1: user: warning: [2024-10-27T20:29:42.485828055Z]: [talos] task stopAllPods (1/1): starting
10.2.1.1: user: warning: [2024-10-27T20:29:42.485845055Z]: [talos] task stopAllPods (1/1): waiting for kubelet lifecycle finalizers
10.2.1.1: user: warning: [2024-10-27T20:29:42.485921055Z]: [talos] task stopAllPods (1/1): failed waiting for kubelet lifecycle finalizers: resource KubeletLifecycles.kubernetes.talos.dev(k8s/kubelet@undefined) doesn't exist
10.2.1.1: user: warning: [2024-10-27T20:29:42.485936055Z]: [talos] task stopAllPods (1/1): shutting down kubelet gracefully
10.2.1.1: user: warning: [2024-10-27T20:29:52.948892055Z]: [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://10.35.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\": dial tcp 10.35.0.1:6443: i/o timeout"}

This fails in a similar manner. The node never reboots, because the reboot waits for pods to stop, which requires communicating with the unavailable k8s API endpoint.

Can a --force flag be added to the reset and reboot commands that does not wait for a graceful pod stop (or anything else really), and immediately performs the action?

Logs

See above

Environment

  • Talos version: v1.8.1
  • Kubernetes version: v1.31.1
  • Platform: metal/amd64

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 26 (24 by maintainers)

Most upvoted comments

Node CSRs don’t matter here at all, as this is not api-server -> kubelet communication.

Probably something else was going on there.