I’ve been having a really hard time getting Talos/kubernetes to run properly. This is my first time using pxe boot, so most of that was my fault, but I’m stumped on this one. I have proxmox vms, esx vms, and lenovo hardware that I’m trying to get ingested into this system. I decided to start with the proxmox vms, since I figured that was fairly standard, and I have the serial console output below.

The issue appears to be an untrusted certificate authority, invalid cert, or some other PKI issue at the very least. I’m simply following the getting started guide, and I’ve also walked through the bootstrap guide. I read an issue here that adding my endpoint dns record to the machine.certSANs might fix that, but as I entered it during the bootstrap process as my endpoint, it was already there. The x509 errors continue until a timeout is hit, then it reboots.

Cluster setup

talosctl cluster create --kubernetes-version 1.26.1 --talos-version v1.3.2 --nameservers=10.100.1.1,10.100.50.100 --name sidero-demo -p 69:69/udp,8081:8081/tcp,51821:51821/udp --workers 0 --endpoint talos.mimir-tech.org

kubectl taint node sidero-demo-controlplane-1 node-role.kubernetes.io/control-plane:NoSchedule-

export SIDERO_CONTROLLER_MANAGER_HOST_NETWORK=true
export SIDERO_CONTROLLER_MANAGER_API_ENDPOINT=talos.mimir-tech.org
export SIDERO_CONTROLLER_MANAGER_SIDEROLINK_ENDPOINT=talos.mimir-tech.org

clusterctl init -b talos -c talos -i sidero

export CONTROL_PLANE_SERVERCLASS=masters
export WORKER_SERVERCLASS=workers
export TALOS_VERSION=v1.3.2
export KUBERNETES_VERSION=v1.26.1
export CONTROL_PLANE_PORT=6443
export CONTROL_PLANE_ENDPOINT=talos.mimir-tech.org

clusterctl generate cluster cp01 -i sidero > cp01.yaml

kubectl get talosconfig -n sidero-system -l cluster.x-k8s.io/cluster-name=cp01 -o yaml -o jsonpath='{.items[0].status.talosConfig}' > cp01-talosconfig.yaml

Serial output from proxmox efi enabled vm.

[    1.219793] ------------[ cut here ]------------
[    1.220593] x86/mm: Found insecure W+X mapping at address 0xffffffffff620000
[    1.221611] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:246 note_page+0x642/0x6b0
[    1.222737] Modules linked in:
[    1.223446] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.15.83-talos #1
[    1.224398] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
[    1.225491] RIP: 0010:note_page+0x642/0x6b0
[    1.226305] Code: 85 49 ff ff ff e9 8d fe ff ff 80 3d d7 63 87 03 00 0f 85 6e fa ff ff 48 c7 c7 68 65 54 aa c6 05 c3 63 87 03 01 e8 fc a5 9b 01 <0f> 0b e9 54 fa ff ff 48 c7 c6 bf 66 54 aa 4c 89 ff e8 78 77 2a 00
[    1.228701] RSP: 0000:ffffbe9bc001fce8 EFLAGS: 00010282
[    1.229600] RAX: 0000000000000000 RBX: ffffbe9bc001fea0 RCX: ffffffffab563048
[    1.230643] RDX: 0000000000000000 RSI: 00000000ffffdfff RDI: ffffffffab483000
[    1.231678] RBP: 0000000000000006 R08: 0000000000000000 R09: ffffbe9bc001fb20
[    1.232710] R10: ffffbe9bc001fb18 R11: 0000000000000003 R12: 0000000000000004
[    1.233739] R13: ffffffffff621000 R14: 0000000000000616 R15: 0000000000000000
[    1.234750] FS:  0000000000000000(0000) GS:ffff9fb1ff700000(0000) knlGS:0000000000000000
[    1.235819] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    1.236712] CR2: 0000000000000000 CR3: 000000007ce14000 CR4: 00000000000006e0
[    1.237749] Call Trace:
[    1.238455]  <TASK>
[    1.239127]  ? sysvec_apic_timer_interrupt+0xa/0x90
[    1.240011]  ptdump_pte_entry+0x57/0x70
[    1.240811]  walk_pgd_range+0x46e/0x6c0
[    1.241673]  walk_page_range_novma+0x5e/0x90
[    1.242495]  ptdump_walk_pgd+0x42/0xb0
[    1.243341]  ptdump_walk_pgd_level_core+0xc6/0xf0
[    1.244255]  ? ptdump_walk_pgd_level_debugfs+0x40/0x40
[    1.245152]  ? hugetlb_get_unmapped_area+0x2e0/0x2e0
[    1.245987]  ? rest_init+0xc0/0xc0
[    1.246712]  ? rest_init+0xc0/0xc0
[    1.247430]  kernel_init+0x3d/0x120
[    1.248148]  ret_from_fork+0x22/0x30
[    1.248870]  </TASK>
[    1.249479] ---[ end trace 6d183cb346bb1ae2 ]---
[    1.250296] x86/mm: Checked W+X mappings: FAILED, 2 W+X pages found.
[    1.251221] x86/mm: Checking user space page tables
[    1.252179] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.253105] Run /init as init process
[    1.471046] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
[    3.198155] random: crng init done
[    3.202384] [talos] [initramfs] booting Talos v1.3.0
[    3.203278] [talos] [initramfs] mounting the rootfs
[    3.204209] loop0: detected capacity change from 0 to 95616
[    3.238959] [talos] [initramfs] bind mounting /lib/firmware
[    3.240660] [talos] [initramfs] entering the rootfs
[    3.241585] [talos] [initramfs] moving mounts to the new rootfs
[    3.242560] [talos] [initramfs] changing working directory into /root
[    3.243480] [talos] [initramfs] moving /root to /
[    3.244266] [talos] [initramfs] changing root directory
[    3.245078] [talos] [initramfs] cleaning up initramfs
[    3.246032] [talos] [initramfs] executing /sbin/init
[    5.795150] [talos] task setupLogger (1/1): done, 499.034µs
[    5.796139] [talos] phase logger (1/7): done, 1.876442ms
[    5.797092] [talos] phase systemRequirements (2/7): 7 tasks(s)
[    5.798147] [talos] task dropCapabilities (7/7): starting
[    5.811052] [talos] task enforceKSPPRequirements (1/7): starting
[    5.819795] [talos] task setupSystemDirectory (2/7): starting
[    5.820837] [talos] task setupSystemDirectory (2/7): done, 8.74232ms
[    5.821928] [talos] task mountBPFFS (3/7): starting
[    5.822866] [talos] task mountCgroups (4/7): starting
[    5.823803] [talos] task mountPseudoFilesystems (5/7): starting
[    5.824829] [talos] task setRLimit (6/7): starting
[    5.825733] [talos] task dropCapabilities (7/7): done, 21.574363ms
[    5.826970] [talos] task mountPseudoFilesystems (5/7): done, 14.837062ms
[    5.828091] [talos] task setRLimit (6/7): done, 15.952245ms
[    5.829283] [talos] task mountCgroups (4/7): done, 17.154505ms
[    5.830313] [talos] task mountBPFFS (3/7): done, 18.191363ms
[    5.844972] [talos] setting resolvers {"component": "controller-runtime", "controller": "network.ResolverSpecController", "resolvers": ["1.1.1.1", "8.8.8.8"]}
[    5.849125] 8021q: adding VLAN 0 to HW filter on device eth0
[    5.851168] [talos] setting time servers {"component": "controller-runtime", "controller": "network.TimeServerSpecController", "addresses": ["pool.ntp.org"]}
[    5.853357] [talos] setting resolvers {"component": "controller-runtime", "controller": "network.ResolverSpecController", "resolvers": ["1.1.1.1", "8.8.8.8"]}
[    5.856055] [talos] setting time servers {"component": "controller-runtime", "controller": "network.TimeServerSpecController", "addresses": ["pool.ntp.org"]}
[    5.858843] [talos] failed looking up "pool.ntp.org", ignored {"component": "controller-runtime", "controller": "time.SyncController", "error": "lookup pool.ntp.org on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable"}
[    5.862524] [talos] task enforceKSPPRequirements (1/7): done, 51.484031ms
[    5.863541] [talos] phase systemRequirements (2/7): done, 66.449326ms
[    5.864509] [talos] phase integrity (3/7): 1 tasks(s)
[    5.865406] [talos] task writeIMAPolicy (1/1): starting
[    5.866358] audit: type=1807 audit(1674503912.493:2): action=dont_measure fsmagic=0x9fa0 res=1
[    5.867543] audit: type=1807 audit(1674503912.493:3): action=dont_measure fsmagic=0x62656572 res=1
[    5.868740] audit: type=1807 audit(1674503912.493:4): action=dont_measure fsmagic=0x64626720 res=1
[    5.869932] audit: type=1807 audit(1674503912.493:5): action=dont_measure fsmagic=0x1021994 res=1
[    5.871234] ima: policy update completed
[    5.872046] audit: type=1807 audit(1674503912.497:6): action=dont_measure fsmagic=0x1cd1 res=1
[    5.873256] audit: type=1807 audit(1674503912.497:7): action=dont_measure fsmagic=0x42494e4d res=1
[    5.874505] audit: type=1807 audit(1674503912.497:8): action=dont_measure fsmagic=0x73636673 res=1
[    5.875697] audit: type=1807 audit(1674503912.497:9): action=dont_measure fsmagic=0xf97cff8c res=1
[    5.876887] audit: type=1807 audit(1674503912.497:10): action=dont_measure fsmagic=0x43415d53 res=1
[    5.878077] audit: type=1807 audit(1674503912.497:11): action=dont_measure fsmagic=0x27e0eb res=1
[    5.882490] [talos] setting resolvers {"component": "controller-runtime", "controller": "network.ResolverSpecController", "resolvers": ["10.100.1.1", "10.100.50.100"]}
[    5.885727] [talos] setting time servers {"component": "controller-runtime", "controller": "network.TimeServerSpecController", "addresses": ["10.100.1.1"]}
[    5.888672] [talos] setting hostname {"component": "controller-runtime", "controller": "network.HostnameSpecController", "hostname": "talos-master-01", "domainname": "mimir-tech.org\u0000"}
[    5.891619] [talos] setting hostname {"component": "controller-runtime", "controller": "network.HostnameSpecController", "hostname": "talos-master-01", "domainname": "mimir-tech.org\u0000"}
[    5.894681] [talos] assigned address {"component": "controller-runtime", "controller": "network.AddressSpecController", "address": "10.100.50.111/24", "link": "eth0"}
[    5.897108] [talos] created route {"component": "controller-runtime", "controller": "network.RouteSpecController", "destination": "default", "gateway": "10.100.50.1", "table": "main", "link": "eth0"}
[    5.899802] [talos] controller failed {"component": "controller-runtime", "controller": "runtime.KmsgLogDeliveryController", "error": "error sending logs: dial tcp [fd7d:d264:2f4a:d503::1]:4001: connect: network is unreachable"}
[    5.902696] [talos] task writeIMAPolicy (1/1): done, 26.185048ms
[    5.903886] [talos] adjusting time (slew) by 97.051757ms via 10.100.1.1, state TIME_OK, status STA_NANO | STA_PLL {"component": "controller-runtime", "controller": "time.SyncController"}
[    5.906696] [talos] controller failed {"component": "controller-runtime", "controller": "siderolink.ManagerController", "error": "error accessing SideroLink API: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup talos.mimir-tech.org on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable\""}
[    5.911467] [talos] controller failed {"component": "controller-runtime", "controller": "v1alpha1.EventsSinkController", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp [fd7d:d264:2f4a:d503::1]:4002: connect: network is unreachable\""}
[    5.915875] [talos] phase integrity (3/7): done, 40.030335ms
[    5.917063] [talos] phase etc (4/7): 2 tasks(s)
[    5.918196] [talos] task createOSReleaseFile (2/2): starting
[    5.919419] [talos] task CreateSystemCgroups (1/2): starting
[    5.920831] [talos] task createOSReleaseFile (2/2): done, 1.425268ms
[    5.922757] [talos] task CreateSystemCgroups (1/2): done, 4.563844ms
[    5.924175] [talos] phase etc (4/7): done, 7.113873ms
[    5.925485] [talos] phase mountSystem (5/7): 1 tasks(s)
[    5.926846] [talos] task mountStatePartition (1/1): starting
[    5.948255] XFS (sda5): Mounting V5 Filesystem
[    5.954228] XFS (sda5): Ending clean mount
[    5.956383] [talos] task mountStatePartition (1/1): done, 29.538968ms
[    5.957938] [talos] phase mountSystem (5/7): done, 32.452464ms
[    5.960121] [talos] phase config (6/7): 1 tasks(s)
[    5.961354] [talos] node identity established {"component": "controller-runtime", "controller": "cluster.NodeIdentityController", "node_id": "xk7fnOGEnBhjtPl1uBkg7PjpqRAeYIU74KMDeFLfBeeP"}
[    5.964521] [talos] task loadConfig (1/1): starting
[    5.967074] [talos] task loadConfig (1/1): persistence is enabled, using existing config on disk
[    5.968632] [talos] task loadConfig (1/1): done, 7.274634ms
[    5.969840] [talos] phase config (6/7): done, 9.719974ms
[    5.971379] [talos] phase unmountSystem (7/7): 1 tasks(s)
[    5.972866] [talos] task unmountStatePartition (1/1): starting
[    5.990087] XFS (sda5): Unmounting Filesystem
[    6.010324] [talos] task unmountStatePartition (1/1): done, 37.466001ms
[    6.011765] [talos] phase unmountSystem (7/7): done, 40.390462ms
[    6.013137] [talos] initialize sequence: done: 219.27712ms
[    6.014482] [talos] install sequence: 0 phase(s)
[    6.015682] [talos] install sequence: done: 1.198817ms
[    6.016965] [talos] boot sequence: 22 phase(s)
[    6.018089] [talos] phase saveStateEncryptionConfig (1/22): 1 tasks(s)
[    6.019455] [talos] task SaveStateEncryptionConfig (1/1): starting
[    6.020722] [talos] task SaveStateEncryptionConfig (1/1): done, 1.266291ms
[    6.022098] [talos] phase saveStateEncryptionConfig (1/22): done, 4.008409ms
[    6.023496] [talos] phase mountState (2/22): 1 tasks(s)
[    6.024648] [talos] task mountStatePartition (1/1): starting
[    6.030772] [talos] service[machined](Preparing): Running pre state
[    6.033067] [talos] service[machined](Preparing): Creating service runner
[    6.035082] [talos] service[apid](Waiting): Waiting for service "containerd" to be "up", api certificates
[    6.036879] [talos] service[machined](Running): Service started as goroutine
[    6.039095] [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://talos.mimir-tech.org:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[    6.046140] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\n\terror getting node: Get \"https://talos.mimir-tech.org:6443/api/v1/nodes/talos-master-01?timeout=30s\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[    6.055681] XFS (sda5): Mounting V5 Filesystem
[    6.062170] XFS (sda5): Ending clean mount
[    6.063871] [talos] task mountStatePartition (1/1): done, 39.220819ms
[    6.064923] [talos] phase mountState (2/22): done, 41.427229ms
[    6.065916] [talos] phase validateConfig (3/22): 1 tasks(s)
[    6.066918] [talos] task validateConfig (1/1): starting
[    6.067920] [talos] task validateConfig (1/1): done, 1.002614ms
[    6.068917] [talos] phase validateConfig (3/22): done, 3.002471ms
[    6.069936] [talos] phase saveConfig (4/22): 1 tasks(s)
[    6.070907] [talos] task saveConfig (1/1): starting
[    6.072275] [talos] task saveConfig (1/1): done, 1.366816ms
[    6.073243] [talos] phase saveConfig (4/22): done, 3.306964ms
[    6.074254] [talos] phase memorySizeCheck (5/22): 1 tasks(s)
[    6.075209] [talos] task memorySizeCheck (1/1): starting
[    6.076212] [talos] NOTE: recommended memory size is 3946 MiB
[    6.077160] [talos] NOTE: current total memory size is 1939 MiB
[    6.078138] [talos] task memorySizeCheck (1/1): done, 2.928063ms
[    6.079118] [talos] phase memorySizeCheck (5/22): done, 4.864701ms
[    6.080095] [talos] phase diskSizeCheck (6/22): 1 tasks(s)
[    6.081029] [talos] task diskSizeCheck (1/1): starting
[    6.081924] [talos] disk size is OK
[    6.082706] [talos] disk size is 51200 MiB
[    6.083532] [talos] task diskSizeCheck (1/1): done, 2.504088ms
[    6.084500] [talos] phase diskSizeCheck (6/22): done, 4.406013ms
[    6.085466] [talos] phase env (7/22): 1 tasks(s)
[    6.086581] [talos] task setUserEnvVars (1/1): starting
[    6.087708] [talos] task setUserEnvVars (1/1): done, 1.128166ms
[    6.088670] [talos] phase env (7/22): done, 3.204613ms
[    6.089574] [talos] phase containerd (8/22): 1 tasks(s)
[    6.090500] [talos] task startContainerd (1/1): starting
[    6.091423] [talos] service[containerd](Preparing): Running pre state
[    6.092408] [talos] service[containerd](Preparing): Creating service runner
[    6.211727] [talos] controller failed {"component": "controller-runtime", "controller": "v1alpha1.EventsSinkController", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp [fd7d:d264:2f4a:d503::1]:4002: connect: network is unreachable\""}
[    6.226492] [talos] controller failed {"component": "controller-runtime", "controller": "runtime.KmsgLogDeliveryController", "error": "error sending logs: dial tcp [fd7d:d264:2f4a:d503::1]:4001: connect: network is unreachable"}
[    6.328717] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\n\terror getting node: Get \"https://talos.mimir-tech.org:6443/api/v1/nodes/talos-master-01?timeout=30s\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[    6.412003] [talos] created new link {"component": "controller-runtime", "controller": "network.LinkSpecController", "link": "siderolink", "kind": "wireguard"}
[    6.414545] [talos] reconfigured wireguard link {"component": "controller-runtime", "controller": "network.LinkSpecController", "link": "siderolink", "peers": 1}
[    6.419170] [talos] changed MTU for the link {"component": "controller-runtime", "controller": "network.LinkSpecController", "link": "siderolink", "mtu": 1280}
[    6.425190] [talos] assigned address {"component": "controller-runtime", "controller": "network.AddressSpecController", "address": "fd7d:d264:2f4a:d503:7b9d:59d3:7f64:4ec9/64", "link": "siderolink"}
[    7.033211] [talos] service[apid](Waiting): Waiting for service "containerd" to be "up"
[    7.034667] [talos] service[machined](Running): Health check successful
[    7.333338] [talos] service[containerd](Running): Process Process(["/bin/containerd" "--address" "/system/run/containerd/containerd.sock" "--state" "/system/run/containerd" "--root" "/system/var/lib/containerd"]) started with PID 691
[    7.373166] [talos] service[containerd](Running): Health check successful
[    7.374561] [talos] task startContainerd (1/1): done, 1.290125659s
[    7.375707] [talos] phase containerd (8/22): done, 1.292207111s
[    7.376849] [talos] phase dbus (9/22): 1 tasks(s)
[    7.377885] [talos] service[apid](Preparing): Running pre state
[    7.379334] [talos] task startDBus (1/1): starting
[    7.380536] [talos] service[apid](Preparing): Creating service runner
[    7.384033] [talos] task startDBus (1/1): done, 6.179769ms
[    7.385396] [talos] phase dbus (9/22): done, 8.596992ms
[    7.386687] [talos] phase ephemeral (10/22): 1 tasks(s)
[    7.388025] [talos] task mountEphemeralPartition (1/1): starting
[    7.399470] [talos] formatting the partition "/dev/sda6" as "xfs" with label "EPHEMERAL"
[    7.482476] XFS (sda6): Mounting V5 Filesystem
[    7.490771] XFS (sda6): Ending clean mount
[    7.509158] [talos] task mountEphemeralPartition (1/1): done, 121.819749ms
[    7.510933] [talos] phase ephemeral (10/22): done, 124.953919ms
[    7.512244] [talos] phase var (11/22): 1 tasks(s)
[    7.513320] [talos] task setupVarDirectory (1/1): starting
[    7.514584] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.KubeletServiceController", "error": "error writing kubelet PKI: open /etc/kubernetes/bootstrap-kubeconfig: read-only file system"}
[    7.520098] [talos] task setupVarDirectory (1/1): done, 6.81478ms
[    7.521490] [talos] phase var (11/22): done, 9.300702ms
[    7.522711] [talos] phase overlay (12/22): 1 tasks(s)
[    7.523983] [talos] task mountOverlayFilesystems (1/1): starting
[    7.526011] [talos] task mountOverlayFilesystems (1/1): done, 2.042331ms
[    7.527315] [talos] phase overlay (12/22): done, 4.630762ms
[    7.528576] [talos] phase legacyCleanup (13/22): 1 tasks(s)
[    7.529811] [talos] task cleanupLegacyStaticPodFiles (1/1): starting
[    7.531164] [talos] task cleanupLegacyStaticPodFiles (1/1): done, 1.360291ms
[    7.532376] [talos] phase legacyCleanup (13/22): done, 3.822294ms
[    7.533516] [talos] phase udevSetup (14/22): 1 tasks(s)
[    7.534610] [talos] task writeUdevRules (1/1): starting
[    7.535781] [talos] task writeUdevRules (1/1): done, 1.1773ms
[    7.536861] [talos] phase udevSetup (14/22): done, 3.364552ms
[    7.537930] [talos] phase udevd (15/22): 1 tasks(s)
[    7.538846] [talos] task startUdevd (1/1): starting
[    7.540070] [talos] service[udevd](Preparing): Running pre state
[    7.564947] [talos] service[udevd](Preparing): Creating service runner
[    7.575926] [talos] service[udevd](Running): Process Process(["/sbin/udevd" "--resolve-names=never"]) started with PID 716
[    7.576624] udevd[716]: starting version 3.2.11
[    7.586390] udevd[716]: starting eudev-3.2.11
[    7.781976] [talos] service[udevd](Running): Health check successful
[    7.783050] [talos] task startUdevd (1/1): done, 245.590947ms
[    7.784195] [talos] phase udevd (15/22): done, 247.666811ms
[    7.785145] [talos] phase userDisks (16/22): 1 tasks(s)
[    7.786213] [talos] task mountUserDisks (1/1): starting
[    7.787131] [talos] task mountUserDisks (1/1): done, 923.288µs
[    7.788164] [talos] phase userDisks (16/22): done, 3.037685ms
[    7.789229] [talos] phase userSetup (17/22): 1 tasks(s)
[    7.790269] [talos] task writeUserFiles (1/1): starting
[    7.791145] [talos] task writeUserFiles (1/1): done, 881.449µs
[    7.792121] [talos] phase userSetup (17/22): done, 2.908945ms
[    7.793066] [talos] phase lvm (18/22): 1 tasks(s)
[    7.793951] [talos] task activateLogicalVolumes (1/1): starting
[    7.901764] [talos] task activateLogicalVolumes (1/1): done, 108.419059ms
[    7.902840] [talos] phase lvm (18/22): done, 110.418534ms
[    7.903763] [talos] phase startEverything (19/22): 1 tasks(s)
[    7.904818] [talos] task startAllServices (1/1): starting
[    7.905859] [talos] task startAllServices (1/1): waiting for 8 services
[    7.906901] [talos] service[cri](Waiting): Waiting for network
[    7.907917] [talos] service[trustd](Waiting): Waiting for service "containerd" to be "up", time sync, network
[    7.909292] [talos] service[etcd](Waiting): Waiting for service "cri" to be "up", time sync, network, etcd spec
[    7.910656] [talos] service[cri](Preparing): Running pre state
[    7.912844] [talos] service[trustd](Preparing): Running pre state
[    7.914034] [talos] task startAllServices (1/1): service "apid" to be "up", service "containerd" to be "up", service "cri" to be "up", service "etcd" to be "up", service "kubelet" to be "up", service "machined" to be "up", service "trustd" to be "up", service "udevd" to be "up"
[    7.917518] [talos] service[trustd](Preparing): Creating service runner
[    7.918969] [talos] service[cri](Preparing): Creating service runner
[    7.920617] [talos] service[cri](Running): Process Process(["/bin/containerd" "--address" "/run/containerd/containerd.sock" "--config" "/etc/cri/containerd.toml"]) started with PID 1598
[    8.036546] [talos] service[kubelet](Waiting): Waiting for service "cri" to be "up", time sync, network
[    8.503297] [talos] service[trustd](Running): Started task trustd (PID 1647) for container trustd
[    8.510405] [talos] service[apid](Running): Started task apid (PID 1646) for container apid
[    8.905490] [talos] service[etcd](Waiting): Waiting for service "cri" to be "up", etcd spec
[    8.915890] [talos] service[cri](Running): Health check successful
[    8.916937] [talos] service[kubelet](Preparing): Running pre state
[    8.921793] [talos] service[trustd](Running): Health check successful
[    9.900329] [talos] service[etcd](Waiting): Waiting for etcd spec
[   11.414260] [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://talos.mimir-tech.org:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[   11.421745] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\n\terror getting node: Get \"https://talos.mimir-tech.org:6443/api/v1/nodes/talos-master-01?timeout=30s\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[   12.339027] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\n\terror getting node: Get \"https://talos.mimir-tech.org:6443/api/v1/nodes/talos-master-01?timeout=30s\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[   12.533698] [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://talos.mimir-tech.org:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[   13.368519] [talos] service[apid](Running): Health check successful
[   14.120641] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\n\terror getting node: Get \"https://talos.mimir-tech.org:6443/api/v1/nodes/talos-master-01?timeout=30s\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[   15.511030] [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://talos.mimir-tech.org:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[   16.150585] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\n\terror getting node: Get \"https://talos.mimir-tech.org:6443/api/v1/nodes/talos-master-01?timeout=30s\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[   21.269858] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\n\terror getting node: Get \"https://talos.mimir-tech.org:6443/api/v1/nodes/talos-master-01?timeout=30s\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[   21.602793] [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to list *v1.Endpoints: Get \"https://talos.mimir-tech.org:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}
[   22.860243] [talos] task startAllServices (1/1): service "etcd" to be "up", service "kubelet" to be "up"
[   27.555807] [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeLabelsApplyController", "error": "1 error(s) occurred:\n\terror getting node: Get \"https://talos.mimir-tech.org:6443/api/v1/nodes/talos-master-01?timeout=30s\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"kubernetes\")"}