talos: Talos cannot read some Kubernetes APIs after K8s v1.32.0
Bug Report
Talos cannot read some Kubernetes APIs after K8s v1.32.0.
Logs
[talos] kubernetes registry node watch error {"component": "controller-runtime", "controller": "cluster.KubernetesPullController", "error": "failed to list *v1.Node: nodes is forbidden: User \"system:node:m0\" cannot list resource \"nodes\" in API group \"\" at the cluster scope: node 'm0' cannot read all nodes, only its own Node object"}
Environment
- Talos version: v1.9.0
- Kubernetes version: v1.32.0
- Platform: metal
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 9
- Comments: 34 (11 by maintainers)
Commits related to this issue
- docs: document Kubernetes service registry incompat with K8s 1.32 Fixes #9980 Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com> — committed to smira/talos by smira 2 years ago
- docs: document Kubernetes service registry incompat with K8s 1.32 Fixes #9980 Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com> — committed to smira/talos by smira 2 years ago
- docs: document Kubernetes service registry incompat with K8s 1.32 Fixes #9980 Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com> — committed to smira/talos by smira 2 years ago
- docs: document Kubernetes service registry incompat with K8s 1.32 Fixes #9980 Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com> (cherry picked from commit c12b52491456d1e52204eb290d0686a... — committed to smira/talos by smira 2 years ago
- fix(talos): disable k8s discovery ref: https://github.com/siderolabs/talos/issues/9980 — committed to zebernst/homelab by zebernst 2 years ago
So this issue is caused by Kubernetes 1.32.0 enabling by default a feature gate
AuthorizeNodeWithSelectorswhich is implemented in https://github.com/kubernetes/kubernetes/pull/125571, specifically https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/4601-authorize-with-selectors#node-authorizer-changes.Due to the nature of the Kubernetes discovery in Talos, it can’t rely on other mechanisms to bootstrap itself, so it picks up the kubelet’s client kubeconfig and tries to use it to watch the
Noderesource. This was allowed by thesystem:nodeClusterRole previously, but withAuthorizeNodeWithSelectorsNodeRestrictionplugin implements more tight permission to check to allow thesystem:node:<foo>to only see its ownNode<foo>(which makes perfect sense from a security point of view).The workaround for now is to apply the following config patch (Kubernetes 1.32+ only) to all controlplane nodes:
But this change is not recommended, as this disables other checks/security enhancements as well.
So all in all, the workaround will be documented, but the Kubernetes discovery is likely to be going away in Talos.
With the changes to Kubernetes API server, it would be nice if Sidero could reconsider the license on the discovery service since it doesn’t seem like there is a workaround (at least a secure one), or in the very least give more documentation around running Talos in an airgapped environment with the nuances discussed here summarized.
I know Sidero has to pay employees and turn profit however I am only suggesting this due to the changes to the API server and the concerns people have brought up in this issue which were not present when the discovery service was first released & licensed.
Does this mean usage of Talos in production in air gapped environments is only possible via a commercial license? This makes Talos unsuitable for us, and we have to start looking for open alternatives.
Use the public discovery service. The data transiting it is all encrypted and unreadable by the service.
On Fri, Mar 7, 2025 at 5:24 PM Devin Buhl @.***> wrote:
Discovery service is under BUSL, so non-commercial, non production deployment is explicitly allowed.
Which basically shoots this whole project in the lab in the foot, policies do not allow using external services or dependencies for cluster / devops / security infrastructure, which now means if i move this POC forward i also need a license and have to deploy discovery locally. Maybe its just time to drop talos in light of this, i had hopes but, not sure this is going to fly, we arent a fortune 500 that can afford huge licensing fees for a simple feature.
I know see 8 months of work prototyping in the lab completely wasted.
Will SideroLabs allow non-commercial deployments of discovery-service to replace Kubernetes discovery for people/organizations that do not want an external dependency or are airgapped?