talos: Talos cannot read some Kubernetes APIs after K8s v1.32.0

Bug Report

Talos cannot read some Kubernetes APIs after K8s v1.32.0.

Logs

[talos] kubernetes registry node watch error {"component": "controller-runtime", "controller": "cluster.KubernetesPullController", "error": "failed to list *v1.Node: nodes is forbidden: User \"system:node:m0\" cannot list resource \"nodes\" in API group \"\" at the cluster scope: node 'm0' cannot read all nodes, only its own Node object"}

Environment

  • Talos version: v1.9.0
  • Kubernetes version: v1.32.0
  • Platform: metal

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 9
  • Comments: 34 (11 by maintainers)

Commits related to this issue

Most upvoted comments

So this issue is caused by Kubernetes 1.32.0 enabling by default a feature gate AuthorizeNodeWithSelectors which is implemented in https://github.com/kubernetes/kubernetes/pull/125571, specifically https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/4601-authorize-with-selectors#node-authorizer-changes.

Due to the nature of the Kubernetes discovery in Talos, it can’t rely on other mechanisms to bootstrap itself, so it picks up the kubelet’s client kubeconfig and tries to use it to watch the Node resource. This was allowed by the system:node ClusterRole previously, but with AuthorizeNodeWithSelectors NodeRestriction plugin implements more tight permission to check to allow the system:node:<foo> to only see its own Node <foo> (which makes perfect sense from a security point of view).

The workaround for now is to apply the following config patch (Kubernetes 1.32+ only) to all controlplane nodes:

cluster:
  apiServer:
    extraArgs:
      feature-gates: AuthorizeNodeWithSelectors=false

But this change is not recommended, as this disables other checks/security enhancements as well.

So all in all, the workaround will be documented, but the Kubernetes discovery is likely to be going away in Talos.

With the changes to Kubernetes API server, it would be nice if Sidero could reconsider the license on the discovery service since it doesn’t seem like there is a workaround (at least a secure one), or in the very least give more documentation around running Talos in an airgapped environment with the nuances discussed here summarized.

I know Sidero has to pay employees and turn profit however I am only suggesting this due to the changes to the API server and the concerns people have brought up in this issue which were not present when the discovery service was first released & licensed.

Does this mean usage of Talos in production in air gapped environments is only possible via a commercial license? This makes Talos unsuitable for us, and we have to start looking for open alternatives.

Use the public discovery service. The data transiting it is all encrypted and unreadable by the service.

On Fri, Mar 7, 2025 at 5:24 PM Devin Buhl @.***> wrote:

With the changes to Kubernetes API server, it would be nice if the discovery service was re-licensed under AGPL / MPL / MIT / Apache or whatever since it doesn’t seem like there is a workaround (at least a secure one).

— Reply to this email directly, view it on GitHub https://github.com/siderolabs/talos/issues/9980#issuecomment-2707863154, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQGWG5MO5PPYJUHSRK4QKOL2TJBE7AVCNFSM6AAAAABTY37YOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBXHA3DGMJVGQ . You are receiving this because you commented.Message ID: @.***> [image: onedr0p]onedr0p left a comment (siderolabs/talos#9980) https://github.com/siderolabs/talos/issues/9980#issuecomment-2707863154

With the changes to Kubernetes API server, it would be nice if the discovery service was re-licensed under AGPL / MPL / MIT / Apache or whatever since it doesn’t seem like there is a workaround (at least a secure one).

— Reply to this email directly, view it on GitHub https://github.com/siderolabs/talos/issues/9980#issuecomment-2707863154, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQGWG5MO5PPYJUHSRK4QKOL2TJBE7AVCNFSM6AAAAABTY37YOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBXHA3DGMJVGQ . You are receiving this because you commented.Message ID: @.***>

Discovery service is under BUSL, so non-commercial, non production deployment is explicitly allowed.

No, you can still use Kubespan / Kubeprism and all of Talos free, in production, without a license, by using our free public Discovery Service.

The only constraint is if you want to run air-gapped, or for some other reason need to run your own Discovery service - that requires a license.

Which basically shoots this whole project in the lab in the foot, policies do not allow using external services or dependencies for cluster / devops / security infrastructure, which now means if i move this POC forward i also need a license and have to deploy discovery locally. Maybe its just time to drop talos in light of this, i had hopes but, not sure this is going to fly, we arent a fortune 500 that can afford huge licensing fees for a simple feature.

I know see 8 months of work prototyping in the lab completely wasted.

Will SideroLabs allow non-commercial deployments of discovery-service to replace Kubernetes discovery for people/organizations that do not want an external dependency or are airgapped?