talos: Unable to get Talos working with IPv6

Bug Report

Talos’s doc does not contain any info about how to get IPv6 working. After guessing some parameters, it still does not work for multiple reasons. If I tried Cilium, everything here is when using Flannel with default settings but the pods and services subnet CIDRs. Both are specified with IPv4 and IPv6 ranges. IPv4 is the default 10.244/16 and IPv6 is a /60. Services CIDRs are also default, 10.96 and a /112 after I figured that from my other experiences.

Description

Setup is 9 nodes (3 controlplanes and 6 workers) using Talos’s OVA v1.6.1. IPv4 DHCP reservation for each node (names / ip also registered in the DNS) DHCPv6 available in the segment. Router Advertisement is in managed mode (DHCPv6 but not SLAAC)

So fresh boot without a single bit changed in the OVA and all nodes come up with the proper IPv4, proper hostname and no IPv6.

Machine config files used for: –a VIP shared by the controlplanes (IPv6) –configuring the NTP Reference –configuring strict-arp –configuring pods and services CIDR –fixing each node’s IPv4 and IPv6 addresses, define default route (IPv4) and nameservers (one v4 and one v6) –pointing to the extra manifest required by VMTools (downloaded from a local static URL) –add extras SANs in certs

Once applied, all nodes received their config and none are ready because cluster is not bootstrapped yet.

Last step is to bootstrap the cluster, pointing to the IPv6 address of the first controlplane.

Subnets too large

Trying to guess how to do it, I failed and gave up, moving to Ubuntu instead. With Ubuntu and Kubeadm, I kept trying. The difference is that I received useful error messages, like some subnet masks being too large.

Ex: for service subnet CIDR, Ubuntu / Kubeadm complains when using /64. It seems not to accept subnets larger than a /112.

While reading some other people debugs, some ended up with problems when their subnet CIDR ended up not large enough to be splitted among all of their nodes. They had a /64 subnet, the CNI was trying to give /66 ranges to each node but failed because there was not enough ranges for all the nodes (only 4 /66 in a /64 and they had like 8 nodes).

Here, I have to rely much more on horizontal scaling then vertical. As such, I have a total of 9 nodes. The /60 contains 16 /64, so I should be Ok despite, according to some documentation, the first of them is not used (not sure why…).

Internal node IPs

After deploying the cluster with settings that were accepted and did not push anything in crashloopbackoff, Talos was still not consistent.

Ex: All nodes where configured with static IPv4 and IPv6 from a machineconfig file applied before bootstrapping the cluster. The cluster (3 controlplanes and 6 workers) is then bootstrapped pointing to controller No1’s IPv6 address. Once the cluster up and running, internal addressing is messed up.

kubectl get nodes -o wide

Shows all 3 controlplane with their static IPv6 address as internal IP while the 6 workers are showed with their IPv4. I had to restrict kubelet’s valid subnet to IPv6 to force all nodes to show their IPv6 as internal address.

Validation webhook unreachable : No route to host

When I try to deploy MetalLB, it complains that the validation webhook is unreachable because there is no route to host. I received the same kind of error message with nginx ingress.

I modified the deployment of MetalLB, from IPv6 first / IPv4 second to IPv4 first / IPv6 second. Only then it came online functional thanks to IPv4. So again, this is evidence that it is specifically IPv6 that is not working.

One of my suspect is kubeprism but I have no evidence about this. Should that one be limited to IPv4, it would explain why webhook validation paths (metallb or nginx) were unreachable.

DHCPv6

Another problem is about getting fixed IPv6. Whenever I reset the cluster to maintenance mode, nodes come up with a different DHCPv6 UID, so static reservation does not work. When trying to fix that UID in machineconfig, it seems to be ignored. I suspect it is ignored because I did not put it in the proper format Talos is looking for. But again, because Talos’s documentation about IPv6 is a big Zero, there is no way for me to figure out what format would work (or it format is indeed the problem or not).

To derive the UID from the MAC address would allow a different ID on each node while remaining consistent after reset to maintenance mode.

There is a ton of network config in a kubernetes cluster : –Addresses in the DMZ where all nodes are deployed (if they are all in the same subnet like mines) –pod subnet –service subnet –load balancer’s IPs

Routing and Nating for letting pods connect resources outside the cluster –Even IPv6 needs NAT (when it is not supposed to support it…) –Load Balancer also needs some connectivity settings (L2 or BGP) and again, for both IPFamilies –IPv6 uses the Router Advertisement service and you are not supposed to point a route to a public IPv6. Routes should re-use the local-link addresses.

Because Talos uses Flannel as its default CNI, documentation should be complete at least for that one. If you wish to support another one like Cilium, how to configure it for IPv6 should be explained in detail too.

Logs

Environment

  • Talos version: [talosctl version --nodes <problematic nodes>]

1.6.1

  • Kubernetes version: [kubectl version --short]

1.29

  • Platform:

vmware-amd64.ova deployed in ESXi 6.7U3

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 27 (11 by maintainers)

Most upvoted comments

Ok I figured this out. By Pure luck too. And Its so dumb :

When writing your configuration file, you need to take care of the order in which you define your podSubnets and serviceSubnets params :

THIS :

cluster:
  ...
  network:
    podSubnets:
      - fda0:0:0:1000::/64 # random ULA subnet
      - 10.209.0.0/16
    serviceSubnets:
      # WARNING: IPv6 service subnet cannot be larger than /108 (previous discussion suggested /112)
      #   see linked configs from https://github.com/siderolabs/talos/issues/8115#issuecomment-2068026656
      - fda0:0:0:2000::/108 # random ULA subnet
      - 10.96.0.0/16

will work

But THIS:

cluster
  ...
  network:
    podSubnets:
      - 10.209.0.0/16
      - fda0:0:0:1000::/64 # random ULA subnet
    serviceSubnets:
      - 10.96.0.0/16
      # WARNING: IPv6 service subnet cannot be larger than /108 (previous discussion suggested /112)
      #   see linked configs from https://github.com/siderolabs/talos/issues/8115#issuecomment-2068026656
      - fda0:0:0:2000::/108 # random ULA subnet

will fail. The cluster will never come up and you will see (InternalError): an error on the server ("proxy error") has prevented the request from succeeding (get pods) when querying the API

Why ? kube-api wont start due to a missmatch between IP families of Services and External IP :

2024-08-23T09:32:25.871070123Z stderr F I0823 09:32:25.870881       1 options.go:221] external host was not specified, using 2a02:XXXX:2403:3:be24:11ff:fef1:13bf
2024-08-23T09:32:25.871623531Z stderr F E0823 09:32:25.871467       1 run.go:74] "command failed" err="service IP family \"10.96.0.0/16\" must match public address family \"2a02:XXXX:2403:3:be24:11ff:fef1:13bf\""

It doesnt seem like this is a Talos issue but rather upstream Kubernetes issue that was never truly fixed. So I think the fix here is to document this somewhere.

For the future people who come in this thread trying to make IPv6 work with Talos and Cilium, here is the TL:DR:

  • Start with @nazarewk config above, but with a few changes:
  • Take care of the order in which you declare cluster.network.podSubnets and cluster.network.serviceSubnets. If you want IPv6/dualstack you need to set the IPV6 CIDR first for each of these fields.
  • Do not use the machine.network.vip.ip as k8sServiceHost in your Cilium config as suggested above. At least for me, using ipv6NativeRouting/ipv4NativeRouting optimizations in Cilium for pod traffic means it didn’t work. Use k8sServiceHost : "::1" in your Cilium chart.
  • Add ::1 to the API certificate SAN or your cluster wont boot. E.g. cluster.apiServer.certSANs : "::1" or you will get tls: internal error when Cilium tries to reach the API and your cluster wont boot.

Thanks for helping me figure this out.

Cheers B.

PS : I will now try to ennable kubePrism again as, as you suggested, I dont think its a good idea to turn it off as suggested by other users above in this thread. It seems Cilium might rely on it for some features since they explicitly mention it in their Talos docs I will edit this post when I manage to make it work.

I had a similar series of issues after trying to switch to new set of machines (recreate cluster) and make them dual-stack at the same time.

I managed to get my setup to work by restricting serviceSubnets to /108 inspired by:

Update: I had to also add this, so the first node doesn’t take the whole /64 IPv6 range:

cluster:
  controllerManager:
    extraArgs:
      node-cidr-mask-size-ipv6: "80" # defaults to /64 - whole assigned ULA subnet

I think it is safe to close this issue. I haven’t had any problems with IPv6 lately.

  • Talos switched to Cloudflare as the default NTP server so that works out-of-the-box now, since they provide AAAA records.
  • ghcr.io still isn’t available under IPv6 but that’s only a problem for IPv6-only setups, and no issue for dual-stack. And the workaround is relatively easy, just set up an OCI mirror locally that can access the IPv4 internet.

There was some cofnfusion about CIDR sizes, and a lot of that boils down to different requirements by different Kubernetes components + Cilium having a different default.

  • Node IPv6 network can be anything really; ULA provided by most commodity hardware tends to work just fine.
  • DNS works too, if the network doesn’t provide a nameserver any of the public ones should work, like Quad9, 2620:fe::9.
  • Kubernetes API server requires a service CIDR no larger than /108. /108 works fine, but /112 also works; but that means it can’t be /64.
  • Pod CIDR can be way larger than that. Kubernetes components default to /64, but weirdly enough, Cilium defaults to /120. Leaving the defaults leads to problems where Kubernetes IPAM could assign something from the default /64 which is outside of the default /120 that Cilium requires, leading to Cilium being unable to route IPv6.
  • Cilium has some limitation of the Pod CIDR size, I think /56 or so is the largest CIDR that it acceps, but I haven’t tried going that large. I just set it to /64 which gives me plenty of available addresses.

In addition to that, both Cilium and Kubernetes require that the difference between elements of the pod CIDR list and the mask size be no larger than 16. The default in Cilium is exactly 16, since the default CIDR list is ["fd00::/104"] and the default mask size is 120, 120-104=16. This limitation is quite annoying and can be very easy to trip over.

Finally, IPv6 native routing also works with Cilium (given the cluster spans a single L2), however one must take care to not assign pod or service CIDRs that overlap the ULA-assigned node IPs, otherwise the nodes themselves become un-routable from within pods. The same is true for IPv4 so this isn’t anything new.

If there anything still not working than we should look into it, but otherwise I think this issue is safe to close. Thank you everyone for the contributions, links and shared config snippets, it was really helpful.

I’ll add my updates here, turns out the KubePrism issue is not really a problem, as long as we don’t disable IPv6 on the loopback interface (which most folks probably don’t do anyway). Eventually I got Talos with Cilium working, with IPv6-only node, pod and service subnets, with a few workarounds:

  • The default time server, until a recent change to Cloudflare, did not have AAAA records. Recently Talos switched to Cloudflare so that is now solved, the default time server is IPv6-ready.
  • Some of the containers don’t have IPv6 reachability. The simple workaround here is to run a local mirror, similarly how one would do in airgapped environments, and just crane over all the dependencies.
  • Cilium L2 advertisements don’t work. (The default Flannel setup might work though, I haven’t tried.) That’s unrelated to Talos though.

Overall it seems to be possible to have a functioning cluster with IPv6-only networking on all physical interfaces, and have IPv4 only on the loopback. I have a couple old configs over here that might point you in the right direction.

Turns out what I need to resolve the issue is:

  • Specify IPv6 pod & service subnets (I use talhelper, which seems to default to IPv4)
  • Disable KubePrism, since now an IPv4 host is hard-coded as of f70b47dddc2599a618c68d8b403d9b37c61f2b71.

I’m filing a new issue about that second bit.

@skandragon OK, but order matters — IPv6/IPv4 dual stack is different than IPv4/IPv6 dual stack — which is what I was specifically asking about.

In any case, for anyone who stumbles upon this issue and has similar concerns, we’ve been running IPv6/IPv4 dual stack on Talos for a few weeks now, and it’s mostly working fine. Any issues we’ve had are more to do with various applications not supporting IPv6/IPv4 dual stack (e.g., Mayastor, though that should be fixed soon) than Talos itself.

Argo CD’s static HA manifests are another example of an application that fails spectacularly on an IPv6/IPv4 dual stack cluster, because the services it creates are SingleStack, which means they’ll be IPv6-only on an IPv6/IPv4 dual stack cluster; but it hard-codes 0.0.0.0 in several places, which means those listeners will fail to bind to an address. The solution here is to use the Argo CD Helm chart, rather than the static manifests. But again, this is an application issue, not a Talos issue.

I don’t know exactly why, but I don’t think it is easy (or possible at all) to modify cluster addressing after creation.

Turns out it is.

I did something basically like this:

  • added ipv6 to the config.
  • For each control node, starting with the control plane nodes, I would
    • Shut the node down via talosctl, and if it hangs on a non-CP node, hard stop via Proxmox console
    • While the node was down, use “kubectl delete node” to remove it
    • Restart the node via Proxmox console
    • The node should re-register and get both an IPv4 and IPv6 config

I also had a little panic, as while doing this, I am making a split-brain cluster. The newly re-created nodes will not be able to communicate with the old nodes. This is because Flannel ends up using the newly created ipv6 tunnels for communication, while the old nodes will only use ipv4. The “fix” is to just press on…

After this, I am now able to get a working ipv4 + ipv6 with no data loss, and minimal downtime. My Ceph cluster can tolerate a single host going down in all cases, but I did have to push the issue about half way through as there were too many hosts down for the cluster to be healthy if one more was lost. Thus, the hard reset on those to force them to shut down (uncleanly) and recover in the new world.

After about an hour of doing this, everything was fully up and working.

I was trying to deploy it as a new install. I gave up and now run a cluster built with Kubeadm under Ubuntu.

I do not have much time to keep fighting my way through this, so I will keep my actual cluster which is working fine with dual stacks…

When the documentation will be updated with specific procedures for IPv6, I may try to migrate to Talos but for now, I will remain as is.

Thanks for your input on this one,