talos: Unable to get Talos working with IPv6
Bug Report
Talos’s doc does not contain any info about how to get IPv6 working. After guessing some parameters, it still does not work for multiple reasons. If I tried Cilium, everything here is when using Flannel with default settings but the pods and services subnet CIDRs. Both are specified with IPv4 and IPv6 ranges. IPv4 is the default 10.244/16 and IPv6 is a /60. Services CIDRs are also default, 10.96 and a /112 after I figured that from my other experiences.
Description
Setup is 9 nodes (3 controlplanes and 6 workers) using Talos’s OVA v1.6.1. IPv4 DHCP reservation for each node (names / ip also registered in the DNS) DHCPv6 available in the segment. Router Advertisement is in managed mode (DHCPv6 but not SLAAC)
So fresh boot without a single bit changed in the OVA and all nodes come up with the proper IPv4, proper hostname and no IPv6.
Machine config files used for: –a VIP shared by the controlplanes (IPv6) –configuring the NTP Reference –configuring strict-arp –configuring pods and services CIDR –fixing each node’s IPv4 and IPv6 addresses, define default route (IPv4) and nameservers (one v4 and one v6) –pointing to the extra manifest required by VMTools (downloaded from a local static URL) –add extras SANs in certs
Once applied, all nodes received their config and none are ready because cluster is not bootstrapped yet.
Last step is to bootstrap the cluster, pointing to the IPv6 address of the first controlplane.
Subnets too large
Trying to guess how to do it, I failed and gave up, moving to Ubuntu instead. With Ubuntu and Kubeadm, I kept trying. The difference is that I received useful error messages, like some subnet masks being too large.
Ex: for service subnet CIDR, Ubuntu / Kubeadm complains when using /64. It seems not to accept subnets larger than a /112.
While reading some other people debugs, some ended up with problems when their subnet CIDR ended up not large enough to be splitted among all of their nodes. They had a /64 subnet, the CNI was trying to give /66 ranges to each node but failed because there was not enough ranges for all the nodes (only 4 /66 in a /64 and they had like 8 nodes).
Here, I have to rely much more on horizontal scaling then vertical. As such, I have a total of 9 nodes. The /60 contains 16 /64, so I should be Ok despite, according to some documentation, the first of them is not used (not sure why…).
Internal node IPs
After deploying the cluster with settings that were accepted and did not push anything in crashloopbackoff, Talos was still not consistent.
Ex: All nodes where configured with static IPv4 and IPv6 from a machineconfig file applied before bootstrapping the cluster. The cluster (3 controlplanes and 6 workers) is then bootstrapped pointing to controller No1’s IPv6 address. Once the cluster up and running, internal addressing is messed up.
kubectl get nodes -o wide
Shows all 3 controlplane with their static IPv6 address as internal IP while the 6 workers are showed with their IPv4. I had to restrict kubelet’s valid subnet to IPv6 to force all nodes to show their IPv6 as internal address.
Validation webhook unreachable : No route to host
When I try to deploy MetalLB, it complains that the validation webhook is unreachable because there is no route to host. I received the same kind of error message with nginx ingress.
I modified the deployment of MetalLB, from IPv6 first / IPv4 second to IPv4 first / IPv6 second. Only then it came online functional thanks to IPv4. So again, this is evidence that it is specifically IPv6 that is not working.
One of my suspect is kubeprism but I have no evidence about this. Should that one be limited to IPv4, it would explain why webhook validation paths (metallb or nginx) were unreachable.
DHCPv6
Another problem is about getting fixed IPv6. Whenever I reset the cluster to maintenance mode, nodes come up with a different DHCPv6 UID, so static reservation does not work. When trying to fix that UID in machineconfig, it seems to be ignored. I suspect it is ignored because I did not put it in the proper format Talos is looking for. But again, because Talos’s documentation about IPv6 is a big Zero, there is no way for me to figure out what format would work (or it format is indeed the problem or not).
To derive the UID from the MAC address would allow a different ID on each node while remaining consistent after reset to maintenance mode.
There is a ton of network config in a kubernetes cluster : –Addresses in the DMZ where all nodes are deployed (if they are all in the same subnet like mines) –pod subnet –service subnet –load balancer’s IPs
Routing and Nating for letting pods connect resources outside the cluster –Even IPv6 needs NAT (when it is not supposed to support it…) –Load Balancer also needs some connectivity settings (L2 or BGP) and again, for both IPFamilies –IPv6 uses the Router Advertisement service and you are not supposed to point a route to a public IPv6. Routes should re-use the local-link addresses.
Because Talos uses Flannel as its default CNI, documentation should be complete at least for that one. If you wish to support another one like Cilium, how to configure it for IPv6 should be explained in detail too.
Logs
Environment
- Talos version: [
talosctl version --nodes <problematic nodes>]
1.6.1
- Kubernetes version: [
kubectl version --short]
1.29
- Platform:
vmware-amd64.ova deployed in ESXi 6.7U3
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 27 (11 by maintainers)
Ok I figured this out. By Pure luck too. And Its so dumb :
When writing your configuration file, you need to take care of the order in which you define your
podSubnetsandserviceSubnetsparams :THIS :
will work
But THIS:
will fail. The cluster will never come up and you will see
(InternalError): an error on the server ("proxy error") has prevented the request from succeeding (get pods)when querying the APIWhy ?
kube-apiwont start due to a missmatch between IP families of Services and External IP :It doesnt seem like this is a Talos issue but rather upstream Kubernetes issue that was never truly fixed. So I think the fix here is to document this somewhere.
For the future people who come in this thread trying to make IPv6 work with Talos and Cilium, here is the TL:DR:
cluster.network.podSubnetsandcluster.network.serviceSubnets. If you want IPv6/dualstack you need to set the IPV6 CIDR first for each of these fields.machine.network.vip.ipask8sServiceHostin your Cilium config as suggested above. At least for me, usingipv6NativeRouting/ipv4NativeRoutingoptimizations in Cilium for pod traffic means it didn’t work. Usek8sServiceHost : "::1"in your Cilium chart.::1to the API certificate SAN or your cluster wont boot. E.g.cluster.apiServer.certSANs : "::1"or you will gettls: internal errorwhen Cilium tries to reach the API and your cluster wont boot.Thanks for helping me figure this out.
Cheers B.
PS : I will now try to ennable kubePrism again as, as you suggested, I dont think its a good idea to turn it off as suggested by other users above in this thread. It seems Cilium might rely on it for some features since they explicitly mention it in their Talos docs I will edit this post when I manage to make it work.
I had a similar series of issues after trying to switch to new set of machines (recreate cluster) and make them dual-stack at the same time.
I managed to get my setup to work by restricting
serviceSubnetsto/108inspired by:Update: I had to also add this, so the first node doesn’t take the whole
/64IPv6 range:I think it is safe to close this issue. I haven’t had any problems with IPv6 lately.
There was some cofnfusion about CIDR sizes, and a lot of that boils down to different requirements by different Kubernetes components + Cilium having a different default.
2620:fe::9.In addition to that, both Cilium and Kubernetes require that the difference between elements of the pod CIDR list and the mask size be no larger than 16. The default in Cilium is exactly 16, since the default CIDR list is
["fd00::/104"]and the default mask size is 120, 120-104=16. This limitation is quite annoying and can be very easy to trip over.Finally, IPv6 native routing also works with Cilium (given the cluster spans a single L2), however one must take care to not assign pod or service CIDRs that overlap the ULA-assigned node IPs, otherwise the nodes themselves become un-routable from within pods. The same is true for IPv4 so this isn’t anything new.
If there anything still not working than we should look into it, but otherwise I think this issue is safe to close. Thank you everyone for the contributions, links and shared config snippets, it was really helpful.
I’ll add my updates here, turns out the KubePrism issue is not really a problem, as long as we don’t disable IPv6 on the loopback interface (which most folks probably don’t do anyway). Eventually I got Talos with Cilium working, with IPv6-only node, pod and service subnets, with a few workarounds:
Overall it seems to be possible to have a functioning cluster with IPv6-only networking on all physical interfaces, and have IPv4 only on the loopback. I have a couple old configs over here that might point you in the right direction.
Turns out what I need to resolve the issue is:
I’m filing a new issue about that second bit.
@skandragon OK, but order matters — IPv6/IPv4 dual stack is different than IPv4/IPv6 dual stack — which is what I was specifically asking about.
In any case, for anyone who stumbles upon this issue and has similar concerns, we’ve been running IPv6/IPv4 dual stack on Talos for a few weeks now, and it’s mostly working fine. Any issues we’ve had are more to do with various applications not supporting IPv6/IPv4 dual stack (e.g., Mayastor, though that should be fixed soon) than Talos itself.
Argo CD’s static HA manifests are another example of an application that fails spectacularly on an IPv6/IPv4 dual stack cluster, because the services it creates are
SingleStack, which means they’ll be IPv6-only on an IPv6/IPv4 dual stack cluster; but it hard-codes0.0.0.0in several places, which means those listeners will fail to bind to an address. The solution here is to use the Argo CD Helm chart, rather than the static manifests. But again, this is an application issue, not a Talos issue.Turns out it is.
I did something basically like this:
I also had a little panic, as while doing this, I am making a split-brain cluster. The newly re-created nodes will not be able to communicate with the old nodes. This is because Flannel ends up using the newly created ipv6 tunnels for communication, while the old nodes will only use ipv4. The “fix” is to just press on…
After this, I am now able to get a working ipv4 + ipv6 with no data loss, and minimal downtime. My Ceph cluster can tolerate a single host going down in all cases, but I did have to push the issue about half way through as there were too many hosts down for the cluster to be healthy if one more was lost. Thus, the hard reset on those to force them to shut down (uncleanly) and recover in the new world.
After about an hour of doing this, everything was fully up and working.
I was trying to deploy it as a new install. I gave up and now run a cluster built with Kubeadm under Ubuntu.
I do not have much time to keep fighting my way through this, so I will keep my actual cluster which is working fine with dual stacks…
When the documentation will be updated with specific procedures for IPv6, I may try to migrate to Talos but for now, I will remain as is.
Thanks for your input on this one,