talos: Unexpected behavior at network configuration

Bug Report

Description

I have an eno1 interface and tried to configure a bridge using the following configuration:

    interfaces:
      - interface: br0
        addresses:
          - <IP>/24
        bridge:
          stp:
            enabled: true
          interfaces:
            - eno1
        routes:
          - network: 0.0.0.0/0
            gateway: <GW>

After applying the configuration, the network on the node went down, and I observed the following logs:

 user: warning: [2025-03-23T16:44:02.086634404Z]: [talos] updating bridge settings {"component": "controller-runtime", "controller": "network.LinkSpecController",     
 "link": "br0", "old": "{STP:{Enabled:false} VLAN:{FilteringEnabled:false}}", "new": "{STP:{Enabled:true} VLAN:{FilteringEnabled:false}}"}                             
 user: warning: [2025-03-23T16:44:02.088479404Z]: [talos] updated bridge settings {"component": "controller-runtime", "controller": "network.LinkSpecController",      
 "link": "br0"}                                                                                                                                                        
 user: warning: [2025-03-23T16:44:02.124169404Z]: [talos] controller failed {"component": "controller-runtime", "controller": "network.RouteSpecController", "error":  
 "1 error occurred:\n\t* error adding route: netlink receive: network is unreachable, message {Family:2 DstLength:0 SrcLength:0 Tos:0 Table:0 Protocol:4 Scope:0 Type: 
 1 Flags:0 Attributes:{Dst:<nil> Src:<nil> Gateway:<GW> OutIface:124 Priority:1024 Table:254 Mark:0 Pref:<nil> Expires:<nil> Metrics:<nil> Multipath:[]}}      
 \n\n"}                                                                                                                                                                
 user: warning: [2025-03-23T16:44:02.124237404Z]: [talos] restarting controller in 358.522766ms {"component": "controller-runtime", "controller":                      
 "network.RouteSpecController"}                                                                                                                                        
 user: warning: [2025-03-23T16:44:02.124413404Z]: [talos] brought link up/down {"component": "controller-runtime", "controller": "network.LinkSpecController",         
 "link": "br0", "up": true}     
user: warning: [2025-03-23T16:44:02.124413404Z]: [talos] brought link up/down {"component": "controller-runtime", "controller": "network.LinkSpecController",         
 "link": "br0", "up": true}                                                                                                                                            
 kern:    info: [2025-03-23T16:44:02.214999404Z]: br0: port 1(eno1) entered blocking state                                                                             
 kern:    info: [2025-03-23T16:44:02.215008404Z]: br0: port 1(eno1) entered disabled state                                                                             
 kern:    info: [2025-03-23T16:44:02.215031404Z]: tg3 0000:02:00.0 eno1: entered allmulticast mode                                                                     
  SUBSYSTEM=pci                                                                                                                                                        
  DEVICE=+pci:0000:02:00.0                                                                                                                                             
 kern:    info: [2025-03-23T16:44:02.215099404Z]: tg3 0000:02:00.0 eno1: entered promiscuous mode                                                                      
  SUBSYSTEM=pci                                                                                                                                                        
  DEVICE=+pci:0000:02:00.0                                                                                                                                             
 user: warning: [2025-03-23T16:44:02.215324404Z]: [talos] enslaved/unslaved link {"component": "controller-runtime", "controller": "network.LinkSpecController",       
 "link": "eno1", "parent": "br0"}                                                                                                                                      
 user: warning: [2025-03-23T16:44:02.377119404Z]: [talos] brought link up/down {"component": "controller-runtime", "controller": "network.LinkSpecController",         
 "link": "eno1", "up": true}                                                                                                                                           
 user: warning: [2025-03-23T16:44:02.386869404Z]: [talos] no suitable node IP found, please make sure .machine.kubelet.nodeIP filters and pod/service subnets are set  
 up correctly {"component": "controller-runtime", "controller": "k8s.NodeIPController"}                                                                                
 user: warning: [2025-03-23T16:44:02.483385404Z]: [talos] controller starting {"component": "controller-runtime", "controller": "network.RouteSpecController"}         
 user: warning: [2025-03-23T16:44:02.485942404Z]: [talos] created route {"component": "controller-runtime", "controller": "network.RouteSpecController",               
 "destination": "default", "gateway": <GW>, "table": "main", "link": "eno1", "priority": 0, "family": "inet4"}                                               
 user: warning: [2025-03-23T16:44:02.486094404Z]: [talos] created route {"component": "controller-runtime", "controller": "network.RouteSpecController",               
 "destination": "default", "gateway": GW, "table": "main", "link": "br0", "priority": 1024, "family": "inet4"}                                             
 kern:    info: [2025-03-23T16:44:05.841838404Z]: tg3 0000:02:00.0 eno1: Link is up at 1000 Mbps, full duplex                                                          
  SUBSYSTEM=pci                                                                                                                                                        
  DEVICE=+pci:0000:02:00.0                                                                                                                                             
 kern:    info: [2025-03-23T16:44:05.841869404Z]: tg3 0000:02:00.0 eno1: Flow control is on for TX and on for RX                                                       
  SUBSYSTEM=pci                                                                                                                                                        
  DEVICE=+pci:0000:02:00.0                                                                                                                                             
 kern:    info: [2025-03-23T16:44:05.841877404Z]: tg3 0000:02:00.0 eno1: EEE is disabled                                                                               
  SUBSYSTEM=pci    
DEVICE=+pci:0000:02:00.0                                                                                                                                             
 kern:    info: [2025-03-23T16:44:05.842069404Z]: br0: port 1(eno1) entered blocking state                                                                             
 kern:    info: [2025-03-23T16:44:05.842080404Z]: br0: port 1(eno1) entered listening state                                                                            
 kern:    info: [2025-03-23T16:44:08.593861404Z]: tg3 0000:02:00.0 eno1: Link is down                                                                                  
  SUBSYSTEM=pci                                                                                                                                                        
  DEVICE=+pci:0000:02:00.0                                                                                                                                             
 kern:    info: [2025-03-23T16:44:08.595875404Z]: br0: port 1(eno1) entered disabled state                                                                             
 user: warning: [2025-03-23T16:44:08.639567404Z]: [talos] no suitable node IP found, please make sure .machine.kubelet.nodeIP filters and pod/service subnets are set  
 up correctly {"component": "controller-runtime", "controller": "k8s.NodeIPController"}                                                                                
 user: warning: [2025-03-23T16:44:16.463060404Z]: [talos] running image cleanup {"component": "controller-runtime", "controller": "runtime.CRIImageGCController"}      
 user: warning: [2025-03-23T16:44:16.463942404Z]: [talos] image is referenced, skipping garbage collection {"component": "controller-runtime", "controller":           
 "runtime.CRIImageGCController", "image": "gcr.io/etcd-development/etcd:v3.5.19"}                                                                                      
 user: warning: [2025-03-23T16:44:16.463998404Z]: [talos] image is referenced, skipping garbage collection {"component": "controller-runtime", "controller":           
 "runtime.CRIImageGCController", "image": "gcr.io/etcd-development/etcd@sha256:c415170328bbb2cb497c79dbd8494b1c8d58886dccf95fbfdd3845c13c6d1ca3"}                      
 user: warning: [2025-03-23T16:44:16.464028404Z]: [talos] image is referenced, skipping garbage collection {"component": "controller-runtime", "controller":           
 "runtime.CRIImageGCController", "image": "ghcr.io/siderolabs/kubelet:v1.32.3"}                                                                                        
 user: warning: [2025-03-23T16:44:16.465685404Z]: [talos] image is referenced, skipping garbage collection {"component": "controller-runtime", "controller":           
 "runtime.CRIImageGCController", "image": "ghcr.io/siderolabs/kubelet@sha256:0b4bea893ea2eb8e41b5d7bc1dc04359060d5e32aba5f609d0c4994590a074a2"}                        
 user: warning: [2025-03-23T16:44:16.465730404Z]: [talos] image is referenced, skipping garbage collection {"component": "controller-runtime", "controller":           
 "runtime.CRIImageGCController", "image": "sha256:0b4bea893ea2eb8e41b5d7bc1dc04359060d5e32aba5f609d0c4994590a074a2"}                                                   
 user: warning: [2025-03-23T16:44:16.465757404Z]: [talos] image is referenced, skipping garbage collection {"component": "controller-runtime", "controller":           
 "runtime.CRIImageGCController", "image": "sha256:c415170328bbb2cb497c79dbd8494b1c8d58886dccf95fbfdd3845c13c6d1ca3"}                                                   
 user: warning: [2025-03-23T16:44:20.661100404Z]: [talos] serving static pod manifests {"component": "controller-runtime", "controller":                               
 "k8s.StaticPodServerController", "size": 12390}

Before configuring the bridge, I was using a static IP address with this configuration:

    interfaces:
      - interface: eno0
        addresses:
          - <IP>/24
        routes:
          - network: 0.0.0.0/0
            gateway: <GW>

However, I believe the eno0 interface in the configuration is incorrect because I do not have this interface. Interestingly, if I change it to a value like eno1000, the configuration applies successfully.

Additionally, I applied a bridge configuration with eno0 (which does not actually exist), and it was created successfully. However, when debugging with bridge-utils, I noticed that the bridge had no interfaces attached. At the same time, eno1 was mysteriously brought up. Since my provider does not offer DHCP, I don’t think it could have been configured automatically.

Environment

  • Talos version: 1.9.5
  • Kubernetes version: 1.32.3
  • Platform: bare metal, amd64

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 15 (15 by maintainers)

Most upvoted comments

Hi, I’ve resolved the issue, but I have a few interesting observations:

  1. I tried setting dhcp: false for an interface without any routes, but eno1 still had routes assigned. I expected Talos to remove all routes from the interface in this case.

  2. My current configuration looks like this:

      - interface: eno1
        dhcp: false
      - interface: eno2
        dhcp: false
      - interface: br0
        addresses:
          - <ADDR>/24
        bridge:
          stp:
            enabled: false
          interfaces:
            - eno1
        routes:
          - network: 0.0.0.0/0
            gateway: <GW>

I expect Talos to disable DHCP on eno1, but I’m still seeing logs like:

 user: warning: [2025-04-13T22:24:44.954883855Z]: [talos] DHCP request/renew failed {"component": "controller-runtime", "controller":                                  
 "network.OperatorSpecController", "operator": "dhcp4", "error": "unable to receive an offer: got an error while the discovery request: no matching response packet    
 received", "link": "eno1"}   

I doesn’t expect DHCP requests when it disabled by config.

To sum up, I think the issue was a conflict between DHCP and static configuration. It seems like DHCP tried to assign an address or route to eno1. Then, when the bridge was configured, both DHCP and my static config were trying to apply the same address/routes, leading to conflicts.

I’m not sure what the issue is, the only thing I can think of is that the link is not brought up by Talos when it’s part of the bridge, you can try adding “empty” - interface: eno1.

You need first to understand what are the links you have, including their physical link status talosctl get links, and configure them accordingly. You don’t need a bridge usually.

Talos can’t validate link names in the machine configuration, as links might appear and disappear, so the configuration will be applied whenever the link pops up.