k3s: [svclb-traefik] can't create /proc/sys/net/ipv4/ip_forward: Read-only file system

I also encounter this problem. Taking the traefik service for an instance

# kubectl get service -A -o wide
NAMESPACE     NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP                                     PORT(S)                      AGE     SELECTOR
default       kubernetes           ClusterIP      10.43.0.1       <none>                                          443/TCP                      2d23h   <none>
kube-system   kube-dns             ClusterIP      10.43.0.10      <none>                                          53/UDP,53/TCP,9153/TCP       2d23h   k8s-app=kube-dns
kube-system   metrics-server       ClusterIP      10.43.146.109   <none>                                          443/TCP                      2d23h   k8s-app=metrics-server
kube-system   traefik-prometheus   ClusterIP      10.43.224.69    <none>                                          9100/TCP                     2d23h   app=traefik,release=traefik
kube-system   traefik              LoadBalancer   10.43.49.23     192.168.192.180,192.168.192.2,192.168.192.205   80:30373/TCP,443:32464/TCP   2d23h   app=traefik,release=traefik

it should be accessible from all three external IPs displayed, however, it is actually accessible only from the external IP of the node where the pod is running, and not the other two.

I have ran the kubectl -n kube-system logs svclb-traefik-pl28k lb-port-80 command on each of the three svclb-traefik pods, and the output returned is as follows:

+ trap exit TERM INT
/usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
+ echo 1
+ true
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 1 '!=' 1 ]
+ iptables -t nat -I PREROUTING '!' -s 10.43.49.23/32 -p TCP --dport 80 -j DNAT --to 10.43.49.23:80
+ iptables -t nat -I POSTROUTING -d 10.43.49.23/32 -p TCP -j MASQUERADE
+ '[' '!' -e /pause ]
+ mkfifo /pause

I would like to know if it is right that the service is accessible only from the node where the traefik pod is running? Or should i manually set the iptables on the every node hosts?

_Originally posted by @tobunka-git in https://github.com/k3s-io/k3s/issues/3116#issuecomment-843680425_

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 4
Comments: 18 (2 by maintainers)

Most upvoted comments

it seems nobody cares about this

tuwchia on May 22, 2021

Adding some new discoveries. I ran iptables -L on all the three nodes in the cluster, and found out that forward policies on the three nodes are different. What makes me curious is that the results of sysctl net.ipv4.ip_forward are the same: net.ipv4.ip_forward = 1. The details are listed below. On the master node

Chain FORWARD (policy DROP)
target     prot opt source               destination         
KUBE-ROUTER-FORWARD  all  --  anywhere             anywhere             /* kube-router netpol - TEMCG2JMHZYE7H7T */
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  VM-8-9-ubuntu/16     anywhere            
ACCEPT     all  --  anywhere             VM-8-9-ubuntu/16

On the first worker node

Chain FORWARD (policy DROP)
target     prot opt source               destination         
KUBE-ROUTER-FORWARD  all  --  anywhere             anywhere             /* kube-router netpol - TEMCG2JMHZYE7H7T */
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  10.42.0.0/16         anywhere            
ACCEPT     all  --  anywhere             10.42.0.0/16

On the second worker node

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
KUBE-ROUTER-FORWARD  all  --  anywhere             anywhere             /* kube-router netpol - TEMCG2JMHZYE7H7T */
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  10.42.0.0/16         anywhere            
ACCEPT     all  --  anywhere             10.42.0.0/16

I’ll try to change the iptables forwarding policy manually, and will report the results later.

tuwchia on May 22, 2021