kubernetes: could not open moddep file modules.dep.bin

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened:

check kube-proxy logs, show these warning and error message:

ERROR: …/libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file ‘/lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin’`, error: exit status 1"
Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed)

W1102 10:08:24.513694       1 server.go:191] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
time="2017-11-02T10:08:25Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file '/lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin'`, error: exit status 1"
W1102 10:08:25.134961       1 server_others.go:263] Flag proxy-mode="" unknown, assuming iptables proxy
I1102 10:08:25.136763       1 server_others.go:117] Using iptables Proxier.
I1102 10:08:25.578433       1 server_others.go:152] Tearing down inactive rules.
E1102 10:08:26.356184       1 proxier.go:699] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
)
I1102 10:08:26.415998       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I1102 10:08:26.416237       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I1102 10:08:26.416375       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I1102 10:08:26.416448       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I1102 10:08:26.471268       1 config.go:202] Starting service config controller
I1102 10:08:26.471358       1 controller_utils.go:1041] Waiting for caches to sync for service config controller
I1102 10:08:26.482155       1 config.go:102] Starting endpoints config controller
I1102 10:08:26.482234       1 controller_utils.go:1041] Waiting for caches to sync for endpoints config controller
I1102 10:08:26.572238       1 controller_utils.go:1048] Caches are synced for service config controller
I1102 10:08:26.582491       1 controller_utils.go:1048] Caches are synced for endpoints config controller

check host path – /lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin, it exist.

[root@m-imulti-192-168-4-99 ~]# ls /lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin
/lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin

check container path – /lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin, it not exist !!

[root@m-imulti-192-168-4-99 ~]# docker ps -a | grep kube-proxy
1eb7b4305d5b        88e2c85d3d02                                      "/usr/local/bin/kube-"   17 hours ago        Up 17 hours                             k8s_kube-proxy_kube-proxy-qzgwn_kube-system_b5eb3bb5-bfb5-11e7-b06b-00e06f69d690_0
32cad0d3e9a5        gcr.io/google_containers/pause-amd64:3.0          "/pause"                 17 hours ago        Up 17 hours                             k8s_POD_kube-proxy-qzgwn_kube-system_b5eb3bb5-bfb5-11e7-b06b-00e06f69d690_0
[root@m-imulti-192-168-4-99 ~]# docker exec -ti 1eb7b4305d5b sh
# ls /lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin
ls: cannot access /lib/modules/3.10.0-514.el7.x86_64/modules.dep.bin: No such file or directory
#

What you expected to happen:

no error message

How to reproduce it (as minimally and precisely as possible):

follow kubeadm install document: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ check kube-proxy logs, in the head, you can see these error message.

Anything else we need to know?:

any thing you want to know, just tell me.

Environment:

Kubernetes version (use kubectl version): v1.8.2
Cloud provider or hardware configuration: none
OS (e.g. from /etc/os-release): CentOS-7
Kernel (e.g. uname -a):Linux m-imulti-192-168-4-99 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Install tools: kubeadm
Others: calico node:v2.5.1 cni: v1.10.0 kube-policy-controller:v0.7.0

Sig: /sig node /sig network

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 19

Commits related to this issue

Merge pull request #55586 from gogeof/myBranch Automatic merge from submit-queue. kubeadm mount /lib/modules to kube-proxy **What this PR does / why we need it**: fix kubeadm kube-proxy manifest e... — committed to kubernetes/kubernetes by deleted user 7 years ago

Most upvoted comments

@gogeof IPVS mode is still feature-gated, it works with the flag --feature-gates=SupportIPVSProxyMode=true . And the --ipvs-sync-period flag is also required.

for example:

  spec:
      containers:
      - command:
        - /usr/local/bin/kube-proxy
        - --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf
        - --cluster-cidr=10.10.0.0/16
        - --feature-gates=SupportIPVSProxyMode=true
        - --proxy-mode=ipvs
        - --ipvs-scheduler=rr
        - --ipvs-sync-period=1m

defcyy on Nov 9, 2017

@defcyy @gogeof when I use the command “kubeadm --feature-gates=SupportIPVSProxyMode=true” get the message unrecognized feature-gate key: SupportIPVSProxyMode what i need to do to fix the problem

sweetcczhang on Dec 18, 2017

@gogeof document about ipvs kernel modules https://github.com/kubernetes/kubernetes/tree/master/pkg/proxy/ipvs#load-ipvs-kernel-modules

defcyy on Nov 10, 2017