kubeedge: Failed to mount configmap/secret volume because of "no such file or directory"
What happened: Failed to mount configmap/secret volume because of “no such file or directory”. Although we can be sure that related resources are included in the sqlite.
I0527 10:35:29.789719 660 edged_volumes.go:54] Using volume plugin "kubernetes.io/empty-dir" to mount wrapped_kube-proxy
I0527 10:35:29.800195 660 process.go:685] get a message {Header:{ID:8b57a409-25c9-454e-a9ae-b23f0b1861a9 ParentID: Timestamp:1590546929789 ResourceVersion: Sync:true} Router:{Source:edged Group:meta Operation:query Resource:kube-system/configmap/kube-proxy} Content:<nil>}
I0527 10:35:29.800543 660 metaclient.go:121] send sync message kube-system/configmap/kube-proxy successed and response: {{ab5f3aab-11ff-48cf-8c3b-c5ded97678db 8b57a409-25c9-454e-a9ae-b23f0b1861a9 1590546929800 false} {metaManager meta response kube-system/configmap/kube-proxy} [{"data":{"config.conf":"apiVersion: kubeproxy.config.k8s.io/v1alpha1\nbindAddress: 0.0.0.0\nclientConnection:\n acceptContentTypes: \"\"\n burst: 0\n contentType: \"\"\n kubeconfig: /var/lib/kube-proxy/kubeconfig.conf\n qps: 0\nclusterCIDR: 192.168.0.0/16\nconfigSyncPeriod: 0s\nconntrack:\n maxPerCore: null\n min: null\n tcpCloseWaitTimeout: null\n tcpEstablishedTimeout: null\nenableProfiling: false\nhealthzBindAddress: \"\"\nhostnameOverride: \"\"\niptables:\n masqueradeAll: false\n masqueradeBit: null\n minSyncPeriod: 0s\n syncPeriod: 0s\nipvs:\n excludeCIDRs: null\n minSyncPeriod: 0s\n scheduler: \"\"\n strictARP: false\n syncPeriod: 0s\nkind: KubeProxyConfiguration\nmetricsBindAddress: \"\"\nmode: \"\"\nnodePortAddresses: null\noomScoreAdj: null\nportRange: \"\"\nudpIdleTimeout: 0s\nwinkernel:\n enableDSR: false\n networkName: \"\"\n sourceVip: \"\"","kubeconfig.conf":"apiVersion: v1\nkind: Config\nclusters:\n- cluster:\n certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\n server: https://10.10.102.78:6443\n name: default\ncontexts:\n- context:\n cluster: default\n namespace: default\n user: default\n name: default\ncurrent-context: default\nusers:\n- name: default\n user:\n tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token"},"metadata":{"creationTimestamp":"2020-04-21T14:50:46Z","labels":{"app":"kube-proxy"},"name":"kube-proxy","namespace":"kube-system","resourceVersion":"193","selfLink":"/api/v1/namespaces/kube-system/configmaps/kube-proxy","uid":"5651c863-c755-4da4-8039-b251efc82470"}}]}
E0527 10:35:29.800949 660 configmap.go:249] Error creating atomic writer: stat /var/lib/edged/pods/25e6f0ea-6364-4bcc-9937-9760b6ec956a/volumes/kubernetes.io~configmap/kube-proxy: no such file or directory
W0527 10:35:29.801070 660 empty_dir.go:392] Warning: Unmount skipped because path does not exist: /var/lib/edged/pods/25e6f0ea-6364-4bcc-9937-9760b6ec956a/volumes/kubernetes.io~configmap/kube-proxy
I0527 10:35:29.801109 660 record.go:24] Warning FailedMount MountVolume.SetUp failed for volume "kube-proxy" : stat /var/lib/edged/pods/25e6f0ea-6364-4bcc-9937-9760b6ec956a/volumes/kubernetes.io~configmap/kube-proxy: no such file or directory
E0527 10:35:29.801199 660 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/configmap/25e6f0ea-6364-4bcc-9937-9760b6ec956a-kube-proxy\" (\"25e6f0ea-6364-4bcc-9937-9760b6ec956a\")" failed. No retries permitted until 2020-05-27 10:37:31.80112802 +0800 CST m=+2599.727653327 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"kube-proxy\" (UniqueName: \"kubernetes.io/configmap/25e6f0ea-6364-4bcc-9937-9760b6ec956a-kube-proxy\") pod \"kube-proxy-gbdgw\" (UID: \"25e6f0ea-6364-4bcc-9937-9760b6ec956a\") : stat /var/lib/edged/pods/25e6f0ea-6364-4bcc-9937-9760b6ec956a/volumes/kubernetes.io~configmap/kube-proxy: no such file or directory"
What you expected to happen: mount successffuly How to reproduce it (as minimally and precisely as possible): Sorry for I can not provide the way of reproduction. Anything else we need to know?:
Environment:
- KubeEdge version(e.g.
cloudcore/edgecore --version
): v1.3.0
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 28 (28 by maintainers)
@zzxgzgz just delete configmap ready file about specific pod. Delete whole
plugins
directory is a convient way, because secret reconcile is no problem.Hey guys! I have been tracking this bug for the last few days.For now, I can draw some conclusions:
Reason: I still do not know what operation or reason causes the bug. If some one also meet this bug, plz tell me the detail.
Phenomena: All pod (beside kube-proxy) which use configmaps mounted into pod will meet this bug, and pending because of configmap mount fail. And it logs :
/var/lib/edged/pods/pod_uid/plugins
to triger re-create directory.And wait for seconds, for me, everything gose ok.@zzxgzgz
Does it mean that it is normal for kube-proxy running on the edge node?
No, for edge scene kubeproxy should not running on the edgenode, you could modify the yaml of kubeproxy daemonset to avoid it.
But for mizar, sounds like a cni plugin. Does it need to connect to api-server via service clusterip, just like calico? If so it may need kubeproxy run well
Is this the reason why the other containers cannot mount successfully?
No, I am working on why fail to mount.
Just a FYI, when I tried to deploy a test pod on the edge node, it was successful, but the Mizar related pods are still failing.
There are many reasons why pod can not run well on edgenodes. I suggest that you could view the logs in mizar container.