kubernetes: Sporadic "MountVolume.SetUp failed for volume ... not registered" failures seen in 1.22.0-1.22.8, 1.23.0-1.23.5
What happened:
We updated our cluster to 1.22.2. We are trying to create following cronjob:
apiVersion: batch/v1
kind: CronJob
metadata:
name: example
namespace: gha-devcloud
spec:
concurrencyPolicy: Forbid
failedJobsHistoryLimit: 1
jobTemplate:
spec:
activeDeadlineSeconds: 600
backoffLimit: 3
template:
spec:
containers:
- image: busybox
imagePullPolicy: Always
name: guillotine
dnsPolicy: ClusterFirst
restartPolicy: OnFailure
ttlSecondsAfterFinished: 180
schedule: '*/1 * * * *'
successfulJobsHistoryLimit: 3
suspend: false
However, it creates job like should and also pod.
When I check events for the pod I see following
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m25s default-scheduler Successfully assigned gha-devcloud/example-27206544--1-2gvfg to nodes-esptnl-eprwhk
Normal Pulling 2m25s kubelet Pulling image "busybox"
Normal Pulled 2m24s kubelet Successfully pulled image "busybox" in 1.049753158s
Normal Created 2m24s kubelet Created container guillotine
Normal Started 2m23s kubelet Started container guillotine
Warning FailedMount 2m22s (x2 over 2m23s) kubelet MountVolume.SetUp failed for volume "kube-api-access-wxtss" : object "gha-devcloud"/"kube-root-ca.crt" not registered
If I take copy from that pod manifest, and remove restartPolicy from the pod manifest, it will start correctly and pod will be executed.
What you expected to happen:
I expect that it will create cronjob and it will also be executed correctly.
How to reproduce it (as minimally and precisely as possible):
- create new cronjob using manifests that I provided
- check the pod result was it started or not
- NOTE: namespace must have
-in its name, without-everything works.
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version): 1.22.2 - Cloud provider or hardware configuration: openstack
- OS (e.g:
cat /etc/os-release): debian buster - Kernel (e.g.
uname -a): - Install tools: kops
- Network plugin and version (if this is a network-related bug):
- Others:
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 18
- Comments: 123 (67 by maintainers)
I am using AWS EKS in version 1.22 and still seing this issue. Since AWS restricts which Kubernetes versions are available to upgrade to, I do not think I am able to just bump the version and benefit from this fix.
Is there a way to manually fix it while waiting for AWS to actually update the version on their end?
we experience the same issue on openshift 4.9.7 which is based on k8s 1.22.2. we are running a
CronJobthat mounts custom configmaps and secrets as volumes:so i think it’s not only related to kube-root-ca.crt.
the job runs fine, but we see the messages above in the events every time the job is triggered.
Able to build the k8s kind images myself. Here is how to build kind node image for kubernetes in case anyone interested.
Confirmed v1.22.9 fixed this bug.
Confirmed v1.23.6 also fixed this bug.
Thanks a lot for the quick fix!
@zetaab As a workaround, if you don’t need the service account token, then you can disable it using
automountServiceAccountToken: falselike this:Also, I think the error message in the original issue description is only a warning, so the jobs are still running, just that you get this warning every time.
@zetaab , I executed the steps you provided. I agree with @cmssczy, this failure message is not related to namespace naming. I also got same mount error when I removed
-from namespace;@protosam btw 1.22.8 also does not have the fix introduced in https://github.com/kubernetes/kubernetes/pull/108754 , so likely it won’t fix this issue.
So there are two separate issues we are talking about:
From what I can tell issue-1 should be fixed by - https://github.com/kubernetes/kubernetes/pull/107831 . It should no longer prevent any pod startup.
For issue-2 here - we will need https://github.com/kubernetes/kubernetes/pull/107824 or a version of it. But that issue in itself does not prevent pod creation (or deletion). If someone has a way to reproduce a failed pod creation after https://github.com/kubernetes/kubernetes/pull/108754 , please add those steps here.
I’m experiencing the same issue some times for the
kube-root-ca.crtand always when the cron-job or the corresponding pod is mounting a configmap with a shell script in it. The warning appears on every run but the job itself runs successful.My cron-job runs a shell script to update the digitalocean dns record for my domain.
Appears since I upgraded my baremetal k3s cluster to v1.22.3
My errors from above are on both v1.23.8+k3s1 as well as v1.24.2+k3s2, so not fixed.
So basically, the problem that seem to be referenced mostly in this issue (being the MountFailed event though happening during pod termination) seem to actually be a long-standing problem. And this is strictly related to: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/pod/pod_manager.go#L191-L196
And I think it’s a long-standing issue (maybe we are just triggerring the race more frequently because of some changes now, but I think it was always there).
The problem is that there is no coordination between where this function is called (processing pod/PLEG updates) and the volumemanager (that is simply calling GetPods() and processing the pods, which in the meantime could have been marked as terminated and thus unregistered).
TBH, I’m not yet sure how to fix it nicely. We can try not unregistering pods in that case and rely on the (already existing) mechanism that we stop watching if nothing is accessing given secrets/configmaps for some time. And unregistering will happen on Deletion. But I think this race (although much else probable because we ignore terminating pods and termination takes some time before deletion) exists also in delete path. So I’m wondering if we can make something better here.
same error.
Cronjob cannot get the kube-api-access volume.
While there was a distinct reproducible issue found and fixed already in v1.22.9+, v1.23.6+, and 1.24+, it seems like there is still another issue remaining that produces a similar symptom.
The comment at https://github.com/kubernetes/kubernetes/issues/105204#issuecomment-1264385137 hints that this message might be a symptom of a pod that is failing to start for some other reason.
It would be helpful if someone still observing this symptom in a reproducible way on a version >= 1.22.9, 1.23.6, or 1.24 could open a new issue to track that with the following info:
That would help in gathering info related to any remaining issues that surface this message.
I can confirm. Don‘t see it on v1.23.8+k3s1
the just-released v1.22.9 and v1.23.6 include https://github.com/kubernetes/kubernetes/pull/107831 - are you able to reproduce the issue on those?
@Morriz, the error observed in my output is still a valid bug that causes configmaps to be unreliable in v1.22.2 specifically and in my testing on k3s, the otomi installation process seems viable from what I’ve observed to replicate the issue in case someone needs to do so.
I’ve got some further testing to do on k3s 1.22.8 later, if I still see issues I will take you up on your offer to help directly, barring the configmap problem still existing. I rank the configmaps working higher than getting otomi installed/working.
@ianhundere can you include the install method and version for Kubernetes? Helps to just verify so it’s not assumed knowledge.
using k3s on arm64
The fact that this is not easily reproducible in a way that prevents successful running of pods makes it seem like a race. Off the top of my head, some possibilities are:
A 1.22 control plane didn’t drive any new behavior in kubelets, but it would have made more use of shared configmap mounts because of the switch to projected service account tokens. An existing bug in 1.20+ kubelets could be exercised more
@zetaab ~Your manifest works fine for me on my 1.23.0-alpha cluster, even with a - in the namespace name, like you have.~ (EDIT: I tried this several days later and now I AM able to reproduce the error even when my namespace doesn’t have a dash. See my more recent comment).
What version was your cluster previously?
I think this could somehow be related to the ServiceAccount admission controller and the Bound Service Account Token Volume.
There was a feature gate that graduated in 1.21 that is supposed to copy the kube-root-ca.crt ConfigMap to every namespace:
I wonder if your namespace existed prior to 1.21, that maybe it never got the “kube-root-ca.crt” ConfigMap.
Do you have that ConfigMap in your gha-devcloud namespace?
If not, you might be able to trigger the rootcacertpublisher by updating your namespace, perhaps by adding an annotation (ie.
kubectl annotate ns gha-devcloud foo=baror something like that). I’m not sure if that will trigger the copy or not. If not, then maybe you can copy it there yourself from the default namespace.